diff --git a/.agents/skills/bump-version-and-release/SKILL.md b/.agents/skills/bump-version-and-release/SKILL.md index 3c056b774..18792a965 100644 --- a/.agents/skills/bump-version-and-release/SKILL.md +++ b/.agents/skills/bump-version-and-release/SKILL.md @@ -64,11 +64,13 @@ Three workflows in `.github/workflows/` interact with `VERSION`: - **Trigger:** push to `main` or `develop` whose changed paths include `.env`, **and** the `VERSION=` line in `.env` differs from the previous commit. Also runs on manual `workflow_dispatch`. - **Behavior on tag change:** 1. Runs on a self-hosted ephemeral GPU runner (`[self-hosted, airstack-ephemeral]`). - 2. `docker compose build` for profiles `desktop,isaac-sim,ms-airsim`. - 3. `docker compose push` to `${PROJECT_DOCKER_REGISTRY}` (set in `.env` — currently `airlab-docker.andrew.cmu.edu/airstack`). - 4. Keyless `cosign sign` of every pushed image digest via GitHub OIDC. - 5. `cosign verify` against the workflow's certificate identity. + 2. Plans per service via `.github/workflows/scripts/docker_image_plan.py` (content fingerprint vs previous versioned image label). + 3. **Unchanged image inputs** → registry retag of the previous `v${PREV}_…` digest to `v${VERSION}_…` and `cache_*` (no rebuild). + 4. **Changed inputs** (or missing/unlabeled previous image, or `force_rebuild=true`) → `docker compose build` / `push` for those services only, labeling the new digest with `org.airstack.content-fingerprint`. + 5. Keyless `cosign sign` of every published image digest via GitHub OIDC. + 6. `cosign verify` against the workflow's certificate identity. - **Skip behavior:** if the merge commit on `main`/`develop` does not actually change `VERSION=`, the build job is skipped (the check-changes job sets `tag-changed=false`). +- **Docs-only VERSION bumps:** still required by `check-version-increment`, but publish should retag rather than rebuild once fingerprints are on the previous images. First publish after this feature lands (or `force_rebuild=true`) must rebuild to write the labels. ### 3. `deploy_docs_from_release.yaml` — versioned docs @@ -76,7 +78,7 @@ Three workflows in `.github/workflows/` interact with `VERSION`: - **Behavior:** runs `mike deploy --push --update-aliases latest`, publishing the docs site under the release tag and pointing the `latest` alias at it. - Companion workflows publish unversioned docs from `main` (default alias `main`) and `develop` (alias `develop`). -So the full release path is: bump `VERSION` → PR → merge to `main`/`develop` (rebuild + push + sign) → cut a GitHub Release matching that VERSION (versioned docs go live). +So the full release path is: bump `VERSION` → PR → merge to `main`/`develop` (retag unchanged images and/or rebuild changed ones + push + sign) → cut a GitHub Release matching that VERSION (versioned docs go live). ## Choosing the Bump Type diff --git a/.agents/skills/run-system-tests/SKILL.md b/.agents/skills/run-system-tests/SKILL.md index 3923b6842..22ce20520 100644 --- a/.agents/skills/run-system-tests/SKILL.md +++ b/.agents/skills/run-system-tests/SKILL.md @@ -98,6 +98,7 @@ The `system-tests.yml` workflow's `Parse pytest args` step automatically prepend - `/pytest -m takeoff_hover_land` → effectively runs `-m "build_packages or takeoff_hover_land"` - `/pytest` (no marks) → pytest defaults (everything) - `/pytest -m build_docker` → unchanged (the build_docker tests rebuild from scratch anyway) +- `/pytest -m build_packages` → **pull-only** (retag `cache_*`, no `image-build`, no Isaac). Add `--no-image-build` on other marks to skip the bake. This guarantees that ROS 2 workspaces are built inside the containers before any launch/liveliness test tries to source them. If you intentionally want to skip `build_packages` (e.g. you trust the prebuilt images), include it explicitly: `-m "liveliness and not build_packages"` would work, but the simpler path is to run locally where the prepend logic doesn't apply. @@ -157,7 +158,7 @@ The `airstack_env` fixture is parametrized over `(sim, num_robots, iteration)` t | Flag | Default | Affects | Becomes | |------|---------|---------|---------| -| `--sim` | `msairsim,isaacsim` | `airstack_env` | One env-tuple per sim | +| `--sim` | `isaacsim` | `airstack_env` | One env-tuple per sim (`msairsim` opt-in) | | `--num-robots` | `1,3` | `airstack_env` | Cross-product with sim | | `--stress-iterations` | `1` | `airstack_env` | Up/down cycles per `(sim, num_robots)` | | `--stable-duration` | `120` | `system.test_liveliness::test_stable` and `system.test_sensors::test_sensor_streams_stable` | Total seconds polled | @@ -356,7 +357,7 @@ If multiple tests need the same setup, add a fixture in `conftest.py` (not in yo - **Running on insufficient hardware**. `liveliness`, `sensors`, and `takeoff_hover_land` require an NVIDIA GPU plus nvidia-container-toolkit; without them the sim container won't get GPU access and topic Hz checks will time out. If you only have a CPU, scope to `-m "build_docker or build_packages"`. - **Expecting interactive sim feedback**. `airstack_env` runs headless by default (`MS_AIRSIM_HEADLESS=true`, `ISAAC_SIM_HEADLESS=true`, `QT_QPA_PLATFORM=offscreen`). Don't add stdin prompts, GUI dialogs, or `input()` calls to test code — they will hang in CI. For local visual debugging only, pass `--gui`. - **Not capturing metrics in a new test**. If a test fails silently (no metric recorded) the regression report has nothing to compare. Always record at least one scalar via `MetricsRecorder` so the test shows up in `metrics.json`. -- **Letting parametrize cardinality explode**. Defaults `--sim msairsim,isaacsim --num-robots 1,3` with `--stress-iterations 3` multiply stack bring-ups for each selected mark (`liveliness`, `sensors`, `takeoff_hover_land`, …) — expensive. Override locally to a single tuple while iterating. +- **Letting parametrize cardinality explode**. Default `--num-robots 1,3` (and `--sim msairsim` if you opt in) multiplies stack bring-ups for each selected mark (`liveliness`, `sensors`, `takeoff_hover_land`, …) — expensive. Override locally to a single tuple while iterating. `--sim` defaults to `isaacsim` only. - **Hardcoded container names**. Always use `find_container`, `get_robot_containers`, or `wait_for_container` — replica suffixes (`-1`, `-2`, `-3`) and compose project prefixes change. - **Asserting on stdout instead of using `read_log_tail`**. The conftest captures each subprocess's combined stdout/stderr in memory; assertions should reference it via `read_log_tail()` (`f"airstack up failed:\n{read_log_tail()}"`) so failures attach the relevant context to the JUnit XML. - **Trying to SSH into a CI runner mid-job**. Workers are ephemeral OpenStack VMs destroyed within ~30s of job completion. Re-running the job creates a fresh VM. For genuine debugging on the runner, see `.github/orchestrator/README.md` (also exposed at `tests/ci-cd-orchestrator.md`) — but in 99% of cases, reproduce locally with `airstack test`. diff --git a/.env b/.env index 04de90a2d..4aa2502f1 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.12" +VERSION="0.19.0-alpha.13" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/.github/orchestrator/README.md b/.github/orchestrator/README.md index c10da3383..f2df1ff22 100644 --- a/.github/orchestrator/README.md +++ b/.github/orchestrator/README.md @@ -1,131 +1,128 @@ -# AirStack CI Orchestrator +# AirStack CI Orchestrator (OSMO backend) -This describes how to use a self-hosted OpenStack VM to run GitHub Actions jobs on truly ephemeral workers. The orchestrator is a Python service that continuously polls GitHub for queued workflow jobs, spawns a fresh OpenStack instance for each one with a single-use JIT runner token, and reaps (deletes) the instance when the job completes. This allows us to run CI workloads on GPU-equipped VMs without sharing any state between runs or exposing long-lived credentials on the worker. +This describes how a small always-on orchestrator service runs GitHub Actions jobs on truly ephemeral GPU workers scheduled by [NVIDIA OSMO](https://nvidia.github.io/OSMO/). The orchestrator is a Python service that continuously polls GitHub for queued workflow jobs, submits a fresh **OSMO workflow** for each one (a single-use JIT runner in a privileged, GPU-enabled container), and reaps it when the job completes. Each CI job runs on a clean pod with no state shared between runs and no long-lived credentials on the worker. -The orchestrator VM is the only host that holds the GitHub PAT and the OpenStack credential; the workers are destroyed after a single job. +This is a drop-in replacement for the previous OpenStack-Nova backend. The GitHub side is unchanged — `system-tests.yml` still uses `runs-on: [self-hosted, airstack-ephemeral]`, the single-use JIT runner config, and the same-repo fork guard. Only the *spawn target* changed from "create a Nova VM" to "submit an OSMO workflow", so the one-job-per-worker, destroy-after semantics are identical: when the runner's `run.sh` exits after one job, the OSMO task completes and the pod is torn down. + +The orchestrator host is the only machine that holds the GitHub PAT and the OSMO service-account token; workers are destroyed after a single job. ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ -│ Orchestrator VM (airstack-ci-cd-orchestrator) │ +│ Orchestrator host (airstack-ci-cd-orchestrator, no GPU) │ │ │ │ airstack-orchestrator.service → orchestrator.py │ │ spawn loop (every 15s): │ │ • GET /repos//actions/runs?status=queued │ │ • POST /repos//actions/runners/generate-jitconfig│ -│ • openstack server create (image, flavor, user_data) │ -│ • record (job_id → server_id) in state.json │ +│ • osmo workflow submit runner-workflow.yaml --pool ... │ +│ • record (job_id → workflow_id) in state.json │ │ reap loop (every 30s): │ -│ • job completed → openstack server delete │ -│ • job age > N min → force delete (straggler) │ -│ • owned but not in state → orphan reap │ +│ • job completed → osmo workflow cancel (if live) │ +│ • job age > N min → osmo workflow cancel (straggler) │ +│ • our-named but not in state → orphan cancel │ │ │ │ /etc/airstack-orchestrator/ │ │ config.yaml │ │ github-pat │ -│ /home/orchestrator/.config/openstack/clouds.yaml │ +│ osmo-token (OSMO service-account token) │ │ /var/lib/airstack-orchestrator/state.json │ +│ /var/lib/airstack-orchestrator/.config/osmo (CLI session) │ └─────────┬─────────────────────────────────┬─────────────────┘ - │ Nova / Neutron API │ GitHub REST API + │ osmo CLI (submit/query/cancel) │ GitHub REST API ▼ ▼ ┌──────────────────────────────────┐ ┌──────────────────────┐ -│ Ephemeral worker (per job) │ │ GitHub Actions │ -│ Image: Ubuntu-24.04-GPU-Headless│ │ workflow_job queue │ -│ cloud-init: │ └──────────────────────┘ -│ install docker + nv toolkit │ -│ download GH runner │ +│ OSMO CI GPU pool (privileged) │ │ GitHub Actions │ +│ Ephemeral runner pod (per job): │ │ workflow_job queue │ +│ Image: airstack-ci-runner │ └──────────────────────┘ +│ start dockerd (DinD) │ │ run.sh --jitconfig │ -│ shutdown -h +1 │ +│ exit → task done → pod reaped │ └──────────────────────────────────┘ ``` Key properties: -- **Truly ephemeral**: every job runs on a clean VM. No Docker layer cache pollution, no leftover networks, no carry-over from prior runs. +- **Truly ephemeral**: every job runs on a clean pod. No Docker layer cache pollution, no leftover containers, no carry-over from prior runs. - **PAT isolation**: the GitHub PAT lives only on the orchestrator. Workers receive a single-use [JIT runner config](https://docs.github.com/en/rest/actions/self-hosted-runners?apiVersion=2022-11-28#create-configuration-for-a-just-in-time-runner-for-a-repository) — a base64 token bound to one runner registration, valid only for a short window. -- **Application-credential auth**: the orchestrator authenticates to OpenStack with an application credential (revocable, scoped, no password), not the user's `openrc.sh`. -- **Crash-safe reaping**: every server we spawn is tagged with `airstack-role=ephemeral-runner`. The reap loop force-deletes any owned server not present in `state.json`, so a crashed orchestrator can't leak instances. +- **Service-account auth**: the orchestrator authenticates to OSMO with a shared, non-personal [service-account token](https://nvidia.github.io/OSMO/main/deployment_guide/appendix/authentication/service_accounts.html) (the analog of the old OpenStack application credential). CI runs never route through an individual's account, so PRs don't consume anyone's personal GPU quota and nothing breaks when a person leaves. +- **Crash-safe reaping**: every workflow is named `gha-runner--`. The reap loop cancels any active workflow with that prefix not present in `state.json`, so a crashed orchestrator can't leak workflows. ## Prerequisites -- OpenStack instance already setup for the orchestrator VM. The orchestrator itself is lightweight and doesn't need a GPU. 1 vCPU, 2GB RAM, and 20GB disk is sufficient for the orchestrator service. Make sure you can ssh into it and that it has outbound internet access. -- An OpenStack flavor with GPU passthrough and enough disk to run Docker + the tests. The orchestrator spawns workers from this flavor, so it must have a GPU and sufficient disk (or `boot_volume_size_gb` must be set) to run the workloads. It's common for GPU flavors to have `disk=0`, which means they boot from an ephemeral disk — in that case, you must set `boot_volume_size_gb` to a value large enough for the OS + Docker images + test assets (e.g., 40GB). If your OpenStack setup supports it, you can also boot from a Cinder volume sourced from an image; in that case, pre-bake Docker and the NVIDIA toolkit into the image to speed up boot time. + +- **Orchestrator host** — a small always-on VM (no GPU). 1 vCPU, 2GB RAM, 20GB disk, outbound internet to `api.github.com` and your OSMO URL. This is the only long-lived piece; the GPU compute is ephemeral pods on OSMO. +- **An OSMO service account + dedicated CI pool.** Ask your OSMO admin to: + 1. Create a service account (e.g. `svc-airstack-ci`) and a long-lived access token — `osmo user create` + `osmo token set`. On IdP-backed deployments (e.g. auth tied to the CMU Andrew directory) this is a non-personal identity, so it survives people graduating/leaving. If policy forbids OSMO-native service accounts, use a *functional/departmental* identity, never a personal one. + 2. Grant that account a role whose policy allows `workflow:Create/Cancel/Query` **scoped to a dedicated CI GPU pool** (e.g. `pool/airstack-ci`) that has its own allocation, so CI doesn't contend with researchers' interactive jobs. + 3. **Enable "Privileged Mode Allowed"** on that pool's platform. The AirStack tests run `airstack up` (docker compose) inside the worker, which requires an inner Docker daemon → a privileged container. Without this, submissions are rejected. + 4. Confirm the API gateway (Envoy) accepts OSMO access tokens for the API (not only interactive IdP logins). +- **A prebaked runner image** pushed to a registry the pool can pull (see below). ## One-time setup -### 1. Create OpenStack application credential +### 1. Build & push the runner image -On your local workstation (not the orchestrator VM): +The worker image bakes in Docker CE + compose, the NVIDIA container toolkit, and the GitHub Actions runner (what cloud-init used to install at boot on the VM), so pod start is fast and the JIT token can't expire mid-bootstrap. ```bash -source ~/.airlabcloud/openrc.sh -openstack application credential create airstack-orchestrator \ - --description "AirStack CI orchestrator — spawns ephemeral test runners" +cd .github/orchestrator +./build-and-push.sh +# or manually: +# docker build -f runner.Dockerfile \ +# --build-arg RUNNER_VERSION=2.334.0 \ +# -t airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0 . +# docker push airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0 ``` -The output prints `id` and `secret`. Build a `clouds.yaml`: - -```yaml -clouds: - airstack: - auth_type: v3applicationcredential - auth: - auth_url: https://airlab-cloud.andrew.cmu.edu:5000/v3/ - application_credential_id: - application_credential_secret: - region_name: Airlab - interface: public - identity_api_version: 3 +No local Docker? Submit the one-shot OSMO builder (needs your Harbor creds in OSMO): + +```bash +osmo workflow submit .github/orchestrator/build-runner-on-osmo.yaml \ + --pool airstack --priority HIGH ``` -### 2. Stage credentials on the orchestrator VM +Set `runner_image: airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0` in `config.yaml` (step 4). Keep `RUNNER_VERSION` in sync with an [actions/runner release](https://github.com/actions/runner/releases). -```bash -# clouds.yaml: install for the orchestrator user (created in step 3) -scp clouds.yaml ubuntu@:/tmp/clouds.yaml +**Pool note (AirLab):** use the Keycloak-autosynced `airstack` pool (`privileged_allowed: true`). A hand-created `airstack-ci` pool is wiped by `synchronize_osmo_team_pools.py`. Ephemerality is per-job OSMO workflows, not a separate pool. + +### 2. Stage credentials on the orchestrator host +```bash # GitHub PAT: needs `Actions: read/write` and `Administration: read/write` # (fine-grained) or classic `repo` scope. -scp ~/.airlabcloud/airstack-github-pat.txt \ - ubuntu@:/tmp/github-pat +scp ~/airstack-github-pat.txt ubuntu@:/tmp/github-pat + +# OSMO service-account token (from `osmo token set`, provided by your admin). +scp ~/svc-airstack-ci-token.txt ubuntu@:/tmp/osmo-token ``` ### 3. Run setup.sh -On the orchestrator VM: +On the orchestrator host: ```bash git clone https://github.com/castacks/AirStack.git /tmp/airstack sudo bash /tmp/airstack/.github/orchestrator/setup.sh ``` -`setup.sh` creates the `orchestrator` system user, builds the Python venv, copies `orchestrator.py` and `cloud-init.yaml.j2` into `/opt/airstack-orchestrator/`, scaffolds `/etc/airstack-orchestrator/`, installs the systemd unit, and consumes `/tmp/github-pat`. - -You still need to put the `clouds.yaml` in place under the orchestrator user's home: - -```bash -sudo install -d -o orchestrator -g orchestrator -m 0700 \ - /home/orchestrator/.config/openstack -sudo install -o orchestrator -g orchestrator -m 0600 \ - /tmp/clouds.yaml /home/orchestrator/.config/openstack/clouds.yaml -sudo shred -u /tmp/clouds.yaml -``` +`setup.sh` creates the `orchestrator` system user, installs the `osmo` CLI, builds the Python venv, copies `orchestrator.py` and `runner-workflow.yaml.j2` into `/opt/airstack-orchestrator/`, scaffolds `/etc/airstack-orchestrator/`, installs the systemd unit, and consumes `/tmp/github-pat` and `/tmp/osmo-token`. ### 4. Fill in `/etc/airstack-orchestrator/config.yaml` -Edit the placeholders the example ships with: - | Field | What goes here | How to find it | |------|---------------|----------------| -| `flavor_name` | OpenStack flavor with GPU + enough disk | `openstack flavor list` | -| `network_name` | Network the workers attach to | `openstack network list` | -| `keypair_name` | SSH keypair for break-glass access | `openstack keypair list` | -| `security_group` | Outbound 443 must be allowed | `openstack security group list` | -| `availability_zone` | Optional AZ for the spawned instance; leave empty to let Nova pick | `openstack availability zone list` | -| `boot_volume_size_gb` | Set >0 if your flavor has `disk=0` (common for GPU flavors) — boots from a Cinder volume of this size sourced from `image_id`; leave 0 for direct image-boot | `openstack flavor show ` (check disk field) | -| `floating_ips` | Pre-allocated FIP pool, rotated through sequentially — each spawn picks the first free one. `max_concurrent` is capped at `len(pool)`. Leave empty to skip FIP attachment | `openstack floating ip list` | +| `osmo_url` | Your OSMO web service URL | from your OSMO admin | +| `pool` | Dedicated CI GPU pool | `osmo pool list` / the OSMO UI | +| `platform` | Optional hardware type within the pool; empty = pool default | `osmo pool list` | +| `runner_image` | Image from step 1 | the registry you pushed to | +| `cpu` / `gpu` / `memory` / `storage` | Resource request for the worker | size for full stack + sim | +| `privileged` | Must be `true` (docker compose inside the pod) | — | +| `priority` | `HIGH` \| `NORMAL` \| `LOW` | — | | `repo` | `owner/name` of the repo to poll | from GitHub URL | -| `runner_version` | Version tag from [actions/runner releases](https://github.com/actions/runner/releases) | check before each major upgrade | +| `runner_version` | Runner version baked into `runner_image` | matches step 1 | +| `max_concurrent` | Max simultaneous in-flight workflows | — | +| `max_job_minutes` | Straggler cancel ceiling | exceed the longest job | ### 5. Start the service @@ -134,7 +131,7 @@ sudo systemctl enable --now airstack-orchestrator.service journalctl -u airstack-orchestrator.service -f ``` -You should see `orchestrator started: repo=... labels=... max_concurrent=N` and then periodic poll activity. +You should see `orchestrator started (OSMO backend): repo=... pool=... max_concurrent=N`, an `osmo login succeeded` line, and then periodic poll activity. ## End-to-end verification @@ -142,147 +139,168 @@ You should see `orchestrator started: repo=... labels=... max_concurrent=N` and # Trigger a fast build-only run. gh workflow run system-tests.yml -f marks=build_docker -# Within ~30s, a server should appear: -openstack server list --metadata airstack-role=ephemeral-runner -# or if your OpenStack setup doesn't support metadata queries: -openstack server list --name '^ephemeral-' +# Within ~30s, a workflow should appear in the CI pool: +osmo workflow list --name gha-runner- --pool airstack-ci # Watch GitHub → Actions → Runners — the ephemeral runner should appear, # pick up the job, then disappear. -# Within ~30s of job completion, the server should be gone: -openstack server list --metadata airstack-role=ephemeral-runner -openstack server list --name '^ephemeral-' +# Within ~30s of job completion, the workflow should be terminal / gone from +# the active list: +osmo workflow list --name gha-runner- --pool airstack-ci --status RUNNING PENDING WAITING ``` ## Operational notes -- **State file**: `/var/lib/airstack-orchestrator/state.json` is the in-flight job tracker. Wiping it triggers an orphan sweep on the next reap iteration — owned servers will be force-deleted. Don't wipe it while jobs are mid-flight unless that's what you want. -- **Stuck instance**: any server older than `max_job_minutes` (default 90) is force-deleted regardless of GitHub job status. Bump this if liveliness/autonomy runs grow longer than ~75 minutes. +- **State file**: `/var/lib/airstack-orchestrator/state.json` is the in-flight job tracker (`job_id → workflow_id`). Wiping it triggers an orphan sweep on the next reap iteration — active `gha-runner-*` workflows will be cancelled. Don't wipe it while jobs are mid-flight unless that's what you want. +- **Straggler**: any workflow whose job has run longer than `max_job_minutes` (default 48h) is force-cancelled regardless of GitHub job status. +- **OSMO token rotation** (tokens expire — default 31 days): mint a new one and restart. + ```bash + # (admin) osmo token set svc-airstack-ci-token-2 --user svc-airstack-ci \ + # --roles osmo-user --expires-at 2027-12-31 + sudo install -o root -g orchestrator -m 0640 /tmp/osmo-token /etc/airstack-orchestrator/osmo-token + sudo systemctl restart airstack-orchestrator.service # re-runs `osmo login` + ``` - **PAT rotation**: `sudo install -o root -g orchestrator -m 0640 /tmp/new-pat /etc/airstack-orchestrator/github-pat && sudo systemctl restart airstack-orchestrator.service`. -- **Pause spawning** (e.g. for maintenance): `sudo systemctl stop airstack-orchestrator.service`. Already-spawned workers will still complete their jobs and self-shutdown; on restart, the reap loop deletes them. -- **Logs**: `journalctl -u airstack-orchestrator.service -f`. Cloud-init logs from individual workers are visible only via `openstack console log show ` while the worker is running. +- **Pause spawning** (e.g. for maintenance): `sudo systemctl stop airstack-orchestrator.service`. Already-submitted workers still complete their jobs; on restart, the reap loop cleans up. +- **Logs**: `journalctl -u airstack-orchestrator.service -f`. Per-worker logs come from `osmo workflow logs `. ## Debugging a failed job -When a GitHub workflow run fails or stalls, the failure can be in any of four places: the orchestrator (didn't spawn), cloud-init (didn't bootstrap), the GH Actions runner (didn't register or crashed), or the workflow steps themselves. Each has a different inspection path. +When a GitHub workflow run fails or stalls, the failure can be in one of four places: the orchestrator (didn't submit), the OSMO task (didn't schedule/pull), the GH Actions runner (didn't register or crashed), or the workflow steps themselves. Each has a different inspection path. -### 1. Find which worker ran the job +### 1. Find which workflow ran the job -`state.json` is the authoritative job ↔ server ↔ floating-IP map: +`state.json` is the authoritative job ↔ workflow map: ```bash -sudo jq -r '.jobs | to_entries[] | "\(.key)\t\(.value.server_id)\t\(.value.floating_ip)\t\(.value.runner_name)"' \ +sudo jq -r '.jobs | to_entries[] | "\(.key)\t\(.value.workflow_id)\t\(.value.workflow_name)"' \ /var/lib/airstack-orchestrator/state.json ``` -Pick the row for your failing `job_id` (visible in the GitHub Actions URL). Save the values: +Pick the row for your failing `job_id` (visible in the GitHub Actions URL): ```bash JOB_ID=73286176852 # from the GitHub UI -SERVER=$(sudo jq -r ".jobs[\"$JOB_ID\"].server_id" /var/lib/airstack-orchestrator/state.json) -FIP=$( sudo jq -r ".jobs[\"$JOB_ID\"].floating_ip" /var/lib/airstack-orchestrator/state.json) +WF=$(sudo jq -r ".jobs[\"$JOB_ID\"].workflow_id" /var/lib/airstack-orchestrator/state.json) ``` -If the job isn't in `state.json`, the orchestrator never spawned for it — see step 2 below. +If the job isn't in `state.json`, the orchestrator never submitted for it — see step 2. -### 2. Did the orchestrator spawn at all? +### 2. Did the orchestrator submit at all? ```bash sudo journalctl -u airstack-orchestrator.service --since "30 min ago" --no-pager ``` -What you want to see for a healthy spawn: +Healthy submit looks like: ```text -spawned server for job () -attached floating IP to server (job ) +submitted workflow for job () ``` -Common things that block a spawn (and how to spot them): +Common things that block a submit (and how to spot them): | Log line / symptom | What it means | Fix | |---|---|---| -| `find_queued_jobs failed: 401 ...` | PAT expired / wrong scope | Rotate the PAT (see Operational notes) | -| `spawn failed for job ...: Block Device Mapping is Invalid` | Flavor has `disk=0` and `boot_volume_size_gb` is 0 | Set `boot_volume_size_gb > 0` | -| `no free floating IP in pool` | All FIPs in `floating_ips` are already in use | Wait for an in-flight job to complete, or expand the pool | -| `floating_ips configured but not found` | Pool addresses don't exist in the project | Double-check `openstack floating ip list` | -| Job is queued in GitHub but no `spawned` log | Runner labels in the workflow's `runs-on` don't match `runner_labels` in config | Make them match | - -### 3. SSH into a running worker +| `find_queued_jobs failed: 401 ...` | GitHub PAT expired / wrong scope | Rotate the PAT | +| `osmo login failed ...` / `auth error` | OSMO token expired/invalid, or Envoy rejects access tokens | Rotate the OSMO token; confirm gateway accepts access tokens | +| `osmo workflow submit failed ... privileged` | Pool platform doesn't allow privileged | Ask admin to enable "Privileged Mode Allowed" on the CI pool | +| `osmo workflow submit failed ... pool` / permission | Service-account role lacks `workflow:Create` on the pool | Fix the role's pool-scoped policy | +| Job queued in GitHub but no `submitted` log | `runs-on` labels don't match `runner_labels` | Make them match | -If the worker is `ACTIVE`, the floating IP is attached and you can connect directly. The keypair was injected during spawn — use the matching private key: +### 3. Inspect the workflow / worker ```bash -ssh -i .pem ubuntu@"$FIP" -``` +# Status and scheduling detail. +osmo workflow query "$WF" --verbose -If your workstation can't reach the FIP subnet, jump through the orchestrator (which is on the same network): +# Scheduling / lifecycle events (image pull, start, evict, ...). +osmo workflow events "$WF" --task runner -```bash -ssh -J ubuntu@ -i .pem ubuntu@"$FIP" +# Combined stdout of the runner task — shows dockerd start, run.sh, and the job. +osmo workflow logs "$WF" --task runner +osmo workflow logs "$WF" --task runner --error # error stream +osmo workflow logs "$WF" --task runner -n 300 # last 300 lines ``` -### 4. SSH into a SHUTOFF worker +### 4. Break-glass shell into a running worker -Workers shut themselves down after `run.sh` exits (whether the job succeeded, failed, or the runner crashed). The orchestrator only deletes a server once GitHub reports the job `completed`, so a SHUTOFF worker is preserved while you debug. +If the workflow is still `RUNNING`, exec into the pod (replaces the old SSH-via-floating-IP path): ```bash -# Optional but safer — keep the orchestrator from reaping mid-session. -sudo systemctl stop airstack-orchestrator.service - -openstack server start "$SERVER" -# Wait ~30s, then SSH using the FIP from state.json. -ssh -i .pem ubuntu@"$FIP" +osmo workflow exec "$WF" runner # /bin/bash in the runner task ``` -When done, delete the worker manually and resume the orchestrator: +Once inside: ```bash -openstack server delete "$SERVER" -sudo jq "del(.jobs[\"$JOB_ID\"])" /var/lib/airstack-orchestrator/state.json \ - | sudo tee /var/lib/airstack-orchestrator/state.json.new >/dev/null -sudo mv /var/lib/airstack-orchestrator/state.json.new /var/lib/airstack-orchestrator/state.json -sudo systemctl start airstack-orchestrator.service +# GitHub Actions runner diagnostics. +ls -lt /home/runner/actions-runner/_diag/ +tail -300 /home/runner/actions-runner/_diag/Runner_*.log +tail -300 /home/runner/actions-runner/_diag/Worker_*.log + +# Inner Docker daemon (a frequent failure point for `airstack up`). +cat /var/log/dockerd.log +docker info 2>&1 | head +nvidia-smi ``` -### 5. What to read once you're on the worker +### 5. Common failure patterns at the worker -```bash -# Combined boot + cloud-init output. Most useful single file: shows every -# line our airstack-runner-bootstrap.sh printed, including run.sh's exit. -sudo less /var/log/cloud-init-output.log -sudo tail -300 /var/log/cloud-init-output.log - -# Cloud-init's structured log — quick way to surface errors. -sudo grep -E 'WARN|ERROR|FAIL' /var/log/cloud-init.log - -# GitHub Actions runner diagnostics. The Worker_*.log corresponds to the -# actual job execution; Runner_*.log covers registration and dispatch. -ls -lt /home/ubuntu/actions-runner/_diag/ -sudo tail -300 /home/ubuntu/actions-runner/_diag/Runner_*.log -sudo tail -300 /home/ubuntu/actions-runner/_diag/Worker_*.log - -# Sanity-check Docker came up cleanly — a frequent failure point. -sudo systemctl status docker -docker info 2>&1 | head +| Symptom in `osmo workflow logs` | Cause | Fix | +|---|---|---| +| `dockerd did not become ready` | Pod not privileged / DinD blocked | Enable privileged on the pool platform | +| `nvidia-smi unavailable` / no GPU | GPU not requested/passed, or toolkit missing | Check `gpu:` request, platform GPUs, privileged | +| `Could not connect to api.github.com` | Egress blocked from the pool | Allow outbound 443 from the CI pool | +| `Bad credentials` / `Invalid ... runnerEvent` | JIT config TTL elapsed before `run.sh` started | Prebake the image (already done) so start is fast | +| `Cannot connect to the Docker daemon` during tests | inner dockerd crashed | Read `/var/log/dockerd.log` via `osmo workflow exec` | +| Runner registered, then `pytest` failed | A normal test failure | Read the GitHub Actions log — the canonical view | +| `No space left on device` | `storage` too small for images + sim assets | Bump `storage` in `config.yaml` | +| `failed to solve: ... mount source: "overlay" ... err: invalid argument` | Docker data-root landed on the pod's overlay rootfs | See "Nested DinD and overlayfs" below | + +### Nested DinD and overlayfs + +The single most likely way to break every Docker build at once. The pod's root +filesystem is overlayfs, and **Linux refuses to use a directory on overlayfs as +an overlay `upperdir`** (it returns `EINVAL`). A dockerd whose data-root sits on +the pod rootfs looks healthy — `docker info` works, image pulls succeed, because +containerd unpacks layers with plain writes — and then every build step that +needs a real mount fails: + +```text +failed to solve: process "/bin/bash -c apt-get ... " did not complete successfully: +mount source: "overlay", +target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/buildkit1459786452", +fstype: overlay, ... err: invalid argument ``` -### 6. Console log fallback +That signature took out all four `build_docker` tests and all four +`build_packages` tests in one run, with each failure looking like an unrelated +`apt-get`/`WORKDIR` problem. -Some flavors on this cloud don't expose the serial console (`openstack console log show` returns *Guest does not have a console available*). For those, the SSH path above is the only option. Where it does work, the console log persists across SHUTOFF and is faster than restarting the VM: +[`runner-entrypoint.sh`](runner-entrypoint.sh) handles this before starting +dockerd. It picks a storage backend by **performing a real overlay mount** to +test each option rather than trusting the filesystem type, and falls back in +this order: + +| Order | Backend | Notes | +|---|---|---| +| 1 | Loopback ext4 image mounted at `/var/lib/docker` | Preferred. Real `overlay2`, self-contained, dies with the pod. Sparse, so it only consumes what Docker writes. Sized to free space on `/` minus 20 GiB, or `DOCKER_LOOP_SIZE_MB`. | +| 2 | A real filesystem already mounted in the pod | Kubernetes `emptyDir`/`hostPath`/PVC volumes live on the node disk, not the overlay rootfs. `/osmo/data/output` and `/osmo/data/socket` are skipped — the OSMO ctrl sidecar owns them. | +| 3 | `fuse-overlayfs` driver | Stacks where the kernel driver won't. Needs `/dev/fuse`. | +| 4 | `vfs` driver | Always works, copies the whole filesystem per layer. Too slow and too large for the sim images — **reaching this is a red flag**, not a working state. | + +The chosen backend is logged at startup, so confirm it in the job log before +debugging anything else: ```bash -openstack console log show "$SERVER" | tail -200 +osmo workflow logs "$WF" --task runner | grep -E 'runner-entrypoint|storage driver' ``` -### 7. Common failure patterns at the worker +Backends 3 and 4 also set `features.containerd-snapshotter: false`, because +`storage-driver` is only honoured by the classic image store. -| Symptom in `cloud-init-output.log` (near end) | Cause | Fix | -|---|---|---| -| `Could not connect to api.github.com` / DNS errors | Security group blocking egress, or no NAT for the network | Allow outbound 443; if behind NAT, ensure FIP networking covers egress | -| `Bad credentials` / `Invalid configuration ... runnerEvent` | JIT config TTL elapsed before `run.sh` started — bootstrap took too long | Pre-bake Docker + nvidia-container-toolkit into the image to shrink bootstrap | -| `nvidia-ctk: command not found` or NVIDIA driver mismatch | Image's driver doesn't match the toolkit version | Use a different image, or pin a compatible toolkit version | -| `apt-get update` fails | Image's apt sources are unreachable from this network | Check network/security-group; or pre-bake packages into the image | -| Runner registered, then `pytest` failed | A normal test failure | Read the GitHub Actions log — that's the canonical view of the workflow output | -| `No space left on device` | `boot_volume_size_gb` too small for Docker images + sim assets | Bump `boot_volume_size_gb` | +The one-shot [`build-runner-on-osmo.yaml`](build-runner-on-osmo.yaml) builder +sidesteps the same problem differently — `vfs` plus `DOCKER_BUILDKIT=0` — which +is fine there because it builds one small image. diff --git a/.github/orchestrator/airstack-orchestrator.service b/.github/orchestrator/airstack-orchestrator.service index 7123232eb..0c5c1843d 100644 --- a/.github/orchestrator/airstack-orchestrator.service +++ b/.github/orchestrator/airstack-orchestrator.service @@ -1,5 +1,5 @@ [Unit] -Description=AirStack CI Orchestrator (spawns ephemeral OpenStack runners) +Description=AirStack CI Orchestrator (submits ephemeral OSMO runner workflows) Documentation=https://github.com/castacks/AirStack/tree/main/.github/orchestrator After=network-online.target Wants=network-online.target @@ -10,17 +10,20 @@ User=orchestrator Group=orchestrator WorkingDirectory=/opt/airstack-orchestrator -# Application credential lives in the orchestrator user's home so openstacksdk -# finds it via the default cloud-config search path. -Environment=HOME=/home/orchestrator -Environment=OS_CLIENT_CONFIG_FILE=/home/orchestrator/.config/openstack/clouds.yaml +# The `osmo` CLI persists its login session under $HOME/XDG dirs. Point them at +# the (writable) state dir so ProtectHome can stay read-only. setup.sh creates +# these directories owned by the orchestrator user. +Environment=HOME=/var/lib/airstack-orchestrator +Environment=XDG_CONFIG_HOME=/var/lib/airstack-orchestrator/.config +Environment=XDG_CACHE_HOME=/var/lib/airstack-orchestrator/.cache +Environment=XDG_STATE_HOME=/var/lib/airstack-orchestrator/.state ExecStart=/opt/airstack-orchestrator/venv/bin/python \ /opt/airstack-orchestrator/orchestrator.py \ --config /etc/airstack-orchestrator/config.yaml \ --pat /etc/airstack-orchestrator/github-pat \ --state /var/lib/airstack-orchestrator/state.json \ - --template /opt/airstack-orchestrator/cloud-init.yaml.j2 + --template /opt/airstack-orchestrator/runner-workflow.yaml.j2 Restart=always RestartSec=10 @@ -33,6 +36,8 @@ KillSignal=SIGTERM NoNewPrivileges=true ProtectSystem=strict ProtectHome=read-only +# ReadWritePaths re-grants write access under ProtectSystem/ProtectHome so the +# OSMO CLI session cache and state.json can be written. ReadWritePaths=/var/lib/airstack-orchestrator PrivateTmp=true diff --git a/.github/orchestrator/build-and-push.sh b/.github/orchestrator/build-and-push.sh new file mode 100755 index 000000000..aacefb002 --- /dev/null +++ b/.github/orchestrator/build-and-push.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Build & push the AirStack CI ephemeral-runner image to AirLab Harbor. +# Run on a linux/amd64 machine (or buildx --platform linux/amd64) with: +# docker login airlab-docker.andrew.cmu.edu +# +# Usage: +# ./build-and-push.sh +# RUNNER_VERSION=2.336.0 ./build-and-push.sh + +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REGISTRY="${REGISTRY:-airlab-docker.andrew.cmu.edu/airstack}" +RUNNER_VERSION="${RUNNER_VERSION:-2.336.0}" +IMAGE="${REGISTRY}/airstack-ci-runner:${RUNNER_VERSION}" + +echo "==> Building ${IMAGE}" +docker build \ + -f "${ROOT}/runner.Dockerfile" \ + --build-arg "RUNNER_VERSION=${RUNNER_VERSION}" \ + -t "${IMAGE}" \ + "${ROOT}" + +echo "==> Pushing ${IMAGE}" +docker push "${IMAGE}" + +echo "==> Done. Set in /etc/airstack-orchestrator/config.yaml:" +echo " runner_image: \"${IMAGE}\"" diff --git a/.github/orchestrator/build-runner-on-osmo.yaml b/.github/orchestrator/build-runner-on-osmo.yaml new file mode 100644 index 000000000..a44f21ab5 --- /dev/null +++ b/.github/orchestrator/build-runner-on-osmo.yaml @@ -0,0 +1,63 @@ +# One-shot OSMO job: build + push airstack-ci-runner to AirLab Harbor. +# Uses the existing privileged DinD workspace image (same as airstack-dev). +# +# Prereq: your OSMO profile has airlab-docker-login (+ auto REGISTRY cred). +# +# osmo workflow submit .github/orchestrator/build-runner-on-osmo.yaml \ +# --pool airstack --priority HIGH +# +# Watch: +# osmo workflow logs --task build +# Cancel when done if it hangs: +# osmo workflow cancel --force + +workflow: + name: build-airstack-ci-runner + resources: + build: + cpu: 8 + gpu: 0 + memory: 16Gi + storage: 100Gi + platform: default + tasks: + - name: build + image: airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest + resource: build + privileged: true + credentials: + airlab-docker-login: + AIRLAB_REGISTRY_USER: username + AIRLAB_REGISTRY_PASS: password + environment: + AIRSTACK_REPO_URL: "https://github.com/castacks/AirStack.git" + AIRSTACK_BRANCH: "ci/osmo-orchestrator" + RUNNER_VERSION: "2.336.0" + REGISTRY: "airlab-docker.andrew.cmu.edu/airstack" + command: + - bash + - -lc + - | + set -euo pipefail + # Nested DinD: overlay-on-overlay breaks BuildKit. Use vfs + legacy builder. + mkdir -p /etc/docker + cat > /etc/docker/daemon.json <<'JSON' + {"storage-driver": "vfs"} + JSON + dockerd >/var/log/dockerd.log 2>&1 & + for _ in $(seq 1 90); do docker info >/dev/null 2>&1 && break; sleep 1; done + docker info >/dev/null 2>&1 || { cat /var/log/dockerd.log; exit 1; } + docker info | grep -i 'Storage Driver' || true + + echo "$AIRLAB_REGISTRY_PASS" | docker login airlab-docker.andrew.cmu.edu \ + -u "$AIRLAB_REGISTRY_USER" --password-stdin + + rm -rf /tmp/AirStack + git clone --depth 1 --branch "$AIRSTACK_BRANCH" "$AIRSTACK_REPO_URL" /tmp/AirStack + cd /tmp/AirStack/.github/orchestrator + IMAGE="${REGISTRY}/airstack-ci-runner:${RUNNER_VERSION}" + DOCKER_BUILDKIT=0 docker build -f runner.Dockerfile \ + --build-arg "RUNNER_VERSION=${RUNNER_VERSION}" \ + -t "$IMAGE" . + docker push "$IMAGE" + echo "PUSHED $IMAGE" diff --git a/.github/orchestrator/cloud-init.yaml.j2 b/.github/orchestrator/cloud-init.yaml.j2 deleted file mode 100644 index 921417c18..000000000 --- a/.github/orchestrator/cloud-init.yaml.j2 +++ /dev/null @@ -1,71 +0,0 @@ -#cloud-config -# Rendered per-spawn by orchestrator.py with two Jinja variables: -# encoded_jit_config - single-use base64 JIT config from GitHub -# runner_version - GitHub Actions runner version (e.g. 2.334.0) -# -# The base image (Ubuntu-24.04-GPU-Headless) already has NVIDIA drivers. -# This cloud-init adds Docker (with the compose plugin), nvidia-container-toolkit, -# downloads the GitHub Actions runner, registers it with the JIT config, runs -# exactly one job (the JIT config + --ephemeral makes the runner exit after one -# job), and shuts the VM down. The orchestrator then deletes the server. - -package_update: true -package_upgrade: false -packages: - - jq - - curl - - ca-certificates - - gnupg - -write_files: - - path: /usr/local/bin/airstack-runner-bootstrap.sh - permissions: "0755" - owner: root:root - content: | - #!/usr/bin/env bash - set -euxo pipefail - - # Install Docker (with compose plugin) from Docker's official channel. - # get.docker.com handles apt repo setup + nvidia-container-toolkit-compatible - # docker-ce, plus the docker-compose-plugin we need for `airstack up`. - curl -fsSL https://get.docker.com | sh - - # nvidia-container-toolkit is required for GPU containers (liveliness / - # autonomy tests). The base image has the NVIDIA *drivers* but we still - # need the container runtime hooks here. - distribution=$(. /etc/os-release; echo "$ID$VERSION_ID") - curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ - | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg - curl -fsSL "https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list" \ - | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ - > /etc/apt/sources.list.d/nvidia-container-toolkit.list - apt-get update - apt-get install -y nvidia-container-toolkit - nvidia-ctk runtime configure --runtime=docker - systemctl restart docker - - usermod -aG docker ubuntu - - # GitHub Actions runner. - RUNNER_VERSION="{{ runner_version }}" - RUNNER_DIR=/home/ubuntu/actions-runner - mkdir -p "$RUNNER_DIR" - cd "$RUNNER_DIR" - curl -fsSL -o runner.tar.gz \ - "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" - tar xzf runner.tar.gz - rm runner.tar.gz - chown -R ubuntu:ubuntu "$RUNNER_DIR" - - # Run exactly one job under the ubuntu user. The JIT config is single-use - # and ephemeral, so run.sh exits after one job completes. - sudo -u ubuntu --preserve-env=HOME -H bash -c \ - "cd '$RUNNER_DIR' && ./run.sh --jitconfig '{{ encoded_jit_config }}'" \ - || echo "runner exited non-zero (job failure or runner error)" - - # Backstop: power down. The orchestrator's reap loop is the authoritative - # deleter — it sees the GitHub job complete and calls Nova delete. - shutdown -h +1 - -runcmd: - - /usr/local/bin/airstack-runner-bootstrap.sh diff --git a/.github/orchestrator/config.example.yaml b/.github/orchestrator/config.example.yaml index 4a47bbe1f..ba1c64384 100644 --- a/.github/orchestrator/config.example.yaml +++ b/.github/orchestrator/config.example.yaml @@ -1,54 +1,66 @@ -# AirStack CI orchestrator configuration. +# AirStack CI orchestrator configuration (OSMO backend). # Copy to /etc/airstack-orchestrator/config.yaml and fill in placeholders. -# --- OpenStack target --- - -# Cloud profile name in ~/.config/openstack/clouds.yaml. -openstack_cloud: airstack - -# Ubuntu-24.04-Desktop (confirmed available on airlab-cloud). -image_id: 2ebb9061-8995-4238-a3cc-e230a3e863aa - -# OpenStack flavor with GPU + enough disk for Docker + sim images. -# Look up with: openstack flavor list -flavor_name: "gpu.rtxpro5000.1" - -# OpenStack network the ephemeral instance attaches to. Must allow outbound -# 443 to api.github.com (no inbound is required: the runner makes an outbound -# long-poll connection to GitHub). -network_name: "airstack.AirLab.Apps_group_network_gates" - -# OpenStack keypair injected into the instance for break-glass SSH access. -# The orchestrator never SSHes into workers itself. -keypair_name: "airstack-ci-cd" - -# Security group applied to spawned instances. Outbound 443 must be allowed. -security_group: "default" - -# OpenStack availability zone to spawn instances in (e.g. nova, gpu-zone-1). -# Leave empty to let Nova pick. -availability_zone: "gates" - -# If the chosen flavor has disk=0 (common for GPU flavors), Nova rejects -# direct image-boot with "Block Device Mapping is Invalid: You specified more -# local devices than the limit allows". Set this to >0 to boot from a Cinder -# volume of that size sourced from image_id (deleted on termination). Leave -# at 0 to boot directly from the image (only works for non-zero-disk flavors). -boot_volume_size_gb: 300 - -# Pre-allocated pool of floating IPs to rotate through for SSH access to -# workers. The orchestrator picks the first free IP from this list, in order, -# for each new spawn. When the worker is destroyed the IP auto-disassociates -# and returns to the pool. If non-empty, max_concurrent is capped at len(pool) -# so the orchestrator never spawns a worker it can't address. -# Allocate via: openstack floating ip create -# Leave empty to skip floating-IP attachment entirely. -floating_ips: [] -# Example: -# floating_ips: -# - 172.19.220.131 -# - 172.19.220.171 -# - 172.19.220.89 +# --- OSMO target --- + +# Path to the `osmo` CLI. The install script (see setup.sh / README) puts it on +# PATH as `osmo`; override with a full path if needed. +osmo_bin: "osmo" + +# URL of your OSMO web service (the control plane the CLI logs into). +# AirLab: https://airlab-share-01.andrew.cmu.edu +osmo_url: "https://airlab-share-01.andrew.cmu.edu" + +# File containing the OSMO service-account access token. This is the shared, +# non-personal "lab" identity — the analog of the old OpenStack application +# credential. The orchestrator runs `osmo login --method token --token-file` +# with it. Created by an OSMO admin via `osmo user create` + `osmo token set` +# (see README). It never leaves this host. +osmo_token_file: "/etc/airstack-orchestrator/osmo-token" + +# GPU pool for ephemeral runners. AirLab team pools are Keycloak-autosynced; +# use the stable `airstack` pool (privileged_allowed=true). A hand-made +# `airstack-ci` pool will be wiped by synchronize_osmo_team_pools.py unless +# it is added to Keycloak. The service-account needs workflow:Create on this +# pool (role osmo-airstack) plus osmo-user for cancel/query. +pool: "airstack" + +# Optional platform (hardware type) to target within the pool. Leave empty to +# use the pool's default platform. List options with `osmo pool list` / the UI. +platform: "default" + +# Scheduling priority: HIGH | NORMAL | LOW. +priority: "NORMAL" + +# --- Runner task (the per-job worker) --- + +# Prebaked image: Docker CE + compose + nvidia-container-toolkit + GH Actions +# runner. Build & push runner.Dockerfile to a registry the pool can pull from. +runner_image: "airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.336.0" + +# Resource request for the runner container. Size for the full stack build + +# sim (Isaac Sim / ms-airsim + robot + gcs) running under docker compose. +cpu: 8 +gpu: 1 +memory: "32Gi" +storage: "300Gi" + +# REQUIRED for the AirStack tests: they run `airstack up` (docker compose) +# inside the pod, which needs an inner Docker daemon -> a privileged container. +# The pool's platform must have "Privileged Mode Allowed" enabled by your OSMO +# admin, otherwise submission/scheduling will be rejected. +privileged: true + +# Use the node's host network for the runner container. Usually not needed +# (the runner only makes outbound calls to GitHub); leave false unless the +# tests need host networking. +host_network: false + +# GitHub Actions runner version to bake into runner_image. Kept here for +# reference/traceability; it is a build arg of runner.Dockerfile, not consumed +# by the orchestrator at runtime. Must match a tag at +# https://github.com/actions/runner/releases +runner_version: "2.336.0" # --- GitHub --- @@ -56,23 +68,22 @@ floating_ips: [] repo: "castacks/AirStack" # Labels the orchestrator polls for. A queued workflow_job whose `labels` -# array is a superset of this list gets a server spawned for it. +# array is a superset of this list gets a workflow submitted for it. These are +# unchanged from the OpenStack backend, so system-tests.yml needs no edits. runner_labels: - self-hosted - airstack-ephemeral -# GitHub Actions runner version (must exist as a release tag at -# https://github.com/actions/runner/releases). -runner_version: "2.334.0" - # --- Limits --- -# Maximum simultaneous in-flight ephemeral instances. +# Maximum simultaneous in-flight workflows the orchestrator will submit. OSMO +# queues anything beyond the pool's capacity on its own, but this caps how many +# jobs we hand it at once. max_concurrent: 3 -# Hard ceiling for a single job. Past this age the reaper force-deletes the -# server even if GitHub still reports the job as in-progress. Must comfortably -# exceed the longest expected job (autonomy/liveliness runs). +# Hard ceiling for a single job. Past this age the reaper cancels the workflow +# even if GitHub still reports the job as in-progress. Must comfortably exceed +# the longest expected job (liveliness / autonomy runs). max_job_minutes: 2880 # 48 hours # --- Polling intervals (seconds) --- @@ -80,8 +91,10 @@ max_job_minutes: 2880 # 48 hours spawn_poll_interval_s: 15 reap_poll_interval_s: 30 -# How long to wait for a freshly-created server to reach ACTIVE before -# treating the spawn as failed. If Nova flips the server to ERROR within this -# window the orchestrator logs the full fault (code/message/details/host/AZ) -# and deletes the server so the next iteration can retry cleanly. -server_active_timeout_s: 300 +# How long to wait for `osmo workflow submit` to return before treating the +# submission as failed. +submit_timeout_s: 180 + +# Name prefix for submitted workflows. Also used by the orphan sweep to find +# workflows this orchestrator owns. Keep the trailing dash. +workflow_name_prefix: "gha-runner-" diff --git a/.github/orchestrator/orchestrator.py b/.github/orchestrator/orchestrator.py index 3e65e906f..4af62555b 100644 --- a/.github/orchestrator/orchestrator.py +++ b/.github/orchestrator/orchestrator.py @@ -1,56 +1,64 @@ #!/usr/bin/env python3 -"""AirStack CI orchestrator. +"""AirStack CI orchestrator (OSMO backend). Polls the GitHub API for queued workflow_jobs whose labels match this -orchestrator's runner_labels, and spawns truly ephemeral OpenStack instances -to execute them. Each ephemeral instance receives a single-use GitHub JIT -runner config via cloud-init; the GitHub PAT never leaves this orchestrator. +orchestrator's runner_labels, and submits truly ephemeral OSMO workflows to +execute them. Each workflow runs a single-job GitHub Actions runner in a +privileged, GPU-enabled container on an OSMO compute pool; the GitHub PAT never +leaves this orchestrator, and an OSMO service-account token (not a personal +account) is used only to submit / query / cancel workflows. + +This is a drop-in replacement for the previous OpenStack-Nova backend: the +GitHub side is unchanged (`runs-on: [self-hosted, airstack-ephemeral]`, the +single-use JIT runner config, the same-repo fork guard). Only the *spawn* +target changed from "create a Nova VM" to "submit an OSMO workflow". The +one-job-per-worker, destroy-after semantics are preserved — when the runner's +`run.sh` exits after a single job, the OSMO task completes and the pod is torn +down. Two cooperating loops: - - spawn loop: discover queued jobs, spawn one Nova server per job - - reap loop: delete servers whose jobs have completed, plus stragglers + - spawn loop: discover queued jobs, submit one OSMO workflow per job + - reap loop: cancel workflows whose jobs have completed, plus stragglers older than max_job_minutes and orphans not in state.json State persists in /var/lib/airstack-orchestrator/state.json so the -orchestrator can survive restarts without leaking instances. +orchestrator can survive restarts without leaking workflows. """ from __future__ import annotations import argparse -import base64 import json import logging import os +import re import signal +import subprocess import sys +import tempfile import threading import time from datetime import datetime, timezone from pathlib import Path from typing import Any -import openstack import requests import yaml from jinja2 import Template DEFAULT_CONFIG_PATH = "/etc/airstack-orchestrator/config.yaml" DEFAULT_PAT_PATH = "/etc/airstack-orchestrator/github-pat" +DEFAULT_OSMO_TOKEN_PATH = "/etc/airstack-orchestrator/osmo-token" DEFAULT_STATE_PATH = "/var/lib/airstack-orchestrator/state.json" -DEFAULT_TEMPLATE_PATH = "/opt/airstack-orchestrator/cloud-init.yaml.j2" - -# Metadata key/value applied to every Nova server we spawn. Used by the -# orphan reaper to identify servers we own even when state.json is missing. -ROLE_META_KEY = "airstack-role" -ROLE_META_VAL = "ephemeral-runner" -JOB_META_KEY = "airstack-job-id" +DEFAULT_TEMPLATE_PATH = "/opt/airstack-orchestrator/runner-workflow.yaml.j2" GITHUB_API = "https://api.github.com" log = logging.getLogger("orchestrator") +# ── file / state helpers ──────────────────────────────────────────────────── + def load_yaml(path: str) -> dict: with open(path) as f: return yaml.safe_load(f) @@ -76,6 +84,16 @@ def save_state(path: str, state: dict) -> None: os.replace(tmp, path) +def now_utc_iso() -> str: + return datetime.now(timezone.utc).isoformat() + + +def parse_iso(s: str) -> datetime: + return datetime.fromisoformat(s) + + +# ── GitHub API (unchanged from the OpenStack backend) ───────────────────────── + def gh_request(method: str, path: str, pat: str, **kwargs: Any) -> Any: url = f"{GITHUB_API}{path}" headers = kwargs.pop("headers", {}) @@ -156,267 +174,134 @@ def get_job_status(repo: str, job_id: str, pat: str) -> dict | None: return r.json() -def render_cloud_init(template_path: str, encoded_jit_config: str, - runner_version: str) -> str: - with open(template_path) as f: - tmpl = Template(f.read()) - return tmpl.render( - encoded_jit_config=encoded_jit_config, - runner_version=runner_version, - ) - - -def spawn_server( - conn: openstack.connection.Connection, - config: dict, - name: str, - job_id: str, - user_data: str, -) -> str: - flavor = conn.compute.find_flavor(config["flavor_name"], ignore_missing=False) - network = conn.network.find_network(config["network_name"], ignore_missing=False) - create_kwargs = dict( - name=name, - flavor_id=flavor.id, - networks=[{"uuid": network.id}], - key_name=config["keypair_name"], - security_groups=[{"name": config["security_group"]}], - user_data=base64.b64encode(user_data.encode()).decode(), - metadata={ - ROLE_META_KEY: ROLE_META_VAL, - JOB_META_KEY: job_id, - }, - ) - - # Flavors with disk=0 (typical for GPU flavors on this cloud) cannot boot - # directly from an image — Nova rejects with "Block Device Mapping is - # Invalid: You specified more local devices than the limit allows". When - # boot_volume_size_gb is set, boot from a Cinder volume sourced from the - # image and delete it on termination. Otherwise fall back to direct image - # boot (works only when the flavor has a non-zero root disk). - boot_volume_size_gb = int(config.get("boot_volume_size_gb") or 0) - if boot_volume_size_gb > 0: - create_kwargs["block_device_mapping"] = [ - { - "uuid": config["image_id"], - "source_type": "image", - "destination_type": "volume", - "boot_index": 0, - "volume_size": boot_volume_size_gb, - "delete_on_termination": True, - } - ] - else: - create_kwargs["image_id"] = config["image_id"] - - az = config.get("availability_zone") - if az: - create_kwargs["availability_zone"] = az - server = conn.compute.create_server(**create_kwargs) - return server.id - - -def delete_server(conn: openstack.connection.Connection, server_id: str) -> None: +# ── OSMO CLI output parsing ─────────────────────────────────────────────────── +# +# The exact JSON keys returned by `osmo workflow {submit,query,list}` can vary +# slightly by OSMO release, so these parsers try a set of likely keys and fall +# back to scraping the human-readable text output. Verify the keys against your +# deployed version with `osmo workflow submit --dry-run` / `--format-type json` +# once and simplify if desired. + +# Live AirLab OSMO 6.2.x returns workflow_uuid on list/query; submit may use +# workflow_id / id / name. Prefer uuid-like keys before "name" so we don't +# accidentally treat the human workflow name as the id when both are present. +_WF_ID_KEYS = ( + "workflow_uuid", "workflow_id", "workflowId", "id", "uuid", "name", "workflow", +) +_STATUS_KEYS = ("status", "state", "workflow_status", "phase") +_KNOWN_STATUSES = { + "RUNNING", "PENDING", "WAITING", "COMPLETED", "FAILED", + "FAILED_EXEC_TIMEOUT", "FAILED_SERVER_ERROR", "FAILED_QUEUE_TIMEOUT", + "FAILED_SUBMISSION", "FAILED_CANCELED", "FAILED_BACKEND_ERROR", + "FAILED_IMAGE_PULL", "FAILED_EVICTED", "FAILED_START_ERROR", + "FAILED_START_TIMEOUT", "FAILED_PREEMPTED", +} +# Non-terminal statuses the orphan sweep considers "still alive". +_ACTIVE_STATUSES = ("RUNNING", "PENDING", "WAITING") + + +def _loads_or_none(text: str | None) -> Any: try: - conn.compute.delete_server(server_id, ignore_missing=True, force=True) - except Exception as e: - log.warning("delete_server(%s) failed: %s", server_id, e) - - -def list_owned_servers(conn: openstack.connection.Connection) -> list[Any]: - """List all Nova servers that carry our role metadata.""" - owned = [] - for s in conn.compute.servers(details=True): - meta = getattr(s, "metadata", None) or {} - if meta.get(ROLE_META_KEY) == ROLE_META_VAL: - owned.append(s) - return owned - - -def find_free_floating_ip( - conn: openstack.connection.Connection, pool: list[str] -) -> Any: - """Return the FloatingIP resource for the first address in `pool` that is - not currently associated with any port. Returns None if all are in use. - - Iterates `pool` in order so attachments rotate through it sequentially. - Logs a warning for any pool member that doesn't exist in this project. - """ - if not pool: + return json.loads(text) # type: ignore[arg-type] + except (json.JSONDecodeError, TypeError): return None - pool_set = set(pool) - fips_by_addr: dict[str, Any] = {} - for fip in conn.network.ips(): - if fip.floating_ip_address in pool_set: - fips_by_addr[fip.floating_ip_address] = fip - missing = pool_set - fips_by_addr.keys() - if missing: - log.warning( - "floating_ips configured but not found in this project: %s", - sorted(missing), - ) - for addr in pool: - fip = fips_by_addr.get(addr) - if fip is not None and not fip.port_id: - return fip - return None - -def check_flavor_capacity( - conn: openstack.connection.Connection, - flavor_name: str, -) -> tuple[bool, str]: - """Pre-flight: ask Nova's placement API whether any host can satisfy this - flavor's resource request right now. - Returns (ok, reason). When ok=False the orchestrator should defer the - spawn iteration; reason is a one-line human-readable explanation - (e.g. "no host can satisfy {'VCPU': 8, 'MEMORY_MB': 32768, 'VGPU': 1}"). - - If the placement API can't be queried for any reason we return - (True, "") and let Nova make the call. The - pre-flight is a fast-path optimization, not a gate — Nova still has the - final say at create_server time (and ERROR-status fallback handles - anything we miss). - """ - try: - flavor = conn.compute.find_flavor(flavor_name, ignore_missing=False) - except Exception as e: - return True, f"flavor lookup failed: {e}" - - # Standard resources every Nova flavor expresses. - resources: dict[str, int] = {} - if getattr(flavor, "vcpus", 0): - resources["VCPU"] = int(flavor.vcpus) - if getattr(flavor, "ram", 0): - resources["MEMORY_MB"] = int(flavor.ram) - if getattr(flavor, "disk", 0): - resources["DISK_GB"] = int(flavor.disk) - - # Custom / specialized resources (VGPU, PCI_*, CUSTOM_*) come from the - # flavor's extra_specs as `resources:=`. This is how Nova - # itself learns to ask placement for GPU capacity. - extra = getattr(flavor, "extra_specs", {}) or {} - for k, v in extra.items(): - if not k.startswith("resources:"): - continue - rc = k.split(":", 1)[1] - try: - resources[rc] = int(v) - except (TypeError, ValueError): - pass - - if not resources: - return True, "flavor expresses no resources — skipping placement check" +def _first_str(d: dict, keys: tuple[str, ...]) -> str | None: + for k in keys: + v = d.get(k) + if isinstance(v, str) and v: + return v + return None - try: - result = conn.placement.allocation_candidates( - resources=resources, limit=1, - ) - if hasattr(result, "allocation_requests"): - candidates = list(result.allocation_requests or []) - else: - candidates = list(result) - except Exception as e: - return True, f"placement query failed ({type(e).__name__}: {e})" - - if candidates: - return True, "" - return False, f"no host can satisfy {resources}" - - -def wait_for_server_active( - conn: openstack.connection.Connection, - server_id: str, - timeout_s: int = 300, - poll_interval_s: float = 3.0, -) -> Any: - """Poll Nova until the server is ACTIVE. Raise with full context if it - enters ERROR or never reaches ACTIVE in time. - - Nova surfaces the actual reason for an ERROR via the `fault` attribute - (message + code + details), so we log it verbatim. We also include - task_state / vm_state / power_state because Nova sometimes leaves the - fault empty and these tell you whether the failure was at scheduling, - networking, or block-device-mapping time. - """ - deadline = time.monotonic() + timeout_s - last_status = "?" - last_task = None - while time.monotonic() < deadline: - s = conn.compute.get_server(server_id) - status = getattr(s, "status", "UNKNOWN") or "UNKNOWN" - task = ( - getattr(s, "task_state", None) - or getattr(s, "OS-EXT-STS:task_state", None) - ) - if status != last_status or task != last_task: - log.info( - "server %s status=%s task_state=%s", server_id, status, task, - ) - last_status, last_task = status, task - if status == "ACTIVE": +def _extract_workflow_id(stdout: str | None) -> str | None: + data = _loads_or_none(stdout) + if isinstance(data, dict): + wid = _first_str(data, _WF_ID_KEYS) + if wid: + return wid + wf = data.get("workflow") + if isinstance(wf, dict): + wid = _first_str(wf, _WF_ID_KEYS) + if wid: + return wid + m = re.search(r"Workflow\s*ID\s*[-:]\s*(\S+)", stdout or "", re.IGNORECASE) + return m.group(1) if m else None + + +def _extract_status(stdout: str | None) -> str | None: + data = _loads_or_none(stdout) + if isinstance(data, dict): + st = _first_str(data, _STATUS_KEYS) + if st: + return st.upper() + wf = data.get("workflow") + if isinstance(wf, dict): + st = _first_str(wf, _STATUS_KEYS) + if st: + return st.upper() + up = (stdout or "").upper() + for s in sorted(_KNOWN_STATUSES, key=len, reverse=True): + if s in up: return s + return None - if status == "ERROR": - fault = getattr(s, "fault", None) or {} - vm_state = ( - getattr(s, "vm_state", None) - or getattr(s, "OS-EXT-STS:vm_state", None) - ) - power_state = ( - getattr(s, "power_state", None) - or getattr(s, "OS-EXT-STS:power_state", None) - ) - host = getattr(s, "compute_host", None) or getattr( - s, "OS-EXT-SRV-ATTR:host", None - ) - az = getattr(s, "availability_zone", None) or getattr( - s, "OS-EXT-AZ:availability_zone", None - ) - raise RuntimeError( - "server " - + str(server_id) - + " entered ERROR: " - + f"fault.code={fault.get('code')!r} " - + f"fault.message={fault.get('message')!r} " - + f"fault.details={fault.get('details')!r} " - + f"task_state={task!r} vm_state={vm_state!r} " - + f"power_state={power_state!r} host={host!r} az={az!r}" - ) - time.sleep(poll_interval_s) +def _extract_workflow_list(stdout: str | None) -> list[dict]: + data = _loads_or_none(stdout) + if isinstance(data, dict): + for key in ("workflows", "items", "results", "data"): + if isinstance(data.get(key), list): + data = data[key] + break + items: list[dict] = [] + if isinstance(data, list): + for entry in data: + if not isinstance(entry, dict): + continue + wid = _first_str(entry, _WF_ID_KEYS) + name = entry.get("name") if isinstance(entry.get("name"), str) else None + status = _first_str(entry, _STATUS_KEYS) + if wid or name: + items.append( + {"id": wid, "name": name, + "status": status.upper() if status else None} + ) + return items + - raise RuntimeError( - f"server {server_id} did not reach ACTIVE within {timeout_s}s " - f"(last status={last_status!r} task_state={last_task!r})" - ) +def _is_terminal(status: str | None) -> bool: + if not status: + return False + return status == "COMPLETED" or status.startswith("FAILED") -def attach_floating_ip( - conn: openstack.connection.Connection, server_id: str, fip: Any -) -> str: - """Wait for the server to have a network port, then associate `fip`. - Returns the floating IP address.""" - for _ in range(60): # ~120s - ports = list(conn.network.ports(device_id=server_id)) - if ports: - break - time.sleep(2) - else: - raise RuntimeError(f"server {server_id} got no network port within 120s") - conn.network.update_ip(fip, port_id=ports[0].id) - return fip.floating_ip_address +def _looks_like_auth_error(r: subprocess.CompletedProcess) -> bool: + blob = f"{r.stdout or ''}\n{r.stderr or ''}".lower() + markers = ("401", "403", "unauthorized", "forbidden", "expired", + "not logged in", "please login", "authentication", + "invalid token", "token is invalid") + return any(m in blob for m in markers) -def now_utc_iso() -> str: - return datetime.now(timezone.utc).isoformat() +def _name_age_minutes(name: str | None) -> float | None: + """Age in minutes parsed from our `...-` name suffix, or None. + OSMO may append its own suffix after the name we submit, so we match the + first 10+ digit run (the unix timestamp) even when trailing chars follow. + """ + m = re.search(r"-(\d{10,})(?:\D.*)?$", name or "") + if not m: + return None + try: + ts = int(m.group(1)) + except ValueError: + return None + return (time.time() - ts) / 60.0 -def parse_iso(s: str) -> datetime: - return datetime.fromisoformat(s) +# ── orchestrator ────────────────────────────────────────────────────────────── class Orchestrator: def __init__(self, config: dict, pat: str, state_path: str, template_path: str): @@ -424,230 +309,308 @@ def __init__(self, config: dict, pat: str, state_path: str, template_path: str): self.pat = pat self.state_path = state_path self.template_path = template_path - self.conn = openstack.connect(cloud=config.get("openstack_cloud", "airstack")) + + # OSMO target. + self.osmo_bin = config.get("osmo_bin", "osmo") + self.osmo_url = config["osmo_url"] + self.token_file = config.get("osmo_token_file", DEFAULT_OSMO_TOKEN_PATH) + self.pool = config["pool"] + self.platform = config.get("platform", "") or "" + self.priority = str(config.get("priority", "NORMAL")).upper() + + # Runner task shape. + self.runner_image = config["runner_image"] + self.cpu = config.get("cpu", 8) + self.gpu = config.get("gpu", 1) + self.memory = config.get("memory", "32Gi") + self.storage = config.get("storage", "300Gi") + self.privileged = bool(config.get("privileged", True)) + self.host_network = bool(config.get("host_network", False)) + + # GitHub. self.repo = config["repo"] self.runner_labels = config["runner_labels"] - self.runner_version = config["runner_version"] + + # Limits / timing. self.max_concurrent = int(config.get("max_concurrent", 3)) - self.floating_ips: list[str] = list(config.get("floating_ips") or []) - # Cap spawns to FIP pool size so we never queue jobs we can't address. - self.effective_max_concurrent = self.max_concurrent - if self.floating_ips: - self.effective_max_concurrent = min( - self.max_concurrent, len(self.floating_ips) - ) - self.max_job_minutes = int(config.get("max_job_minutes", 90)) + self.max_job_minutes = int(config.get("max_job_minutes", 2880)) self.spawn_interval = int(config.get("spawn_poll_interval_s", 15)) self.reap_interval = int(config.get("reap_poll_interval_s", 30)) + self.submit_timeout = int(config.get("submit_timeout_s", 180)) + self.workflow_prefix = config.get("workflow_name_prefix", "gha-runner-") + self.stop_evt = threading.Event() + # Establish the OSMO session up-front for early feedback; individual + # commands re-login on demand if the session lapses. + self._login() + def stop(self, *_: Any) -> None: log.info("stop signal received; draining loops") self.stop_evt.set() + # ── OSMO CLI plumbing ──────────────────────────────────────────────────── + + def _run_osmo(self, args: list[str], timeout: int) -> subprocess.CompletedProcess: + return subprocess.run( + [self.osmo_bin, *args], + capture_output=True, text=True, timeout=timeout, + ) + + def _login(self) -> bool: + try: + r = self._run_osmo( + ["login", self.osmo_url, "--method", "token", + "--token-file", self.token_file], + timeout=60, + ) + except Exception as e: # noqa: BLE001 - startup best-effort + log.warning("osmo login raised: %s", e) + return False + if r.returncode != 0: + log.warning( + "osmo login failed (rc=%d): %s", + r.returncode, (r.stderr or r.stdout).strip(), + ) + return False + log.info("osmo login succeeded (url=%s, token_file=%s)", + self.osmo_url, self.token_file) + return True + + def _osmo(self, args: list[str], timeout: int, + relogin: bool = True) -> subprocess.CompletedProcess: + """Run an osmo CLI command, re-logging-in once on an auth failure.""" + r = self._run_osmo(args, timeout=timeout) + if r.returncode != 0 and relogin and _looks_like_auth_error(r): + log.info("osmo command hit an auth error; re-logging in and retrying") + if self._login(): + r = self._run_osmo(args, timeout=timeout) + return r + + def submit_workflow(self, workflow_file: str) -> tuple[str, str]: + """Submit a workflow. Returns (workflow_id, live_name). + + AirLab OSMO 6.2 submit JSON is typically only {name, overview, logs} + (no uuid), and the service may append a numeric suffix to the name + (e.g. ``...-1``). We immediately query to resolve uuid + live name so + state/reap stay consistent with ``workflow list`` (``workflow_uuid``). + """ + args = ["workflow", "submit", workflow_file, "--pool", self.pool, + "--priority", self.priority, "--format-type", "json"] + r = self._osmo(args, timeout=self.submit_timeout) + if r.returncode != 0: + raise RuntimeError( + f"osmo workflow submit failed (rc={r.returncode}): " + f"{(r.stderr or r.stdout).strip()}" + ) + submitted_name = _extract_workflow_id(r.stdout) or _extract_workflow_id(r.stderr) + if not submitted_name: + raise RuntimeError( + "could not parse workflow id/name from submit output: " + f"{(r.stdout or '').strip()[:500]}" + ) + live_name, uuid = submitted_name, None + q = self._osmo( + ["workflow", "query", submitted_name, "--format-type", "json"], + timeout=60, + ) + if q.returncode == 0: + data = _loads_or_none(q.stdout) or _loads_or_none(q.stderr) + if isinstance(data, dict): + if isinstance(data.get("name"), str) and data["name"]: + live_name = data["name"] + for k in ("uuid", "workflow_uuid", "workflow_id", "id"): + v = data.get(k) + if isinstance(v, str) and v: + uuid = v + break + return (uuid or live_name), live_name + + def query_status(self, workflow_id: str) -> str | None: + r = self._osmo( + ["workflow", "query", workflow_id, "--format-type", "json"], + timeout=60, + ) + if r.returncode != 0: + log.debug("osmo workflow query %s failed: %s", + workflow_id, (r.stderr or r.stdout).strip()) + return None + return _extract_status(r.stdout) or _extract_status(r.stderr) + + def cancel_workflow(self, workflow_id: str) -> None: + r = self._osmo( + ["workflow", "cancel", workflow_id, "--force", + "--message", "orchestrator reap", "--format-type", "json"], + timeout=60, + ) + if r.returncode != 0: + log.warning("osmo workflow cancel %s failed (rc=%d): %s", + workflow_id, r.returncode, (r.stderr or r.stdout).strip()) + + def list_runner_workflows(self) -> list[dict]: + """Active workflows (RUNNING/PENDING/WAITING) named with our prefix.""" + r = self._osmo( + ["workflow", "list", "--name", self.workflow_prefix, + "--pool", self.pool, "--count", "100", + "--status", *_ACTIVE_STATUSES, "--format-type", "json"], + timeout=60, + ) + if r.returncode != 0: + log.warning("osmo workflow list failed: %s", + (r.stderr or r.stdout).strip()) + return [] + return _extract_workflow_list(r.stdout) + + # ── workflow rendering ─────────────────────────────────────────────────── + + def render_workflow(self, workflow_name: str, encoded_jit_config: str) -> str: + with open(self.template_path) as f: + tmpl = Template(f.read()) + return tmpl.render( + workflow_name=workflow_name, + runner_image=self.runner_image, + cpu=self.cpu, + gpu=self.gpu, + memory=self.memory, + storage=self.storage, + platform=self.platform, + privileged="true" if self.privileged else "false", + host_network="true" if self.host_network else "false", + encoded_jit_config=encoded_jit_config, + runner_labels=self.runner_labels, + ) + + def _write_temp_workflow(self, name: str, content: str) -> str: + fd, path = tempfile.mkstemp(prefix=f"{name}-", suffix=".yaml") + with os.fdopen(fd, "w") as f: + f.write(content) + return path + + # ── loops ──────────────────────────────────────────────────────────────── + def spawn_once(self) -> None: state = load_state(self.state_path) active = len(state["jobs"]) - if active >= self.effective_max_concurrent: + if active >= self.max_concurrent: return try: queued = find_queued_jobs(self.repo, self.runner_labels, self.pat) - except Exception as e: + except Exception as e: # noqa: BLE001 log.warning("find_queued_jobs failed: %s", e) return - # Pre-flight capacity check via placement API. Every queued job uses - # the same flavor, so we check once per iteration. When OpenStack is - # out of GPUs / vCPU / RAM we defer the whole iteration — better than - # burning JIT tokens on creates that Nova will flip to ERROR. The - # next iteration retries automatically. - if queued: - ok, reason = check_flavor_capacity( - self.conn, self.config["flavor_name"] - ) - if not ok: - log.warning( - "deferring spawn — OpenStack capacity unavailable: %s. " - "Will retry in %ds.", - reason, self.spawn_interval, - ) - return - elif reason: - # Soft-skip path: placement check couldn't run (e.g. older - # Nova). Surface why so it's debuggable, then proceed. - log.debug("placement pre-flight: %s", reason) - for job in queued: - if active >= self.effective_max_concurrent: + if active >= self.max_concurrent: break job_id = job["job_id"] if job_id in state["jobs"]: continue - # Pre-check FIP availability before minting a JIT token so we - # don't burn one when there's nowhere to attach the worker. - reserved_fip = None - if self.floating_ips: - reserved_fip = find_free_floating_ip(self.conn, self.floating_ips) - if reserved_fip is None: - log.warning( - "no free floating IP in pool (%d configured); " - "deferring spawns until one frees up", - len(self.floating_ips), - ) - break - ts = int(time.time()) - runner_name = f"ephemeral-{job_id}-{ts}" - server_id: str | None = None + # OSMO workflow name doubles as the JIT runner registration name. + workflow_name = f"{self.workflow_prefix}{job_id}-{ts}" + tmp_path: str | None = None try: jit = mint_jit_config( - self.repo, runner_name, self.runner_labels, self.pat + self.repo, workflow_name, self.runner_labels, self.pat ) - user_data = render_cloud_init( - self.template_path, jit, self.runner_version - ) - server_id = spawn_server( - self.conn, self.config, runner_name, job_id, user_data - ) - # Don't move on until Nova reports ACTIVE. If it transitions - # to ERROR, this raises with the Nova fault details so the - # operator can see *why* the spawn failed (quota, scheduling, - # block-device-mapping, networking, etc.). - wait_for_server_active( - self.conn, - server_id, - timeout_s=int(self.config.get("server_active_timeout_s", 300)), - ) - except Exception as e: - # Tag capacity-related Nova faults so log-grepping for - # "capacity unavailable" finds both the pre-flight defer and - # the post-create fallback (e.g. PCI passthrough that - # placement doesn't track). - msg = str(e).lower() - capacity_markers = ( - "no valid host", - "insufficient", - "quotaexceeded", - "out of resource", - "no host can satisfy", - "no allocation candidates", - ) - if any(m in msg for m in capacity_markers): - log.warning( - "spawn failed for job %s — OpenStack capacity " - "unavailable (post-create): %s. Will retry in %ds.", - job_id, e, self.spawn_interval, - ) - else: - log.exception("spawn failed for job %s: %s", job_id, e) - if server_id: - log.warning( - "deleting failed server %s to release its volume / FIP", - server_id, - ) - delete_server(self.conn, server_id) + workflow_yaml = self.render_workflow(workflow_name, jit) + tmp_path = self._write_temp_workflow(workflow_name, workflow_yaml) + workflow_id, live_name = self.submit_workflow(tmp_path) + except Exception as e: # noqa: BLE001 + log.exception("submit failed for job %s: %s", job_id, e) continue - - floating_ip_addr: str | None = None - if reserved_fip is not None: - try: - floating_ip_addr = attach_floating_ip( - self.conn, server_id, reserved_fip - ) - log.info( - "attached floating IP %s to server %s (job %s)", - floating_ip_addr, server_id, job_id, - ) - except Exception as e: - log.exception( - "FIP attach failed for server %s; deleting to avoid " - "leaking a worker without external access: %s", - server_id, e, - ) - delete_server(self.conn, server_id) - continue + finally: + if tmp_path: + try: + os.remove(tmp_path) + except OSError: + pass state["jobs"][job_id] = { "run_id": job["run_id"], - "server_id": server_id, - "runner_name": runner_name, - "spawned_at": now_utc_iso(), + "workflow_id": workflow_id, + "workflow_name": live_name, + "runner_name": workflow_name, + "submitted_at": now_utc_iso(), "name": job["name"], - "floating_ip": floating_ip_addr, } save_state(self.state_path, state) active += 1 log.info( - "spawned server %s for job %s (%s)", server_id, job_id, job["name"] + "submitted workflow %s for job %s (%s)", + workflow_id, job_id, job["name"], ) def reap_once(self) -> None: state = load_state(self.state_path) now = datetime.now(timezone.utc) - # 1. Delete servers for completed jobs. + # 1. Cancel workflows for completed / purged jobs. for job_id in list(state["jobs"].keys()): entry = state["jobs"][job_id] + wid = entry["workflow_id"] try: job = get_job_status(self.repo, job_id, self.pat) - except Exception as e: + except Exception as e: # noqa: BLE001 log.warning("get_job_status(%s) failed: %s", job_id, e) continue + if job is None or job.get("status") == "completed": - log.info("reaping server %s (job %s done)", entry["server_id"], job_id) - delete_server(self.conn, entry["server_id"]) + # The runner usually exits on its own (task self-completes and + # the pod is torn down); only cancel if it's somehow still live. + status = self.query_status(wid) + if not _is_terminal(status): + log.info("reaping workflow %s (job %s done, wf status=%s)", + wid, job_id, status) + self.cancel_workflow(wid) + else: + log.info("workflow %s already terminal (%s) for job %s", + wid, status, job_id) del state["jobs"][job_id] continue # 2. Force-reap stragglers older than max_job_minutes. - spawned = parse_iso(entry["spawned_at"]) - age_min = (now - spawned).total_seconds() / 60.0 + age_min = (now - parse_iso(entry["submitted_at"])).total_seconds() / 60.0 if age_min > self.max_job_minutes: log.warning( - "force-reaping server %s (job %s age %.1fm > %dm)", - entry["server_id"], job_id, age_min, self.max_job_minutes, + "force-reaping workflow %s (job %s age %.1fm > %dm)", + wid, job_id, age_min, self.max_job_minutes, ) - delete_server(self.conn, entry["server_id"]) + self.cancel_workflow(wid) del state["jobs"][job_id] save_state(self.state_path, state) - # 3. Orphan sweep: any server we own that isn't in state and isn't - # in the brief just-spawned window. Catches state.json wipes and - # crashes between spawn and save_state. + # 3. Orphan sweep: our-named workflows still active but absent from + # state (catches state.json wipes and crashes between submit and + # save_state). Skip very fresh ones so we don't race our own submit. try: - owned = list_owned_servers(self.conn) - except Exception as e: - log.warning("list_owned_servers failed: %s", e) + listed = self.list_runner_workflows() + except Exception as e: # noqa: BLE001 + log.warning("list_runner_workflows failed: %s", e) return - tracked_ids = {e["server_id"] for e in state["jobs"].values()} - for s in owned: - if s.id in tracked_ids: + tracked_ids = {e["workflow_id"] for e in state["jobs"].values()} + tracked_names = {e["workflow_name"] for e in state["jobs"].values()} + for wf in listed: + wid, wname = wf.get("id"), wf.get("name") + if (wid and wid in tracked_ids) or (wname and wname in tracked_names): continue - created = getattr(s, "created_at", None) - if created: - try: - age_min = (now - parse_iso(created.replace("Z", "+00:00"))).total_seconds() / 60.0 - except Exception: - age_min = self.max_job_minutes + 1 - else: - age_min = self.max_job_minutes + 1 - # Only reap orphans that have lived past one spawn interval - # (to avoid racing our own freshly-created server). - if age_min < 2: + age = _name_age_minutes(wname) + if age is not None and age < 2: continue - log.warning( - "orphan-reaping server %s (not in state, age %.1fm)", s.id, age_min - ) - delete_server(self.conn, s.id) + target = wid or wname + if not target: + continue + log.warning("orphan-reaping workflow %s (not in state)", target) + self.cancel_workflow(target) def run(self) -> None: log.info( - "orchestrator started: repo=%s labels=%s max_concurrent=%d " - "(effective=%d, floating_ip_pool=%d)", - self.repo, self.runner_labels, self.max_concurrent, - self.effective_max_concurrent, len(self.floating_ips), + "orchestrator started (OSMO backend): repo=%s labels=%s pool=%s " + "platform=%s max_concurrent=%d", + self.repo, self.runner_labels, self.pool, + self.platform or "(pool default)", self.max_concurrent, ) last_spawn = 0.0 last_reap = 0.0 @@ -656,13 +619,13 @@ def run(self) -> None: if now - last_spawn >= self.spawn_interval: try: self.spawn_once() - except Exception: + except Exception: # noqa: BLE001 log.exception("spawn loop iteration failed") last_spawn = now if now - last_reap >= self.reap_interval: try: self.reap_once() - except Exception: + except Exception: # noqa: BLE001 log.exception("reap loop iteration failed") last_reap = now self.stop_evt.wait(timeout=1.0) diff --git a/.github/orchestrator/requirements.txt b/.github/orchestrator/requirements.txt index b69702b59..f710f7167 100644 --- a/.github/orchestrator/requirements.txt +++ b/.github/orchestrator/requirements.txt @@ -1,4 +1,3 @@ -openstacksdk>=3.0,<5 requests>=2.31 PyYAML>=6.0 Jinja2>=3.1 diff --git a/.github/orchestrator/runner-entrypoint.sh b/.github/orchestrator/runner-entrypoint.sh new file mode 100644 index 000000000..3026dbdf4 --- /dev/null +++ b/.github/orchestrator/runner-entrypoint.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# Entry point for the AirStack CI ephemeral-runner container (an OSMO task). +# +# Starts an inner Docker daemon — the AirStack test harness runs `airstack up` +# (docker compose) inside this container — waits for it, then runs exactly ONE +# ephemeral GitHub Actions job via the single-use JIT config. When run.sh exits, +# the OSMO task completes and the pod is destroyed: one job per pod, same as the +# old OpenStack VM. +# +# Requires a privileged pod (dockerd) scheduled on a GPU platform; the NVIDIA +# container toolkit (baked into the image) lets the inner dockerd pass the node +# GPU through to the compose containers. +set -euo pipefail + +: "${ENCODED_JIT_CONFIG:?ENCODED_JIT_CONFIG must be set by the workflow}" + +log() { echo "[runner-entrypoint] $*"; } + +# --------------------------------------------------------------------------- +# Docker storage backend +# +# The pod's root filesystem is overlayfs, and Linux refuses to use a directory +# on overlayfs as an overlay `upperdir` (EINVAL). A dockerd whose data-root sits +# on the pod rootfs still *pulls* images fine — containerd unpacks layers with +# plain writes — but every build step that needs a real mount dies with: +# +# failed to solve: ... mount source: "overlay", +# target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/...", +# err: invalid argument +# +# which is what took out all of build_docker / build_packages. So put the +# data-root somewhere overlay actually works, and verify it by performing a real +# overlay mount rather than trusting the filesystem type. +# --------------------------------------------------------------------------- + +DOCKER_DATA_ROOT=/var/lib/docker +LOOP_IMG=/docker-data.img +# Headroom left for everything that is not the Docker data-root (the runner's +# _work checkout, logs, the loop image's own metadata). +LOOP_HEADROOM_MB=20480 +MIN_BACKING_MB=51200 # Below ~50 GiB the sim images can't fit regardless. + +# True if an overlay mount whose upperdir lives under $1 can actually be made. +overlay_upperdir_works() { + local base=$1 probe rc=1 + probe=$(mktemp -d "$base/.overlay-probe.XXXXXX" 2>/dev/null) || return 1 + mkdir -p "$probe"/{lower,upper,work,merged} + if mount -t overlay overlay \ + -o "lowerdir=$probe/lower,upperdir=$probe/upper,workdir=$probe/work" \ + "$probe/merged" 2>/dev/null; then + umount "$probe/merged" && rc=0 + fi + rm -rf "$probe" + return $rc +} + +free_mb() { df -Pm "$1" | awk 'NR==2 {print $4}'; } + +# Preferred: a loopback ext4 image mounted at the data-root. Self-contained +# (no dependency on how the pool exposes storage), gives real overlay2, and dies +# with the pod. The image file is sparse, so it only consumes what Docker writes. +setup_loopback() { + local size_mb=${DOCKER_LOOP_SIZE_MB:-} + if [[ -z "$size_mb" ]]; then + size_mb=$(( $(free_mb /) - LOOP_HEADROOM_MB )) + fi + if (( size_mb < MIN_BACKING_MB )); then + log "loopback: only ${size_mb}MB usable, need ${MIN_BACKING_MB}MB — skipping" + return 1 + fi + + # /dev/loop-control only exists once the loop module is loaded on the node. + [[ -e /dev/loop-control ]] || modprobe loop 2>/dev/null || true + if [[ ! -e /dev/loop-control ]]; then + log "loopback: no /dev/loop-control — skipping" + return 1 + fi + + log "loopback: creating ${size_mb}MB ext4 image at $LOOP_IMG" + truncate -s "${size_mb}M" "$LOOP_IMG" || return 1 + # No journal and lazy inode-table init: this filesystem never outlives the + # pod, so durability buys nothing and mkfs stays fast. + mkfs.ext4 -q -F -m 0 -O ^has_journal -E lazy_itable_init=1 "$LOOP_IMG" || return 1 + + mkdir -p "$DOCKER_DATA_ROOT" + mount -o loop "$LOOP_IMG" "$DOCKER_DATA_ROOT" || return 1 + + if ! overlay_upperdir_works "$DOCKER_DATA_ROOT"; then + log "loopback: mounted but overlay still rejected — unwinding" + umount "$DOCKER_DATA_ROOT" || true + rm -f "$LOOP_IMG" + return 1 + fi + return 0 +} + +# Fallback: a real filesystem already mounted into the pod. Kubernetes emptyDir, +# hostPath and PVC volumes are backed by the node disk rather than the overlay +# rootfs, so overlay works there. +setup_real_fs() { + local best="" best_free=0 mnt fstype opts avail + while read -r _ mnt fstype opts _; do + case "$fstype" in ext2|ext3|ext4|xfs|btrfs) ;; *) continue ;; esac + [[ -d "$mnt" && -w "$mnt" ]] || continue + [[ ",$opts," == *",ro,"* ]] && continue + # OSMO's ctrl sidecar owns these: /osmo/data/output is uploaded as job + # artifacts and the socket dir is its IPC channel. + case "$mnt" in /osmo/data/output*|/osmo/data/socket*) continue ;; esac + avail=$(free_mb "$mnt") + if (( avail > best_free )); then best_free=$avail; best=$mnt; fi + done < /proc/mounts + + if [[ -z "$best" ]] || (( best_free < MIN_BACKING_MB )); then + log "real-fs: no mounted filesystem with >=${MIN_BACKING_MB}MB free — skipping" + return 1 + fi + + DOCKER_DATA_ROOT="$best/airstack-docker-data" + mkdir -p "$DOCKER_DATA_ROOT" + if ! overlay_upperdir_works "$DOCKER_DATA_ROOT"; then + log "real-fs: overlay rejected under $best — skipping" + DOCKER_DATA_ROOT=/var/lib/docker + return 1 + fi + log "real-fs: using $DOCKER_DATA_ROOT (${best_free}MB free)" + return 0 +} + +mkdir -p /etc/docker +if setup_loopback; then + log "storage: overlay2 on a loopback ext4 image" + printf '{"data-root": "%s"}\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json +elif setup_real_fs; then + log "storage: overlay2 on $DOCKER_DATA_ROOT" + printf '{"data-root": "%s"}\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json +elif [[ -e /dev/fuse ]] && command -v fuse-overlayfs >/dev/null 2>&1; then + # fuse-overlayfs stacks on overlayfs where the kernel driver won't. Slower + # than overlay2 but nowhere near as bad as vfs. `storage-driver` only applies + # to the classic image store, so the containerd snapshotter has to go. + log "storage: fuse-overlayfs (no overlay-capable filesystem found)" + cat > /etc/docker/daemon.json <<'JSON' +{"storage-driver": "fuse-overlayfs", "features": {"containerd-snapshotter": false}} +JSON +else + # Always works, but copies the whole filesystem per layer. The sim images are + # large enough that this will likely exhaust the pod's storage request. + log "WARN: storage: falling back to vfs — builds will be slow and may run out of disk" + cat > /etc/docker/daemon.json <<'JSON' +{"storage-driver": "vfs", "features": {"containerd-snapshotter": false}} +JSON +fi + +# Start dockerd in the background (needs privileged). +dockerd >/var/log/dockerd.log 2>&1 & + +# Wait for the daemon to accept connections (~60s budget). +for _ in $(seq 1 60); do + if docker info >/dev/null 2>&1; then + break + fi + sleep 1 +done +if ! docker info >/dev/null 2>&1; then + echo "ERROR: dockerd did not become ready" >&2 + cat /var/log/dockerd.log >&2 || true + exit 1 +fi + +log "storage driver: $(docker info --format '{{.Driver}}' 2>/dev/null || echo unknown)" \ + "data-root: $(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo unknown)" + +# Non-fatal GPU sanity check — surfaces GPU/privileged/toolkit misconfig early. +nvidia-smi || echo "WARN: nvidia-smi unavailable (check GPU + privileged + toolkit)" + +cd /home/runner/actions-runner +# The JIT config makes this runner single-use + ephemeral; run.sh returns after +# one job, which completes the task and lets OSMO reap the pod. +exec ./run.sh --jitconfig "${ENCODED_JIT_CONFIG}" diff --git a/.github/orchestrator/runner-workflow.yaml.j2 b/.github/orchestrator/runner-workflow.yaml.j2 new file mode 100644 index 000000000..124210ddb --- /dev/null +++ b/.github/orchestrator/runner-workflow.yaml.j2 @@ -0,0 +1,48 @@ +# OSMO workflow rendered per-job by orchestrator.py (replaces the old +# cloud-init.yaml.j2). One workflow == one ephemeral GitHub Actions runner == +# one CI job. Jinja variables injected by the orchestrator: +# +# workflow_name unique name (gha-runner--); also the +# JIT runner registration name +# runner_image prebaked image (docker-ce + compose + nvidia-container- +# toolkit + GH Actions runner) — see runner.Dockerfile +# cpu / gpu / memory / storage resource request for the runner task +# platform optional OSMO platform to target within the pool +# (omitted -> the pool's default platform) +# privileged "true"/"false"; MUST be "true" because the AirStack +# test harness runs `airstack up` (docker compose) inside +# the pod, which needs an inner Docker daemon. Requires a +# platform with "Privileged Mode Allowed" (ask your OSMO +# admin to enable it for the CI pool). +# host_network "true"/"false" +# encoded_jit_config single-use base64 GitHub JIT runner config +# +# The pool is passed by the orchestrator via `osmo workflow submit --pool`, so +# it is intentionally not hard-coded here. +# +# Lifecycle: the container starts dockerd, then runs exactly ONE ephemeral job +# via the JIT config. When run.sh exits, the task completes and OSMO tears the +# pod down — same "destroy after one job" behavior the OpenStack VM had. +workflow: + name: {{ workflow_name }} + resources: + runner: + cpu: {{ cpu }} + gpu: {{ gpu }} + memory: {{ memory }} + storage: {{ storage }} +{% if platform %} platform: {{ platform }} +{% endif %} + tasks: + - name: runner + image: {{ runner_image }} + resource: runner + privileged: {{ privileged }} +{% if host_network == "true" %} hostNetwork: true +{% endif %} + environment: + # Single-use + ephemeral: the runner exits after exactly one job. + ENCODED_JIT_CONFIG: "{{ encoded_jit_config }}" + # The runner refuses to run as root without this; the DinD image is root. + RUNNER_ALLOW_RUNASROOT: "1" + command: ["/usr/local/bin/run-ephemeral-runner.sh"] diff --git a/.github/orchestrator/runner.Dockerfile b/.github/orchestrator/runner.Dockerfile new file mode 100644 index 000000000..18bcf8000 --- /dev/null +++ b/.github/orchestrator/runner.Dockerfile @@ -0,0 +1,67 @@ +# Prebaked image for AirStack CI ephemeral runners on OSMO. +# +# This bakes in what the old cloud-init.yaml.j2 installed on the OpenStack VM +# (Docker CE + compose plugin, NVIDIA container toolkit, the GitHub Actions +# runner) so pod start is fast and the single-use JIT token can't expire during +# a slow apt/bootstrap. Build it and push to a registry your OSMO pool can pull: +# +# docker build -f runner.Dockerfile \ +# --build-arg RUNNER_VERSION=2.336.0 \ +# -t /airstack-ci-runner:2.336.0 . +# docker push /airstack-ci-runner:2.336.0 +# +# Then set `runner_image: /airstack-ci-runner:2.336.0` in config.yaml. +# Keep RUNNER_VERSION in sync with the actions/runner release you want. +# +# GPU-in-Docker-in-Docker: the OSMO task must run privileged (see +# `privileged: true` in runner-workflow.yaml.j2) on a platform with +# "Privileged Mode Allowed" + GPUs. The inner dockerd uses the NVIDIA container +# toolkit installed here to expose the node GPU to the `airstack up` containers. +# The image is linux/amd64 (x86_64 runner tarball); rebuild for arm64 if needed. +FROM ubuntu:24.04 + +ENV DEBIAN_FRONTEND=noninteractive + +# Docker CE (+ compose/buildx plugins), NVIDIA container toolkit, and the tools +# the AirStack test harness / GH runner need (git, jq, python venv, ...). +# e2fsprogs + fuse-overlayfs back the storage-backend selection in +# runner-entrypoint.sh: the pod rootfs is overlayfs, which the kernel rejects as +# an overlay upperdir, so dockerd's data-root has to live on a loopback ext4 +# image (mkfs.ext4) or, failing that, use the fuse-overlayfs driver. +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates curl gnupg jq git sudo iproute2 \ + e2fsprogs fuse-overlayfs kmod mount \ + python3 python3-venv python3-pip \ + && install -m 0755 -d /etc/apt/keyrings \ + && curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ + -o /etc/apt/keyrings/docker.asc \ + && chmod a+r /etc/apt/keyrings/docker.asc \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \ +https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \ + > /etc/apt/sources.list.d/docker.list \ + && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ + && curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + > /etc/apt/sources.list.d/nvidia-container-toolkit.list \ + && apt-get update && apt-get install -y --no-install-recommends \ + docker-ce docker-ce-cli containerd.io \ + docker-buildx-plugin docker-compose-plugin \ + nvidia-container-toolkit \ + && nvidia-ctk runtime configure --runtime=docker \ + && rm -rf /var/lib/apt/lists/* + +# GitHub Actions runner (self-contained; version pinned at build time). +ARG RUNNER_VERSION=2.336.0 +RUN mkdir -p /home/runner/actions-runner \ + && cd /home/runner/actions-runner \ + && curl -fsSL -o runner.tar.gz \ + "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" \ + && tar xzf runner.tar.gz \ + && rm runner.tar.gz \ + && ./bin/installdependencies.sh + +COPY runner-entrypoint.sh /usr/local/bin/run-ephemeral-runner.sh +RUN chmod +x /usr/local/bin/run-ephemeral-runner.sh + +WORKDIR /home/runner/actions-runner diff --git a/.github/orchestrator/setup.sh b/.github/orchestrator/setup.sh index 4803b4a33..adb3c078f 100755 --- a/.github/orchestrator/setup.sh +++ b/.github/orchestrator/setup.sh @@ -1,13 +1,17 @@ #!/usr/bin/env bash -# One-time orchestrator-VM setup. Run as root on the airstack-ci-cd-orchestrator -# OpenStack instance after cloning the repo. +# One-time orchestrator-VM setup (OSMO backend). Run as root on the +# airstack-ci-cd-orchestrator instance after cloning the repo. # # Pre-reqs (do these *before* running this script): -# 1. ~/.config/openstack/clouds.yaml staged for the orchestrator user -# (application credential — see .github/orchestrator/README.md). -# 2. /tmp/github-pat exists with the GitHub PAT contents. +# 1. /tmp/github-pat exists with the GitHub PAT contents. +# 2. /tmp/osmo-token exists with the OSMO service-account access token +# (from `osmo token set` — see .github/orchestrator/README.md). Optional +# at setup time; you can stage it later before starting the service. # 3. This repo cloned somewhere readable (this script copies code out of # its containing directory). +# +# The orchestrator host is lightweight and needs NO GPU — it only polls GitHub +# and submits OSMO workflows. 1 vCPU / 2GB RAM / 20GB disk is plenty. set -euo pipefail @@ -30,18 +34,33 @@ fi echo "==> Installing system packages" apt-get update -apt-get install -y python3 python3-venv python3-pip +apt-get install -y python3 python3-venv python3-pip curl ca-certificates + +echo "==> Installing the OSMO CLI" +if command -v osmo >/dev/null 2>&1; then + echo " osmo already installed ($(command -v osmo)); skipping" +else + # Latest client. Pin to a release from https://github.com/NVIDIA/OSMO/releases + # if you need a specific version. + curl -fsSL https://raw.githubusercontent.com/NVIDIA/OSMO/refs/heads/main/install.sh | bash + command -v osmo >/dev/null 2>&1 \ + || echo "WARNING: osmo not on PATH after install — check the installer output" >&2 +fi echo "==> Creating directories" install -d -o "$USER_NAME" -g "$USER_NAME" -m 0750 "$INSTALL_DIR" install -d -o root -g "$USER_NAME" -m 0750 "$CONFIG_DIR" install -d -o "$USER_NAME" -g "$USER_NAME" -m 0750 "$STATE_DIR" +# XDG dirs for the osmo CLI login session (see the systemd unit's HOME/XDG env). +install -d -o "$USER_NAME" -g "$USER_NAME" -m 0700 "$STATE_DIR/.config" +install -d -o "$USER_NAME" -g "$USER_NAME" -m 0700 "$STATE_DIR/.cache" +install -d -o "$USER_NAME" -g "$USER_NAME" -m 0700 "$STATE_DIR/.state" echo "==> Copying orchestrator files to $INSTALL_DIR" install -o "$USER_NAME" -g "$USER_NAME" -m 0755 \ "$REPO_DIR/orchestrator.py" "$INSTALL_DIR/orchestrator.py" install -o "$USER_NAME" -g "$USER_NAME" -m 0644 \ - "$REPO_DIR/cloud-init.yaml.j2" "$INSTALL_DIR/cloud-init.yaml.j2" + "$REPO_DIR/runner-workflow.yaml.j2" "$INSTALL_DIR/runner-workflow.yaml.j2" echo "==> Building Python venv" sudo -u "$USER_NAME" python3 -m venv "$INSTALL_DIR/venv" @@ -63,11 +82,14 @@ fi install -o root -g "$USER_NAME" -m 0640 /tmp/github-pat "$CONFIG_DIR/github-pat" shred -u /tmp/github-pat -echo "==> Verifying clouds.yaml" -CLOUDS_YAML="/home/$USER_NAME/.config/openstack/clouds.yaml" -if [[ ! -f "$CLOUDS_YAML" ]]; then - echo "WARNING: $CLOUDS_YAML missing." >&2 - echo " Create it (application credential) before starting the service." >&2 +echo "==> Installing OSMO service-account token (from /tmp/osmo-token)" +if [[ -f /tmp/osmo-token ]]; then + install -o root -g "$USER_NAME" -m 0640 /tmp/osmo-token "$CONFIG_DIR/osmo-token" + shred -u /tmp/osmo-token +else + echo "WARNING: /tmp/osmo-token not found." >&2 + echo " Stage the OSMO service-account token before starting the service:" >&2 + echo " sudo install -o root -g $USER_NAME -m 0640 /tmp/osmo-token $CONFIG_DIR/osmo-token" >&2 fi echo "==> Installing systemd unit" @@ -78,7 +100,8 @@ systemctl daemon-reload echo echo "Setup complete. Next steps:" -echo " 1. Edit $CONFIG_DIR/config.yaml — fill flavor/network/keypair/security_group." -echo " 2. Verify $CLOUDS_YAML exists with the application credential." +echo " 1. Edit $CONFIG_DIR/config.yaml — set osmo_url, pool, platform," +echo " runner_image, and resources." +echo " 2. Ensure $CONFIG_DIR/osmo-token holds the OSMO service-account token." echo " 3. systemctl enable --now airstack-orchestrator.service" echo " 4. journalctl -u airstack-orchestrator.service -f" diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index add5f5953..403864781 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -12,6 +12,11 @@ on: required: false default: 'desktop,isaac-sim,ms-airsim' type: string + force_rebuild: + description: 'Force a full rebuild of every service (skip retag)' + required: false + default: false + type: boolean env: DEFAULT_PROFILES: 'desktop,isaac-sim,ms-airsim' @@ -21,6 +26,8 @@ jobs: runs-on: ubuntu-latest outputs: tag-changed: ${{ steps.check-changes.outputs.tag-changed }} + current-version: ${{ steps.check-changes.outputs.current-version }} + previous-version: ${{ steps.check-changes.outputs.previous-version }} steps: - name: Checkout code uses: actions/checkout@v4 @@ -33,20 +40,22 @@ jobs: run: | # Get the current VERSION value CURRENT_TAG=$(grep "^VERSION=" .env | cut -d '=' -f2- | tr -d '"' | tr -d "'") - + # Get the previous VERSION value git show HEAD~1:.env > .env.prev 2>/dev/null || echo "" > .env.prev PREVIOUS_TAG=$(grep "^VERSION=" .env.prev | cut -d '=' -f2- | tr -d '"' | tr -d "'" || echo "") - + echo "Current tag: $CURRENT_TAG" echo "Previous tag: $PREVIOUS_TAG" - + echo "current-version=$CURRENT_TAG" >> "$GITHUB_OUTPUT" + echo "previous-version=$PREVIOUS_TAG" >> "$GITHUB_OUTPUT" + if [ "$CURRENT_TAG" != "$PREVIOUS_TAG" ] && [ -n "$CURRENT_TAG" ]; then echo "VERSION has changed from '$PREVIOUS_TAG' to '$CURRENT_TAG'" - echo "tag-changed=true" >> $GITHUB_OUTPUT + echo "tag-changed=true" >> "$GITHUB_OUTPUT" else echo "VERSION has not changed" - echo "tag-changed=false" >> $GITHUB_OUTPUT + echo "tag-changed=false" >> "$GITHUB_OUTPUT" fi docker-build: @@ -78,7 +87,6 @@ jobs: - name: Verify .env file and extract tag run: | - # Ensure .env file exists and is readable if [ ! -f .env ]; then echo "Error: .env file not found" exit 1 @@ -89,68 +97,184 @@ jobs: # Some compose files expect this file to exist, even if it is empty. mkdir -p simulation/isaac-sim/docker : > simulation/isaac-sim/docker/omni_pass.env - - # Display the current VERSION for debugging + DOCKER_TAG=$(grep "^VERSION=" .env | cut -d '=' -f2- | tr -d '"' | tr -d "'") echo "Building with VERSION: $DOCKER_TAG" - + if [ -z "$DOCKER_TAG" ]; then echo "Error: VERSION is empty" exit 1 fi - - name: Run Docker Compose Build + - name: Resolve compose profiles and previous VERSION + id: prep run: | - # Load environment variables and run docker compose build - set -a # Export all variables + set -a source .env - set +a # Stop exporting - - # Always override COMPOSE_PROFILES for all trigger types + set +a + if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" else export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" fi - - docker compose build - - name: Run Docker Compose Push + PREVIOUS_VERSION="${{ needs.check-docker-tag-change.outputs.previous-version }}" + CURRENT_VERSION="${{ needs.check-docker-tag-change.outputs.current-version }}" + if [ -z "$CURRENT_VERSION" ]; then + CURRENT_VERSION=$(grep "^VERSION=" .env | cut -d '=' -f2- | tr -d '"' | tr -d "'") + fi + + FORCE_REBUILD=false + if [ "${{ github.event_name }}" == "workflow_dispatch" ] && [ "${{ github.event.inputs.force_rebuild }}" == "true" ]; then + FORCE_REBUILD=true + fi + + echo "COMPOSE_PROFILES=$COMPOSE_PROFILES" >> "$GITHUB_ENV" + echo "PREVIOUS_VERSION=$PREVIOUS_VERSION" >> "$GITHUB_ENV" + echo "CURRENT_VERSION=$CURRENT_VERSION" >> "$GITHUB_ENV" + echo "FORCE_REBUILD=$FORCE_REBUILD" >> "$GITHUB_ENV" + echo "profiles=$COMPOSE_PROFILES" >> "$GITHUB_OUTPUT" + echo "previous-version=$PREVIOUS_VERSION" >> "$GITHUB_OUTPUT" + echo "force-rebuild=$FORCE_REBUILD" >> "$GITHUB_OUTPUT" + + # Content-aware publish: retag previous versioned images when image inputs + # are unchanged; rebuild only services whose fingerprint differs (or when + # force_rebuild / unlabeled previous images force a cold build). + - name: Plan retag vs rebuild + id: plan run: | - # Load environment variables and run docker compose push - set -a # Export all variables + set -a source .env - set +a # Stop exporting - - # Always override COMPOSE_PROFILES for all trigger types - if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" - else - export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" + set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" + + FORCE_ARGS=() + if [ "${{ env.FORCE_REBUILD }}" = "true" ]; then + FORCE_ARGS+=(--force-rebuild) fi - - docker compose push - - name: Sign pushed images with Cosign (keyless) - env: - COSIGN_YES: "true" + python3 .github/workflows/scripts/docker_image_plan.py \ + --version "${{ env.CURRENT_VERSION }}" \ + --previous-version "${{ env.PREVIOUS_VERSION }}" \ + --profiles "${{ env.COMPOSE_PROFILES }}" \ + --plan-out docker-image-plan.json \ + --override-out docker-compose.fingerprint.yaml \ + "${FORCE_ARGS[@]}" + + echo "Plan summary:" + jq -r '.services | to_entries[] | " \(.key): \(.value.action) (\(.value.reason))"' docker-image-plan.json + + - name: Retag unchanged images + run: | + set -euo pipefail + RETAG_COUNT=$(jq '[.services[] | select(.action=="retag")] | length' docker-image-plan.json) + echo "Services to retag: $RETAG_COUNT" + if [ "$RETAG_COUNT" -eq 0 ]; then + echo "Nothing to retag." + exit 0 + fi + + jq -c '.services | to_entries[] | select(.value.action=="retag") | .value' docker-image-plan.json \ + | while IFS= read -r row; do + IMAGE=$(echo "$row" | jq -r '.image') + PREV=$(echo "$row" | jq -r '.previous_image') + CACHE=$(echo "$row" | jq -r '.cache_tag // empty') + echo "Retagging $PREV → $IMAGE" + CREATE_ARGS=(--tag "$IMAGE") + if [ -n "$CACHE" ] && [ "$CACHE" != "null" ]; then + echo " also → $CACHE" + CREATE_ARGS+=(--tag "$CACHE") + fi + docker buildx imagetools create "${CREATE_ARGS[@]}" "$PREV" + done + + # Build/push one service at a time so a single Dockerfile failure (e.g. + # isaac-sim PX4 apt) does not discard successful siblings before push. + - name: Build and push changed images + id: build_push run: | + set -uo pipefail set -a source .env set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" - if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" - else - export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" + mapfile -t BUILD_SERVICES < <(jq -r '.services | to_entries[] | select(.value.action=="build") | .key' docker-image-plan.json | sort) + if [ "${#BUILD_SERVICES[@]}" -eq 0 ]; then + echo "No services require a rebuild." + echo "built_services=" >> "$GITHUB_OUTPUT" + exit 0 fi - IMAGES=$(docker compose config --images | sort -u) - if [ -z "$IMAGES" ]; then - echo "No images resolved from compose config; nothing to sign." + echo "Building services sequentially: ${BUILD_SERVICES[*]}" + FAILED=() + BUILT=() + for SVC in "${BUILD_SERVICES[@]}"; do + echo "::group::Build $SVC" + if docker compose \ + -f docker-compose.yaml \ + -f docker-compose.fingerprint.yaml \ + build "$SVC"; then + echo "::endgroup::" + echo "::group::Push $SVC" + if docker compose \ + -f docker-compose.yaml \ + -f docker-compose.fingerprint.yaml \ + push "$SVC"; then + CACHE=$(jq -r --arg s "$SVC" '.services[$s].cache_tag // empty' docker-image-plan.json) + if [ -n "$CACHE" ]; then + echo "Pushing cache tag $CACHE" + docker push "$CACHE" || echo "::warning::Failed to push cache tag $CACHE" + fi + BUILT+=("$SVC") + else + echo "::error::Push failed for $SVC" + FAILED+=("$SVC") + fi + echo "::endgroup::" + else + echo "::endgroup::" + echo "::error::Build failed for $SVC" + FAILED+=("$SVC") + fi + done + + echo "built_services=${BUILT[*]}" >> "$GITHUB_OUTPUT" + if [ "${#FAILED[@]}" -gt 0 ]; then + echo "Failed services: ${FAILED[*]}" exit 1 fi + # Sign whatever was published even if a sibling service build failed. + - name: Sign published images with Cosign (keyless) + if: always() && steps.plan.outcome == 'success' + env: + COSIGN_YES: "true" + run: | + set -euo pipefail + set -a + source .env + set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" + + # Retagged services are always published; rebuilt ones only if push succeeded. + BUILT_CSV="${{ steps.build_push.outputs.built_services }}" + IMAGES=$( + { + jq -r '.services | to_entries[] | select(.value.action=="retag") | .value.image' docker-image-plan.json + if [ -n "$BUILT_CSV" ]; then + for SVC in $BUILT_CSV; do + jq -r --arg s "$SVC" '.services[$s].image // empty' docker-image-plan.json + done + fi + } | awk 'NF' | sort -u + ) + if [ -z "$IMAGES" ]; then + echo "No published images to sign." + exit 0 + fi + for IMG in $IMAGES; do DIGEST=$(docker buildx imagetools inspect "$IMG" --format '{{.Manifest.Digest}}') if [ -z "$DIGEST" ]; then @@ -164,18 +288,25 @@ jobs: done - name: Verify Cosign signatures + if: always() && steps.plan.outcome == 'success' run: | + set -euo pipefail set -a source .env set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" - if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" - else - export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" - fi - - IMAGES=$(docker compose config --images | sort -u) + BUILT_CSV="${{ steps.build_push.outputs.built_services }}" + IMAGES=$( + { + jq -r '.services | to_entries[] | select(.value.action=="retag") | .value.image' docker-image-plan.json + if [ -n "$BUILT_CSV" ]; then + for SVC in $BUILT_CSV; do + jq -r --arg s "$SVC" '.services[$s].image // empty' docker-image-plan.json + done + fi + } | awk 'NF' | sort -u + ) for IMG in $IMAGES; do DIGEST=$(docker buildx imagetools inspect "$IMG" --format '{{.Manifest.Digest}}') REPO="${IMG%:*}" @@ -187,13 +318,21 @@ jobs: > /dev/null done - - name: Optional - Run Docker Compose Up (uncomment if needed) + - name: Fail job if any service build/push failed + if: always() && steps.build_push.outcome == 'failure' run: | - # Uncomment the following lines if you also want to start the services - # set -a - # source .env - # set +a - # docker compose up -d + echo "One or more services failed to build or push (see Build and push changed images)." + exit 1 + + - name: Upload image plan artifact + if: always() + uses: actions/upload-artifact@v4 + with: + name: docker-image-plan-${{ github.run_id }} + path: | + docker-image-plan.json + docker-compose.fingerprint.yaml + if-no-files-found: ignore notify: needs: [check-docker-tag-change, docker-build] @@ -203,8 +342,8 @@ jobs: - name: Notify build and push result run: | if [ "${{ needs.docker-build.result }}" == "success" ]; then - echo "✅ Docker Compose build and push completed successfully" + echo "✅ Docker Compose build/retag and push completed successfully" else - echo "❌ Docker Compose build or push failed" + echo "❌ Docker Compose build/retag or push failed" exit 1 - fi \ No newline at end of file + fi diff --git a/.github/workflows/scripts/docker_image_plan.py b/.github/workflows/scripts/docker_image_plan.py new file mode 100755 index 000000000..dcaf1a7ae --- /dev/null +++ b/.github/workflows/scripts/docker_image_plan.py @@ -0,0 +1,544 @@ +#!/usr/bin/env python3 +"""Plan retag-vs-rebuild for AirStack docker-build.yml publishes. + +For each compose service with a ``build:`` section under the selected profiles, +compute a content fingerprint of its image inputs. If the previous versioned +image carries the same ``org.airstack.content-fingerprint`` label, the service +is marked ``retag``; otherwise ``build``. + +Also writes an ephemeral compose override that applies the fingerprint as a +build label on services that will be rebuilt. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import subprocess +import sys +from pathlib import Path +from typing import Any + +FINGERPRINT_LABEL = "org.airstack.content-fingerprint" + +# Explicit roots so broad compose ``context:`` dirs do not hash the whole tree. +# Keys are compose service names after ``docker compose config`` resolution. +SERVICE_FINGERPRINT_ROOTS: dict[str, list[str]] = { + "robot-desktop": [ + "robot/docker/Dockerfile.robot", + "robot/docker/docker-compose.yaml", + "robot/docker/robot-base-docker-compose.yaml", + "robot/docker/custom_rosdep.yaml", + "robot/docker/wait_for_px4.py", + "robot/docker/.bashrc", + "robot/docker/robot_name_map", + ], + "gcs": [ + "gcs/docker/Dockerfile.gcs", + "gcs/docker/docker-compose.yaml", + "gcs/docker/gcs-base-docker-compose.yaml", + "gcs/docker/.bashrc", + "gcs/docker/resources", + "gcs/docker/Foxglove", + ], + "isaac-sim": [ + "simulation/isaac-sim/docker/Dockerfile.isaac-ros", + "simulation/isaac-sim/docker/docker-compose.yaml", + "simulation/isaac-sim/docker/fastdds.xml", + "simulation/isaac-sim/docker/.bashrc", + "simulation/isaac-sim/docker/omniverse.toml", + ], + "ms-airsim": [ + "simulation/ms-airsim/docker/Dockerfile", + "simulation/ms-airsim/docker/docker-compose.yaml", + "simulation/ms-airsim/docker/entrypoint.sh", + ], +} + +# Extra roots when DOCKER_IMAGE_BUILD_MODE=prebuilt (workspace baked into image). +PREBUILT_EXTRA_ROOTS: dict[str, list[str]] = { + "robot-desktop": [ + "robot/ros_ws/src", + "common/ros_packages", + "common/fastdds.xml", + ], + "gcs": [ + "gcs/ros_ws", + "common/ros_packages", + ], +} + +# .env keys whose values affect image tags or layers (included in fingerprint). +ENV_FINGERPRINT_KEYS = ( + "DOCKER_IMAGE_BUILD_MODE", + "PROJECT_DOCKER_REGISTRY", + "PROJECT_NAME", + "CACHE_TAG", +) + + +def repo_root() -> Path: + return Path(__file__).resolve().parents[3] + + +def parse_env_value(raw: str) -> str: + """Parse a .env value, honoring quotes and stripping trailing comments.""" + raw = raw.strip() + if not raw: + return "" + if raw[0] in "\"'": + quote = raw[0] + end = raw.find(quote, 1) + if end != -1: + return raw[1:end] + return raw[1:] + # Unquoted: drop an inline ` # comment` (space-hash) or a leading `#`. + if " #" in raw: + raw = raw.split(" #", 1)[0].rstrip() + return raw.strip().strip('"').strip("'") + + +def load_dotenv(path: Path) -> dict[str, str]: + env: dict[str, str] = {} + if not path.is_file(): + return env + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + env[key.strip()] = parse_env_value(value) + return env + + +def run(cmd: list[str], *, cwd: Path, check: bool = True) -> subprocess.CompletedProcess[str]: + return subprocess.run( + cmd, + cwd=cwd, + check=check, + text=True, + capture_output=True, + ) + + +def compose_config(root: Path, profiles: str, env: dict[str, str]) -> dict[str, Any]: + cmd_env = os.environ.copy() + cmd_env.update(env) + cmd_env["COMPOSE_PROFILES"] = profiles + proc = subprocess.run( + ["docker", "compose", "-f", "docker-compose.yaml", "config", "--format", "json"], + cwd=root, + env=cmd_env, + text=True, + capture_output=True, + check=False, + ) + if proc.returncode != 0: + raise RuntimeError( + "docker compose config failed:\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return json.loads(proc.stdout) + + +def git_ls_files(root: Path, pathspec: str) -> list[str]: + proc = run( + ["git", "ls-files", "-z", "--", pathspec], + cwd=root, + check=False, + ) + if proc.returncode != 0: + return [] + return [p for p in proc.stdout.split("\0") if p] + + +def collect_tracked_files(root: Path, roots: list[str]) -> list[str]: + files: set[str] = set() + for rel in roots: + path = root / rel + if path.is_file(): + files.add(rel) + continue + if path.is_dir(): + for tracked in git_ls_files(root, rel): + # Skip local secrets / generated pass files + if tracked.endswith("omni_pass.env"): + continue + if "/.dev/" in f"/{tracked}/" or tracked.endswith("/.dev"): + continue + files.add(tracked) + return sorted(files) + + +def file_sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + + +def resolve_dockerfile(root: Path, service_cfg: dict[str, Any]) -> Path | None: + build = service_cfg.get("build") or {} + if not isinstance(build, dict): + return None + dockerfile = build.get("dockerfile") + context = build.get("context") or "." + if not dockerfile: + return None + # compose config usually resolves dockerfile to an absolute path + df = Path(dockerfile) + if df.is_absolute(): + return df + return (Path(context) / df).resolve() if Path(context).is_absolute() else (root / context / df).resolve() + + +def find_dockerignore(dockerfile: Path, context: Path) -> Path | None: + for candidate in ( + context / ".dockerignore", + dockerfile.parent / ".dockerignore", + ): + if candidate.is_file(): + return candidate + return None + + +def previous_image_ref(image: str, version: str, previous_version: str) -> str | None: + if not previous_version or not version or version == previous_version: + return None + # Tags look like ...:v{VERSION}_suffix — replace only the version segment. + needle = f":v{version}_" + if needle not in image: + # Fallback: replace first occurrence of the bare version in the tag. + tag_part = image.rsplit(":", 1) + if len(tag_part) != 2 or version not in tag_part[1]: + return None + return f"{tag_part[0]}:{tag_part[1].replace(version, previous_version, 1)}" + return image.replace(needle, f":v{previous_version}_", 1) + + +def cache_tag_from_build(build: dict[str, Any], cache_tag: str) -> str | None: + tags = build.get("tags") or [] + pfx = f":{cache_tag}_" + for tag in tags: + if isinstance(tag, str) and pfx in tag: + return tag + return None + + +def inspect_fingerprint_label(image: str) -> str | None: + """Return the fingerprint label from a registry image, or None if unavailable.""" + proc = subprocess.run( + [ + "docker", + "buildx", + "imagetools", + "inspect", + image, + "--format", + "{{json .}}", + ], + text=True, + capture_output=True, + check=False, + ) + if proc.returncode != 0: + return None + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError: + return None + + # buildx JSON shape varies by version; hunt for Labels in common places. + def walk(obj: Any) -> str | None: + if isinstance(obj, dict): + labels = obj.get("Labels") or obj.get("labels") + if isinstance(labels, dict) and FINGERPRINT_LABEL in labels: + return labels[FINGERPRINT_LABEL] + for v in obj.values(): + found = walk(v) + if found: + return found + elif isinstance(obj, list): + for item in obj: + found = walk(item) + if found: + return found + return None + + return walk(data) + + +def compute_fingerprint( + root: Path, + service_name: str, + service_cfg: dict[str, Any], + env: dict[str, str], +) -> str: + build = service_cfg.get("build") or {} + roots = list(SERVICE_FINGERPRINT_ROOTS.get(service_name, [])) + + dockerfile = resolve_dockerfile(root, service_cfg) + if dockerfile and dockerfile.is_file(): + try: + rel = str(dockerfile.relative_to(root)) + except ValueError: + rel = str(dockerfile) + if rel not in roots: + roots.insert(0, rel) + + mode = env.get("DOCKER_IMAGE_BUILD_MODE", "dev") + if mode == "prebuilt": + roots.extend(PREBUILT_EXTRA_ROOTS.get(service_name, [])) + + # If service has no map entry, fall back to dockerfile directory. + if not roots and dockerfile is not None: + try: + roots = [str(dockerfile.parent.relative_to(root))] + except ValueError: + roots = [] + + tracked = collect_tracked_files(root, roots) + + h = hashlib.sha256() + h.update(f"service:{service_name}\n".encode()) + h.update(f"DOCKER_IMAGE_BUILD_MODE:{mode}\n".encode()) + + for key in ENV_FINGERPRINT_KEYS: + h.update(f"env:{key}={env.get(key, '')}\n".encode()) + + args = build.get("args") or {} + if isinstance(args, dict): + for k in sorted(args): + h.update(f"arg:{k}={args[k]}\n".encode()) + + context = build.get("context") + if context: + h.update(f"context:{context}\n".encode()) + ctx_path = Path(context) if Path(context).is_absolute() else root / context + dockerignore = find_dockerignore(dockerfile, ctx_path) if dockerfile else None + if dockerignore and dockerignore.is_file(): + h.update(f"dockerignore:{dockerignore.name}\n".encode()) + h.update(dockerignore.read_bytes()) + h.update(b"\n") + + for rel in tracked: + path = root / rel + if not path.is_file(): + continue + h.update(f"file:{rel}\n".encode()) + h.update(file_sha256(path).encode()) + h.update(b"\n") + + return h.hexdigest() + + +def write_override(path: Path, build_services: dict[str, str]) -> None: + """Write compose override that sets build.labels fingerprint for rebuilds.""" + lines = [ + "# Generated by docker_image_plan.py — do not commit.", + "services:", + ] + if not build_services: + # Valid empty mapping; compose merge ignores it when nothing rebuilds. + lines[-1] = "services: {}" + else: + for name, fingerprint in sorted(build_services.items()): + lines.append(f" {name}:") + lines.append(" build:") + lines.append(" labels:") + lines.append(f" {FINGERPRINT_LABEL}: \"{fingerprint}\"") + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def build_plan( + root: Path, + *, + version: str, + previous_version: str, + profiles: str, + force_rebuild: bool, + env: dict[str, str], +) -> dict[str, Any]: + config = compose_config(root, profiles, env) + services_out: dict[str, Any] = {} + cache_tag = env.get("CACHE_TAG") or "cache" + + for name, cfg in (config.get("services") or {}).items(): + if not isinstance(cfg, dict): + continue + build = cfg.get("build") + if not isinstance(build, dict) or not build: + continue + + image = cfg.get("image") + if not isinstance(image, str) or not image: + continue + + fingerprint = compute_fingerprint(root, name, cfg, env) + prev_image = previous_image_ref(image, version, previous_version) + cache_image = cache_tag_from_build(build, cache_tag) + + action = "build" + reason = "force_rebuild" if force_rebuild else "default_build" + prev_fp = None + if force_rebuild: + action = "build" + reason = "force_rebuild" + elif not prev_image: + action = "build" + reason = "no_previous_image" + else: + prev_fp = inspect_fingerprint_label(prev_image) + if prev_fp is None: + action = "build" + reason = "previous_missing_or_unlabeled" + elif prev_fp == fingerprint: + action = "retag" + reason = "fingerprint_match" + else: + action = "build" + reason = "fingerprint_mismatch" + + services_out[name] = { + "image": image, + "previous_image": prev_image, + "cache_tag": cache_image, + "fingerprint": fingerprint, + "previous_fingerprint": prev_fp, + "action": action, + "reason": reason, + } + + return { + "previous_version": previous_version, + "version": version, + "profiles": profiles, + "force_rebuild": force_rebuild, + "label": FINGERPRINT_LABEL, + "services": services_out, + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=None, help="Repo root (default: AirStack/)") + parser.add_argument("--version", default="", help="Current VERSION (default: from .env)") + parser.add_argument("--previous-version", default="", help="Previous VERSION for retag source") + parser.add_argument( + "--profiles", + default="", + help="Compose profiles (default: COMPOSE_PROFILES or desktop,isaac-sim,ms-airsim)", + ) + parser.add_argument( + "--force-rebuild", + action="store_true", + help="Mark every service as build", + ) + parser.add_argument( + "--plan-out", + type=Path, + default=Path("docker-image-plan.json"), + help="Where to write the plan JSON", + ) + parser.add_argument( + "--override-out", + type=Path, + default=Path("docker-compose.fingerprint.yaml"), + help="Compose override with build labels for rebuild services", + ) + parser.add_argument( + "--github-output", + type=Path, + default=None, + help="Optional path to append GITHUB_OUTPUT keys", + ) + args = parser.parse_args(argv) + + root = (args.root or repo_root()).resolve() + env = load_dotenv(root / ".env") + # Prefer process env overlays (CI exports .env via set -a). + for key in ( + "VERSION", + "DOCKER_IMAGE_BUILD_MODE", + "PROJECT_DOCKER_REGISTRY", + "PROJECT_NAME", + "CACHE_TAG", + "COMPOSE_PROFILES", + ): + if os.environ.get(key): + env[key] = os.environ[key] + + version = args.version or env.get("VERSION") or "" + if not version: + print("ERROR: VERSION is empty", file=sys.stderr) + return 1 + + previous_version = args.previous_version + profiles = ( + args.profiles + or os.environ.get("COMPOSE_PROFILES") + or env.get("COMPOSE_PROFILES") + or "desktop,isaac-sim,ms-airsim" + ) + + plan = build_plan( + root, + version=version, + previous_version=previous_version, + profiles=profiles, + force_rebuild=args.force_rebuild, + env=env, + ) + + build_services = { + name: svc["fingerprint"] + for name, svc in plan["services"].items() + if svc["action"] == "build" + } + retag_services = [name for name, svc in plan["services"].items() if svc["action"] == "retag"] + + args.plan_out = args.plan_out if args.plan_out.is_absolute() else root / args.plan_out + args.override_out = ( + args.override_out if args.override_out.is_absolute() else root / args.override_out + ) + args.plan_out.write_text(json.dumps(plan, indent=2, sort_keys=True) + "\n", encoding="utf-8") + write_override(args.override_out, build_services) + + print(f"Wrote plan → {args.plan_out}") + print(f"Wrote override → {args.override_out}") + for name, svc in sorted(plan["services"].items()): + print( + f" {name}: action={svc['action']} reason={svc['reason']} " + f"fp={svc['fingerprint'][:12]}…" + ) + + build_list = " ".join(sorted(build_services)) + retag_list = " ".join(sorted(retag_services)) + if args.github_output: + with args.github_output.open("a", encoding="utf-8") as fh: + fh.write(f"plan_path={args.plan_out}\n") + fh.write(f"override_path={args.override_out}\n") + fh.write(f"build_services={build_list}\n") + fh.write(f"retag_services={retag_list}\n") + fh.write(f"build_count={len(build_services)}\n") + fh.write(f"retag_count={len(retag_services)}\n") + else: + # Also support GITHUB_OUTPUT env when set by Actions. + gh_out = os.environ.get("GITHUB_OUTPUT") + if gh_out: + with open(gh_out, "a", encoding="utf-8") as fh: + fh.write(f"plan_path={args.plan_out}\n") + fh.write(f"override_path={args.override_out}\n") + fh.write(f"build_services={build_list}\n") + fh.write(f"retag_services={retag_list}\n") + fh.write(f"build_count={len(build_services)}\n") + fh.write(f"retag_count={len(retag_services)}\n") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/system-tests.yml b/.github/workflows/system-tests.yml index 3ddebda58..2818af58c 100644 --- a/.github/workflows/system-tests.yml +++ b/.github/workflows/system-tests.yml @@ -15,8 +15,8 @@ on: default: "liveliness or takeoff_hover_land" required: false sim: - description: "Sim targets, comma-separated: msairsim,isaacsim" - default: msairsim,isaacsim + description: "Sim targets, comma-separated: isaacsim,msairsim. Default isaacsim; pass msairsim to opt in." + default: isaacsim required: false num_robots: description: "Robot counts, comma-separated (e.g. 1,3)" @@ -152,16 +152,28 @@ jobs: print(f'::error::Could not parse pytest args from comment: {e}', file=sys.stderr) sys.exit(1) + # CI-only flag: do not forward to pytest. + no_image_build = False + stripped = [] + for a in args: + if a in ('--no-image-build', '--pull-only'): + no_image_build = True + else: + stripped.append(a) + args = stripped + # Pull out --sim and -m so the image-prep step can scope profiles # and decide whether to skip (build_docker tests rebuild themselves). # When --sim isn't given we mirror conftest's default so prep covers # whatever pytest will actually exercise. - sim = 'msairsim,isaacsim' + sim = 'isaacsim' + sim_explicit = False marks = '' marks_idx = -1 for i, a in enumerate(args): if a == '--sim' and i + 1 < len(args): sim = args[i + 1] + sim_explicit = True elif a == '-m' and i + 1 < len(args): marks = args[i + 1] marks_idx = i + 1 @@ -174,6 +186,24 @@ jobs: marks = f'build_packages or {marks}' args[marks_idx] = marks + # colcon tests do not need a sim image. Default --sim would otherwise + # bake isaac-sim before a 1s colcon test. Pull registry cache tags + # instead; never image-build. + marks_norm = marks.replace('"', '').replace("'", '').strip() + args_blob = ' '.join(args) + heavy = any(m in marks_norm for m in ( + 'liveliness', 'sensors', 'takeoff_hover_land', 'autonomy', 'build_docker', + )) + only_packages = marks_norm == 'build_packages' or ( + not heavy and any(s in args_blob for s in ( + 'test_build_packages', 'test_colcon_', + )) + ) + if only_packages: + no_image_build = True + if not sim_explicit: + sim = 'msairsim' + skip_prep = 'build_docker' in marks quoted = ' '.join(shlex.quote(a) for a in args) @@ -181,10 +211,12 @@ jobs: f.write(f'pytest_args={quoted}\n') f.write(f'sim={sim}\n') f.write(f'skip_image_prep={"true" if skip_prep else "false"}\n') + f.write(f'no_image_build={"true" if no_image_build else "false"}\n') print(f'Resolved pytest args: {quoted or "(none — pytest defaults)"}') print(f'Resolved sim profile: {sim}') print(f'Skip image prep: {skip_prep}') + print(f'No image build (pull/retag only): {no_image_build}') PYEOF # Reply on the PR thread so the commenter sees their /pytest was @@ -200,7 +232,10 @@ jobs: const args = ${{ toJSON(steps.parse.outputs.pytest_args) }}; const cmd = `pytest tests/ ${args}`.trim(); const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`; - const note = `Note: \`build_packages\` is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.`; + const pullOnly = '${{ steps.parse.outputs.no_image_build }}' === 'true'; + const note = pullOnly + ? `Note: pull-only image prep (no \`image-build\`). \`-m build_packages\` does not pull Isaac Sim. Add \`--no-image-build\` on other marks to skip rebuilds.` + : `Note: \`build_packages\` is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.`; await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, @@ -268,6 +303,10 @@ jobs: # inline cache (build_docker tests get layer-reuse speedup) and pre-pulls # before `airstack up` (other tests skip the implicit rebuild). When # secrets are absent both steps are skipped and behavior is unchanged. + # + # Read-only on purpose: AIRSTACK_REGISTRY_CACHE_PUSH stays unset here so a + # PR can consume the floating cache tag but never republish it. Only + # docker-build.yml (main/develop) writes it. - name: Log in to internal Docker registry id: docker_login if: ${{ vars.DOCKER_REGISTRY_URL != '' && env.DOCKER_REGISTRY_PASSWORD != '' }} @@ -284,26 +323,30 @@ jobs: - name: Ensure airstack.sh is executable run: chmod +x airstack.sh + - name: Disable compose image builds + if: ${{ steps.parse.outputs.no_image_build == 'true' }} + run: echo "AIRSTACK_NO_IMAGE_BUILD=1" >> "$GITHUB_ENV" + # The ephemeral runner starts with no local images. `airstack_env` in # tests/conftest.py fails fast if compose images are missing, so prep # them here. Profile-gated services (ms-airsim, isaac-sim) are skipped # by compose unless their profile is active, so we mirror the fixture's - # profile selection from the parsed --sim. Pull-only by default; fall - # back to a full build only if the registry doesn't have everything - # (e.g. new branch with no published image yet). Skipped when the - # marks expression contains build_docker — those tests build per-service - # themselves. + # profile selection from the parsed --sim. Pull versioned tags, then + # retag floating cache_* tags onto the VERSION name (PR tags never + # exist). Fall back to image-build only when --no-image-build is off. + # Skipped when marks contain build_docker — those tests build themselves. - name: Ensure Docker images present if: ${{ steps.parse.outputs.skip_image_prep != 'true' }} env: AIRSTACK_ROOT: ${{ github.workspace }} SIM_INPUT: ${{ steps.parse.outputs.sim }} + NO_IMAGE_BUILD: ${{ steps.parse.outputs.no_image_build }} run: | profiles=desktop [[ ",$SIM_INPUT," == *,msairsim,* ]] && profiles="$profiles,ms-airsim" [[ ",$SIM_INPUT," == *,isaacsim,* ]] && profiles="$profiles,isaac-sim" export COMPOSE_PROFILES="$profiles" - echo "Pulling images for COMPOSE_PROFILES=$COMPOSE_PROFILES" + echo "Pulling images for COMPOSE_PROFILES=$COMPOSE_PROFILES (no_image_build=$NO_IMAGE_BUILD)" # Pull from registry; tolerate per-image failures so we can detect # what's still missing afterwards instead of aborting on the first @@ -311,6 +354,25 @@ jobs: # still surface on stderr. ./airstack.sh --progress=quiet image-pull --ignore-pull-failures || true + # VERSION tags miss on every PR. Seed from floating cache_* tags. + cache_tag="$(grep -E '^CACHE_TAG=' .env 2>/dev/null | cut -d= -f2 | tr -d '"' || true)" + cache_tag="${cache_tag:-cache}" + while IFS= read -r img; do + [[ -z "$img" ]] && continue + if docker image inspect "$img" --format '{{.Id}}' >/dev/null 2>&1; then + continue + fi + # Replace :v_ with :_ (PR versioned tags never exist) + cache_img="$(python3 -c "import re,sys; print(re.sub(r':v[^_]+_', f':{sys.argv[2]}_', sys.argv[1], count=1))" "$img" "$cache_tag")" + echo "Versioned tag missing; trying cache tag $cache_img" + if docker pull --quiet "$cache_img"; then + docker tag "$cache_img" "$img" + echo "Retagged $cache_img -> $img" + else + echo "Cache tag pull failed for $cache_img" + fi + done < <(docker compose -f docker-compose.yaml config --images) + missing=() while IFS= read -r img; do [[ -z "$img" ]] && continue @@ -320,11 +382,16 @@ jobs: done < <(docker compose -f docker-compose.yaml config --images) if (( ${#missing[@]} > 0 )); then - echo "Pull did not produce these images; falling back to build:" + echo "Images still missing after pull/retag:" printf ' - %s\n' "${missing[@]}" + if [[ "$NO_IMAGE_BUILD" == "true" ]]; then + echo "::error::Pull-only mode (--no-image-build or -m build_packages) will not image-build. Run /pytest -m build_docker once, or omit --no-image-build." + exit 1 + fi + echo "Falling back to image-build" ./airstack.sh --progress=quiet image-build else - echo "All required images present after pull — skipping build." + echo "All required images present after pull/retag — skipping build." fi - name: Run tests diff --git a/.gitignore b/.gitignore index d9dd6a07b..37066a0de 100644 --- a/.gitignore +++ b/.gitignore @@ -103,5 +103,10 @@ common/rayfronts/ # Docker build cache (root-owned subdirs cause permission warnings on `git add`) robot/docker/cache/ + +# Ephemeral outputs from docker_image_plan.py (docker-build.yml) +docker-image-plan.json +docker-compose.fingerprint.yaml + .DS_Store gcs/.DS_Store diff --git a/AGENTS.md b/AGENTS.md index 88d524483..76987f8eb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ AirStack/ ├── tests/ # System tests (pytest) + metrics reporting ├── .github/ │ ├── workflows/ # GitHub Actions CI (system-tests, docker-build, etc.) -│ └── orchestrator/ # OpenStack-backed ephemeral self-hosted runners +│ └── orchestrator/ # OSMO-backed ephemeral self-hosted runners └── .agents/skills/ # Detailed workflow guides for agents ``` @@ -289,16 +289,28 @@ GitHub Actions workflows live in [`.github/workflows/`](.github/workflows/): ### Ephemeral Runner Orchestrator -GPU-required jobs (`runs-on: [self-hosted, airstack-ephemeral]`) execute on **OpenStack VMs spawned per-job and destroyed on completion**. The orchestrator service code lives in [`.github/orchestrator/`](.github/orchestrator/): +GPU-required jobs (`runs-on: [self-hosted, airstack-ephemeral]`) execute on **ephemeral pods scheduled by [NVIDIA OSMO](https://nvidia.github.io/OSMO/) — one per job, destroyed on completion**. The GitHub side is unchanged from the old OpenStack backend (same labels, JIT tokens, fork guard); only the spawn target moved from "create a Nova VM" to "submit an OSMO workflow". The orchestrator service code lives in [`.github/orchestrator/`](.github/orchestrator/): -- [`orchestrator.py`](.github/orchestrator/orchestrator.py) — Python service: spawn loop polls GitHub for queued jobs matching configured runner labels, mints single-use JIT runner tokens, creates an OpenStack server with cloud-init bootstrap; reap loop deletes the server when the job completes (or after `max_job_minutes`) -- [`cloud-init.yaml.j2`](.github/orchestrator/cloud-init.yaml.j2) — bootstraps Docker + nvidia-container-toolkit + GH Actions runner on the worker, registers with the JIT token, runs one job, then `shutdown -h` -- [`config.example.yaml`](.github/orchestrator/config.example.yaml) — flavor / network / keypair / floating-IP pool / runner labels / repo +- [`orchestrator.py`](.github/orchestrator/orchestrator.py) — Python service: spawn loop polls GitHub for queued jobs matching configured runner labels, mints single-use JIT runner tokens, and submits one OSMO workflow per job (`osmo workflow submit`); reap loop cancels the workflow when the job completes (or after `max_job_minutes`), plus an orphan sweep via `osmo workflow list` +- [`runner-workflow.yaml.j2`](.github/orchestrator/runner-workflow.yaml.j2) + [`runner.Dockerfile`](.github/orchestrator/runner.Dockerfile) + [`runner-entrypoint.sh`](.github/orchestrator/runner-entrypoint.sh) — the per-job worker: a **privileged**, GPU-enabled OSMO task (prebaked image) that starts an inner Docker daemon (the tests run `airstack up` = docker compose), registers with the JIT token, runs one job, then exits so OSMO reaps the pod +- [`config.example.yaml`](.github/orchestrator/config.example.yaml) — osmo_url / pool / platform / runner_image / resources / runner labels / repo - [`airstack-orchestrator.service`](.github/orchestrator/airstack-orchestrator.service) + [`setup.sh`](.github/orchestrator/setup.sh) — systemd unit and one-time installer -**Why ephemeral:** clean Docker cache per run, no leaked containers, GitHub PAT and OpenStack credentials only on the orchestrator host (workers receive a single-use JIT token bound to one runner registration). State map at `/var/lib/airstack-orchestrator/state.json`; logs via `journalctl -u airstack-orchestrator.service -f`. +**Why ephemeral:** clean Docker cache per run, no leaked containers; the GitHub PAT and the OSMO service-account token live only on the orchestrator host (workers receive a single-use JIT token bound to one runner registration). CI authenticates to OSMO as a shared, non-personal [service account](https://nvidia.github.io/OSMO/main/deployment_guide/appendix/authentication/service_accounts.html) scoped to a dedicated CI GPU pool, so runs don't consume individuals' quotas. The CI pool's platform must have **"Privileged Mode Allowed"** enabled (docker-in-docker). State map at `/var/lib/airstack-orchestrator/state.json`; logs via `journalctl -u airstack-orchestrator.service -f`. -**Setup, debugging a failed job, and SSH-into-worker procedures:** [`.github/orchestrator/README.md`](.github/orchestrator/README.md) (also exposed as [`tests/ci-cd-orchestrator.md`](tests/ci-cd-orchestrator.md) symlink for the docs site). +**Nested DinD needs a non-overlayfs Docker data-root.** The OSMO pod's root filesystem is overlayfs, and Linux rejects a directory on overlayfs as an overlay `upperdir` (`EINVAL`). A dockerd storing data on the pod rootfs pulls images fine but fails every build step that needs a real mount, with errors that masquerade as `apt-get`/`WORKDIR` failures: + +``` +failed to solve: ... mount source: "overlay", target: ".../buildkit/containerd-overlayfs/cachemounts/...", err: invalid argument +``` + +[`runner-entrypoint.sh`](.github/orchestrator/runner-entrypoint.sh) picks a backend by attempting a real overlay mount, preferring a loopback ext4 image at `/var/lib/docker` (real `overlay2`), then a real filesystem already mounted in the pod, then `fuse-overlayfs`, then `vfs`. Landing on `vfs` means builds will be slow and probably run out of disk — check the `[runner-entrypoint] storage:` line in the job log first when Docker builds misbehave. Details: [orchestrator README → Nested DinD and overlayfs](.github/orchestrator/README.md). + +**Docker layer cache is a floating tag, not the versioned one.** Every compose service lists two `cache_from` entries: the versioned image (`airstack:v${VERSION}_`) and a floating one (`airstack:${CACHE_TAG:-cache}_`). Only the floating tag can ever hit on a PR — `check-version-increment` forces `VERSION` up on every PR, so the versioned tag it builds under has by definition never been pushed. Reading and writing are separate switches: `AIRSTACK_REGISTRY_CACHE=1` (set by `system-tests.yml`) pulls and builds with `BUILDKIT_INLINE_CACHE=1`, while `AIRSTACK_REGISTRY_CACHE_PUSH=1` (set only by `docker-build.yml` on main/develop) also publishes both tags. PR runs stay read-only so an unmerged branch can't poison the shared cache or publish an unreleased version. If you add a service with a `build:` section, give it both entries or its builds will always be cold. + +**Publish retags when image inputs are unchanged.** `docker-build.yml` runs [`.github/workflows/scripts/docker_image_plan.py`](.github/workflows/scripts/docker_image_plan.py) on VERSION bumps: each service gets a content fingerprint (`org.airstack.content-fingerprint`). If the previous versioned image already has that label, the job registry-retags (`imagetools create`) instead of rebuilding; only changed services rebuild (and refresh `cache_*`). Use `workflow_dispatch` with `force_rebuild=true` to rebuild everything. PR `build_docker` tests still perform real builds. + +**Setup, debugging a failed job, and exec-into-worker procedures:** [`.github/orchestrator/README.md`](.github/orchestrator/README.md) (also exposed as [`tests/ci-cd-orchestrator.md`](tests/ci-cd-orchestrator.md) symlink for the docs site). ## Documentation Requirements diff --git a/CHANGELOG.md b/CHANGELOG.md index 68f37e9b0..8c99c905e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,12 +19,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Ephemeral CI GPU runners spawn via NVIDIA OSMO (not OpenStack); `system-tests.yml` / `docker-build.yml` still use `airstack-ephemeral` +- Default system-test `--sim` is `isaacsim`; pass `--sim msairsim` to opt in to Microsoft AirSim +- `-m build_packages` CI runs pull `cache_*` images instead of baking sim images +- `docker-build.yml` retags unchanged images on VERSION bumps (content fingerprint) instead of always rebuilding; floating `cache_*` tags still seed PR layer cache - `robot-l4t` compose service knobs are now env-overridable (`AUTONOMY_ROLE`, `FCU_URL`, and the rosbag path via `BAG_STORAGE_PATH`); `FCU_URL` unquoted so the literal serial path reaches MAVROS - `zed-l4t` image: ZED SDK 4.2 → 5.2 with the coupled ROS deps (`zed_msgs` 5.2.1, `point_cloud_transport(_plugins)` 4.x, add `backward_ros`) - Unit tests are defined by `tests/colcon_unit_test_packages.yaml`: `conftest.py` collects each listed package's co-located `test/` dir under `--import-mode=importlib` and marks it `unit` (ament lint files are skipped and run under `colcon test`) ### Fixed +- Isaac Sim image: PX4 `ubuntu.sh` no longer fails dpkg configure on the NVIDIA base (`ca-certificates` / `software-properties-common`); use `--no-nuttx --no-sim-tools` like ms-airsim +- Robot image: pin `pytest<8.1` and disable `launch_testing` for colcon unit tests so ROS Jazzy's outdated pytest hook does not abort `colcon test` - Robot name resolution now honors a pre-set `ROBOT_NAME` (e.g. injected via docker compose) instead of always overriding it from the container/hostname mapping (`robot/docker/.bashrc`) - Robot name-map catch-all fallback now maps to `unknown_robot` (valid ROS namespace token) instead of `unknown-robot` (`default_robot_name_map.yaml`) - l4t robot image: replace dustynv's `/ros_entrypoint.sh` with a passthrough so its prebuilt source-ROS libs (older `fastcdr`) no longer shadow the apt Jazzy runtime and crash apt-built nodes like MAVROS diff --git a/airstack.sh b/airstack.sh index 1c57c47cb..d42c5b6e1 100755 --- a/airstack.sh +++ b/airstack.sh @@ -824,6 +824,46 @@ function ensure_robot_l4t_stack_base() { run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${_ga[@]}" build "${build_opts[@]}" robot-l4t-stack-base } +# `docker compose push` only publishes each service's `image:`. The floating +# cache tags are declared in `build.tags`, so they need an explicit push. Read +# them back out of the resolved config instead of reconstructing the names here, +# so this stays correct as services are added. +function push_cache_tags() { + local -n _ga="$1" + local -n _sc="$2" + + if ! command -v jq >/dev/null 2>&1; then + log_warn "jq not found; skipping cache-tag push (floating cache will go stale)" + return 0 + fi + + # Service names only — drop any flags that were passed through to the subcommand. + local services=() + for arg in "${_sc[@]}"; do + [[ "$arg" == -* ]] || services+=("$arg") + done + + local tags + tags=$(run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${_ga[@]}" config --format json 2>/dev/null \ + | jq -r --arg svcs "${services[*]}" --arg pfx ":${CACHE_TAG:-cache}_" ' + .services | to_entries[] + | select($svcs == "" or (($svcs | split(" ")) | index(.key))) + | (.value.build.tags // [])[] + | select(contains($pfx)) + ' 2>/dev/null | sort -u) + + if [[ -z "$tags" ]]; then + log_warn "No cache tags resolved from compose config; nothing to publish" + return 0 + fi + + while IFS= read -r tag; do + [[ -n "$tag" ]] || continue + log_info "Pushing cache tag $tag" + docker push "$tag" || log_warn "Failed to push cache tag $tag" + done <<< "$tags" +} + function cmd_up { check_docker @@ -857,7 +897,12 @@ function cmd_up { fi log_info "Starting services..." - run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" up "${subcmd_args[@]}" -d + local up_opts=() + if [[ "${AIRSTACK_NO_IMAGE_BUILD:-}" == "1" ]]; then + log_info "AIRSTACK_NO_IMAGE_BUILD=1 → compose up --no-build" + up_opts+=(--no-build) + fi + run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" up "${up_opts[@]}" "${subcmd_args[@]}" -d log_info "Services brought up successfully" } @@ -877,9 +922,16 @@ function cmd_image_build { # Registry-cache mode (CI / opt-in): pre-pull existing images to seed the # local cache, build with BUILDKIT_INLINE_CACHE=1 so the resulting image - # carries layer-cache metadata, and push so the next run benefits. The - # cache_from declarations in each component compose file make BuildKit - # actually reuse the pulled layers. No-op when the env var is unset. + # carries layer-cache metadata. The cache_from declarations in each + # component compose file make BuildKit actually reuse the pulled layers. + # No-op when the env var is unset. + # + # Reading and publishing the cache are separate switches. A PR bumps VERSION + # (check-version-increment enforces it), so the versioned cache_from entry is + # guaranteed to miss and the floating CACHE_TAG entry is what actually hits. + # PR runs must not write that floating tag: an unmerged branch would poison + # the shared cache and publish an unreleased VERSION. Only trusted branches + # set AIRSTACK_REGISTRY_CACHE_PUSH=1. if [[ "${AIRSTACK_REGISTRY_CACHE:-}" == "1" ]]; then log_info "AIRSTACK_REGISTRY_CACHE=1 → pulling for cache seed..." run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" pull --ignore-pull-failures "${subcmd_args[@]}" || \ @@ -888,9 +940,14 @@ function cmd_image_build { log_info "Building services with BUILDKIT_INLINE_CACHE=1..." run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" build --build-arg BUILDKIT_INLINE_CACHE=1 "${subcmd_args[@]}" - log_info "Pushing built images for next-run cache..." - run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" push --ignore-push-failures "${subcmd_args[@]}" || \ - log_warn "Post-build push encountered failures; future runs may not benefit from cache" + if [[ "${AIRSTACK_REGISTRY_CACHE_PUSH:-}" == "1" ]]; then + log_info "Pushing built images for next-run cache..." + run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" push --ignore-push-failures "${subcmd_args[@]}" || \ + log_warn "Post-build push encountered failures; future runs may not benefit from cache" + push_cache_tags global_args subcmd_args + else + log_info "AIRSTACK_REGISTRY_CACHE_PUSH is not 1 → cache is read-only for this run" + fi else log_info "Building services..." run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" build "${subcmd_args[@]}" diff --git a/docs/development/intermediate/testing/ci_cd.md b/docs/development/intermediate/testing/ci_cd.md index a5afaf265..a53f5926d 100644 --- a/docs/development/intermediate/testing/ci_cd.md +++ b/docs/development/intermediate/testing/ci_cd.md @@ -1 +1,568 @@ -# CI/CD Pipeline \ No newline at end of file +# CI/CD Pipeline on OSMO + +AirStack's continuous integration runs the **full drone stack** — simulator, +robot autonomy, and GCS — on a GPU for every change. Because that needs a +GPU, a Docker daemon, and a clean filesystem, jobs cannot run on GitHub's +hosted runners and should not run on a shared always-on machine. Instead, a +small orchestrator service watches the GitHub Actions queue and submits one +**ephemeral [NVIDIA OSMO](https://nvidia.github.io/OSMO/) pod per job**. The pod +registers as a single-use GitHub Actions runner, executes exactly one job, and +is destroyed. + +This page documents the whole system: the architecture, the job lifecycle, +what each test suite actually catches, how to trigger and read a run, and how +to fit CI into your day-to-day development loop. + +!!! note "Related pages" + - [`tests/README.md`](../../../../tests/README.md) — the test suite reference: marks, fixtures, metrics, CLI flags. + - [CI/CD Orchestrator](../../../../tests/ci-cd-orchestrator.md) — the lab-admin runbook: pool prerequisites, credential staging, `setup.sh`, rotation, break-glass debugging. + - [AirStack on OSMO](../../../tutorials/airstack_on_osmo.md) — the *interactive* OSMO dev pod (Remote-SSH + Isaac Sim streaming). Different workflow, same compute pool. + +--- + +## The short version + +| Question | Answer | +|---|---| +| Where do CI jobs run? | A fresh GPU pod on the OSMO `airstack` pool, one per job, destroyed after. | +| What triggers a run? | A PR being **opened**, a `/pytest` comment from a maintainer, or manual `workflow_dispatch`. | +| What gets tested? | Docker image builds, `colcon` builds, unit tests, stack liveliness, sensor rates, takeoff/hover/land, fixed-trajectory tracking. | +| How do I see results? | A metrics report comment on the PR, plus the `test-results-*` artifact (`summary.txt`, `results.xml`, `metrics.json`). | +| What fails the build? | Any failed test, **or** a metric regressing more than 20 % against the base branch's last run. | +| Who holds the secrets? | Only the orchestrator host. Workers get a single-use JIT token valid for one registration. | + +--- + +## Architecture + +Three planes, each owning one job. GitHub owns the queue and the logs. The +orchestrator owns the credentials and the job ↔ pod bookkeeping. OSMO owns the +GPU compute and the pod lifecycle. + +```mermaid +flowchart LR + subgraph gh [GitHub] + pr["Pull request / comment / dispatch"] + queue["Actions queue
workflow_job: queued
labels: self-hosted, airstack-ephemeral"] + api["REST API"] + pr --> queue + queue --- api + end + + subgraph orch ["Orchestrator host (no GPU, always on)"] + svc["airstack-orchestrator.service
orchestrator.py"] + spawn["spawn loop — every 15s"] + reap["reap loop — every 30s"] + creds["/etc/airstack-orchestrator
github-pat + osmo-token + config.yaml"] + state["/var/lib/airstack-orchestrator/state.json
job_id → workflow_id"] + svc --> spawn + svc --> reap + svc --- creds + spawn --- state + reap --- state + end + + subgraph osmo ["OSMO airstack pool (GPU, privileged)"] + wf["Workflow gha-runner-JOBID-TS"] + pod["Ephemeral runner pod
airstack-ci-runner image"] + wf --> pod + end + + api -- "poll queued jobs" --> spawn + spawn -- "mint JIT runner config" --> api + spawn -- "osmo workflow submit" --> wf + pod -- "register + long-poll for work" --> api + reap -- "osmo workflow cancel" --> wf +``` + +Key properties that fall out of this shape: + +- **Truly ephemeral.** Every job starts from the prebaked image with an empty Docker cache. No leftover containers, no dangling networks, no "works because the last run left something behind". +- **PAT isolation.** The GitHub PAT never leaves the orchestrator. The pod receives a [JIT runner config](https://docs.github.com/en/rest/actions/self-hosted-runners#create-configuration-for-a-just-in-time-runner-for-a-repository) — a base64 blob bound to exactly one runner registration, short-lived. +- **Non-personal OSMO identity.** The orchestrator authenticates with a service-account token scoped to the CI pool, so runs never consume an individual's GPU quota and nothing breaks when someone graduates. +- **Crash-safe.** Every workflow is named `gha-runner--`. The reap loop cancels any active workflow with that prefix that is missing from `state.json`, so a crashed or restarted orchestrator cannot leak pods. + +--- + +## Job lifecycle + +From "you comment `/pytest`" to "the pod is gone", in order: + +```mermaid +sequenceDiagram + autonumber + participant Dev as Developer + participant GH as GitHub Actions + participant Orch as Orchestrator + participant OSMO as OSMO scheduler + participant Pod as Runner pod + + Dev->>GH: open PR / comment /pytest / dispatch + GH->>GH: queue job with labels self-hosted + airstack-ephemeral + Orch->>GH: poll queued jobs (15s) + Orch->>GH: POST generate-jitconfig + GH-->>Orch: encoded_jit_config (single use) + Orch->>Orch: render runner-workflow.yaml.j2 + Orch->>OSMO: osmo workflow submit --pool airstack + OSMO-->>Orch: workflow name, then uuid via query + Orch->>Orch: record job_id to workflow_id in state.json + OSMO->>Pod: schedule privileged GPU pod + Pod->>Pod: start inner dockerd, nvidia-smi check + Pod->>GH: run.sh --jitconfig, register ephemeral runner + GH->>Pod: dispatch the one job + Pod->>Pod: checkout, pull images, pytest + Pod-->>GH: logs, conclusion, results artifact + Pod->>Pod: run.sh exits after one job, task completes + OSMO->>OSMO: tear the pod down + Orch->>GH: poll job status (30s) + GH-->>Orch: completed + Orch->>OSMO: cancel if still live, then drop from state.json +``` + +Two safety nets run on top of the happy path: + +- **Straggler reap.** Any tracked job older than `max_job_minutes` (default 48 h) is force-cancelled regardless of what GitHub reports. +- **Orphan sweep.** Active `gha-runner-*` workflows that are not in `state.json` and are more than two minutes old get cancelled. The two-minute grace window prevents the sweep from racing a submit that has not been recorded yet. + +--- + +## Anatomy of a runner pod + +The worker is a prebaked image — everything the job needs is already in the +layer cache when the pod starts, so a slow `apt-get` can never outlive the JIT +token's validity window. + +```mermaid +flowchart TB + subgraph pod ["OSMO task — privileged, 1 GPU, 8 CPU, 32Gi RAM, 300Gi disk"] + entry["run-ephemeral-runner.sh"] + dockerd["inner dockerd
+ nvidia-container-toolkit"] + runner["actions-runner run.sh --jitconfig"] + subgraph compose ["docker compose stack started by airstack up"] + sim["isaac-sim or ms-airsim"] + robot["robot-desktop x NUM_ROBOTS"] + gcs["gcs"] + end + entry --> dockerd + entry --> runner + runner -- "pytest tests/ runs airstack up" --> dockerd + dockerd --> sim + dockerd --> robot + dockerd --> gcs + end + gpu["Node GPU"] --> dockerd +``` + +| Piece | File | What it contributes | +|---|---|---| +| Image | [`runner.Dockerfile`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner.Dockerfile) | Ubuntu 24.04 + Docker CE + compose/buildx + NVIDIA container toolkit + pinned `actions/runner` (2.334.0) | +| Entrypoint | [`runner-entrypoint.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-entrypoint.sh) | Starts `dockerd`, waits up to 60 s for it, runs `nvidia-smi` as a non-fatal GPU sanity check, then `exec`s `run.sh --jitconfig` | +| Pod shape | [`runner-workflow.yaml.j2`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-workflow.yaml.j2) | Resource request, `privileged: true`, the JIT config and `RUNNER_ALLOW_RUNASROOT` env | +| Sizing | `config.yaml` | `cpu: 8`, `gpu: 1`, `memory: 32Gi`, `storage: 300Gi` — sized for sim + robot + GCS images plus Isaac assets | + +!!! warning "Privileged is mandatory" + The tests run `airstack up`, which is `docker compose`, which needs a Docker + daemon *inside* the pod. That requires the pool's platform to have + **Privileged Mode Allowed** enabled. Without it, submissions are rejected and + `osmo workflow logs` shows `dockerd did not become ready`. + +Build and publish the image with +[`build-and-push.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/build-and-push.sh), +or — if you have no local Docker — submit +[`build-runner-on-osmo.yaml`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/build-runner-on-osmo.yaml), +a one-shot OSMO job that builds the runner image inside an OSMO pod and pushes +it to Harbor. + +--- + +## Triggering a run + +### The three entry points + +| Trigger | When it fires | What it runs | +|---|---|---| +| `pull_request` (`types: [opened]`) | Only when the PR is first opened, and only for same-repo branches | pytest's `conftest` defaults — the full mark set | +| `/pytest` PR comment | Any time, from a user with `OWNER`/`MEMBER`/`COLLABORATOR` association | Whatever args you put on the first line of the comment | +| `workflow_dispatch` | Manual, from the Actions tab | The form inputs: `marks`, `sim`, `num_robots`, `stress_iterations`, `stable_duration`, `baseline_run_id` | + +Pushes to an open PR deliberately do **not** re-trigger. GPU pods are a shared +resource, so re-runs are opt-in via `/pytest`. + +### Comment syntax + +The first line is parsed with `shlex`; everything after it is free-form notes. + +```text +/pytest -m liveliness --sim msairsim --num-robots 1 --stress-iterations 1 + +Checking whether the DDS bridge fix holds under 3 robots — see thread above. +``` + +`-m build_packages` is **pull-only**: it retags floating `cache_*` images onto the PR `VERSION` tag and never runs `image-build` (and does not pull Isaac Sim). Use that when iterating on colcon/pytest failures. For other marks, add `--no-image-build` to skip the bake: + +```text +/pytest -m build_packages +/pytest -m liveliness --sim msairsim --no-image-build +``` + +The workflow replies on the thread with the exact `pytest` command it resolved +and a link to the run, and opens a **Check Run** pinned to the PR head SHA so +comment-triggered runs still show up in the PR's Checks tab. + +!!! tip "`build_packages` is prepended for you" + Whenever you pass `-m`, the workflow rewrites the expression to + `build_packages or `. Launch tests are useless against a stale + `install/` tree, and this removes the most common way to waste a 40-minute + GPU run. It is skipped when you already named `build_packages`, and when you + pass no marks at all (pytest then runs everything anyway). + +### What the job does, step by step + +```mermaid +flowchart TD + a["Resolve PR head — issue_comment only"] --> b["Parse pytest args
prepend build_packages, extract --sim"] + b --> c["Ack comment + open in-progress Check Run"] + c --> d["Checkout PR head with submodules"] + d --> e["Write omni_pass.env — guest Nucleus creds"] + e --> f["Create venv, install tests/requirements.txt"] + f --> g{"Registry secrets present?"} + g -- yes --> h["docker login, set AIRSTACK_REGISTRY_CACHE=1
read-only: PRs never republish the cache"] + g -- no --> i["Skip — build from scratch"] + h --> j{"marks contain build_docker?"} + i --> j + j -- yes --> l["Skip image prep — those tests build themselves"] + j -- no --> k["airstack image-pull for the active profiles
fall back to image-build for anything missing"] + k --> m["pytest tests/ with resolved args"] + l --> m + m --> n["Upload tests/results/ artifact, 90-day retention"] + n --> o["Finalize Check Run with the job conclusion"] + o --> p["report job on ubuntu-latest"] +``` + +The image-prep step is what makes runs on a cold pod tolerable: it pulls the +published images for exactly the compose profiles the selected `--sim` implies, +then falls back to a local build only for images the registry did not have (a +new branch that has not been released yet, for example). + +### Layer cache: the floating `cache_*` tag + +Every pod starts with an empty Docker cache, so `build_docker` is only fast if +BuildKit can import layers from the registry. Each compose service therefore +declares two `cache_from` entries: + +| Entry | Example | Who writes it | +|---|---|---| +| Versioned | `airstack:v0.19.0-alpha.7_isaac-sim` | `docker-build.yml`, per release | +| Floating | `airstack:cache_isaac-sim` | `docker-build.yml`, republished every build | + +The versioned entry alone cannot work on a pull request. `check-version-increment` +requires every PR to raise `VERSION`, so the tag a PR builds under is by +definition one that has never been pushed — the pull misses and the build runs +cold from the first `RUN` layer: + +``` +Image ...airstack:v0.19.0-alpha.7_isaac-sim Pulling +Image ...airstack:v0.19.0-alpha.7_isaac-sim failed to resolve reference +``` + +The floating tag is the one that actually hits. It tracks the newest build from +`main`/`develop` rather than any particular version, so a PR imports the layers +its base branch already produced and rebuilds only what it changed. + +Reading and writing the cache are separate switches, and PR runs get read only: + +- `AIRSTACK_REGISTRY_CACHE=1` — pull to seed, build with `BUILDKIT_INLINE_CACHE=1`. + Set by `system-tests.yml` whenever registry secrets are available. +- `AIRSTACK_REGISTRY_CACHE_PUSH=1` — additionally publish the versioned and + floating tags. Set only by `docker-build.yml`. + +Keeping the write switch off for pull requests means an unmerged branch can +neither poison the shared cache for everyone else nor publish an unreleased +`VERSION` tag. Override the tag name with `CACHE_TAG` (default `cache`) to keep +an experimental cache line separate. + +### Publish path: retag when image inputs are unchanged + +`check-version-increment` forces every PR to raise `VERSION`, including +docs-only changes. On `main`/`develop`, that would otherwise mean a full +multi-hour rebuild of every image for a no-op Docker change. + +[`docker-build.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/docker-build.yml) +therefore plans per service before building: + +1. [`.github/workflows/scripts/docker_image_plan.py`](https://github.com/castacks/AirStack/blob/main/.github/workflows/scripts/docker_image_plan.py) + hashes each service’s Dockerfile, compose-related files, build args, and + tracked fingerprint roots into `org.airstack.content-fingerprint`. +2. It inspects the **previous** versioned image’s label (from `HEAD~1`’s + `VERSION=`). +3. **Match** → registry-side retag with + `docker buildx imagetools create` (new `v${VERSION}_…` tag and floating + `cache_*` tag, same digest — no rebuild). +4. **Mismatch / missing / unlabeled / `force_rebuild`** → `docker compose build` + for that service only, with the fingerprint applied as a build label via an + ephemeral `docker-compose.fingerprint.yaml` override. + +PR `system-tests` / `build_docker` are unchanged: they still run real builds so +Dockerfiles keep being proven. Floating `cache_*` remains the layer-cache seed +for those rebuilds. + +Manual dispatch accepts `force_rebuild=true` to rebuild and relabel everything +(useful the first time after this lands, or to refresh `cache_*` from scratch). + +--- + +## What the pipeline tests, and what that catches + +Tests are selected with pytest marks. Collection order is fixed in +`tests/conftest.py` so cheap and prerequisite suites always run first — a +`colcon` break fails in minutes instead of after a sim bring-up. + +```mermaid +flowchart LR + u["unit
seconds, no Docker"] --> bd["build_docker
image builds"] + bd --> bp["build_packages
colcon build in containers"] + bp --> lv["liveliness
stack comes up"] + lv --> sn["sensors
streams flow at rate"] + sn --> th["takeoff_hover_land
flight chain"] + th --> au["autonomy
trajectory tracking"] +``` + +| Mark | Module | What it verifies | Bugs it is good at catching | +|---|---|---|---| +| `unit` | `tests/robot/`, `tests/sim/` proxies | Hermetic Python/numpy logic co-located with each ROS 2 package | Off-by-one and boundary errors in filters, converters, validators; regressions in pure algorithm code | +| `build_docker` | `system/test_build_docker.py` | Every image builds; records image sizes | Broken Dockerfiles, deleted apt packages, upstream base-image drift, accidental image bloat | +| `build_packages` | `system/test_build_packages.py` | `colcon build` inside robot, GCS, and ms-airsim workspaces | Missing `package.xml` dependencies, uninstalled launch/config files, C++ breakage on a clean tree | +| `liveliness` | `system/test_liveliness.py` | Containers reach Running, `/clock` publishes, tmux panes alive, sentinel ROS 2 nodes present, compute snapshot, stability poll | Launch files that crash on start, nodes that die after 30 s, `ROBOT_NAME`/domain-ID misconfiguration, runaway CPU or memory | +| `sensors` | `system/test_sensors.py` | Stereo and depth publish rates on both sim and robot side, filtered LiDAR liveness plus geometry sanity, sim real-time factor, time-series stability | Broken sim-to-ROS bridges, sensor Hz that silently halves, RTF collapse from a heavy new node, LiDAR filter range regressions | +| `takeoff_hover_land` | `system/test_takeoff_hover_land.py` | Four-phase chain per (sim, robots, iteration, velocity): PX4 ready → takeoff to 10 m → hover → land | Controller tuning regressions, altitude overshoot, hover drift, state-estimation bias against ground truth, PX4/MAVROS handshake breakage | +| `autonomy` | `system/test_fixed_trajectory.py` | Same chain with a Circle / Figure8 / Racetrack / Line pattern in the middle; records cross-track error and path RMSE | Path-tracker regressions, trajectory-library math errors, velocity/acceleration limit violations that show up as corner-cutting | + +### The flight chain + +Both flight suites run as an ordered chain per parametrization, so the drone +always ends on the ground before the next configuration starts: + +```mermaid +flowchart LR + r["test_px4_ready
MAVROS + EKF"] --> t["test_takeoff
within 10% of 10 m"] + t --> x["test_hover or test_fixed_trajectory"] + x --> l["test_landing
final altitude < 0.5 m"] + r -. "failure" .-> s["remaining phases skipped"] + t -. "failure" .-> s + x -. "failure still lands" .-> l +``` + +A failure in the middle phase (`test_hover` or `test_fixed_trajectory`) does +**not** skip landing — a bad tracker must not leave a drone stuck in the air +blocking the rest of the sweep. A failure in `test_px4_ready` or `test_takeoff` +does skip the remaining phases for that configuration. + +### Bring-up scope, and why mark selection costs money + +`airstack_env` is **class-scoped** and parametrized over +`(sim, num_robots, iteration)`. Each test class does its own `airstack up` and +`airstack down`. Selecting two suites with `or` therefore performs **two full +stack cycles per tuple**: + +```text +-m liveliness → 1 bring-up per (sim, robots, iter) +-m "liveliness or sensors" → 2 bring-ups per (sim, robots, iter) +--sim msairsim → opt in; both sims doubles all of the above +--num-robots 1,3 → doubles it again +``` + +Run one mark at a time unless you genuinely need both. + +--- + +## Reading the results + +### The PR comment + +After `run-tests` finishes — pass or fail — a `report` job on `ubuntu-latest` +downloads the current artifact plus a **baseline** artifact and runs +[`parse_metrics.py`](https://github.com/castacks/AirStack/blob/main/tests/parse_metrics.py) +in diff mode. + +| Run type | Baseline used | +|---|---| +| PR opened or `/pytest` | Latest `system-tests.yml` artifact on the PR's base branch | +| `workflow_dispatch` with `baseline_run_id` | That specific run | +| `workflow_dispatch` without it | Latest artifact on `main` | + +The comment has three sections per test module: a flat **Metrics** table, a +**Sim publishing rates** pivot (topic Hz aggregates from the `sensors` mark), +and a **Compute usage** pivot (CPU / memory / GPU per container). Regressions +are marked with a red circle, improvements with a green one, and the job +**fails** if any metric moves more than the 20 % threshold in the wrong +direction. That is the mechanism that catches slow degradation — the kind of +change where nothing throws but the tracker is quietly 30 % worse. + +### The artifact + +`test-results--`, retained 90 days: + +```text +tests/results/2026-08-06_14-30-00/ +├── summary.txt # human-readable per-chain summary — open this first +├── results.xml # JUnit XML: durations, pass/fail per test +└── metrics.json # every recorded metric, including time series +``` + +There are no per-test log files. Live output streams to the Actions log via +pytest's `log_cli`, and failed assertions embed the tail of the relevant +`docker` or `ros2` subprocess output directly in the failure message. + +Regenerate a report locally from a downloaded artifact: + +```bash +python tests/parse_metrics.py \ + --current path/to/current-run/ \ + --baseline path/to/baseline-run/ \ + --threshold 20 +``` + +--- + +## Using CI well while developing + +The pipeline is expensive at the far end and nearly free at the near end. Push +each class of failure as far left as it will go. + +```mermaid +flowchart TD + q{"What did you change?"} + q -- "Pure Python / numpy logic" --> u["airstack test -m unit
seconds, no GPU"] + q -- "Dockerfile / dependency" --> b["airstack test -m build_docker or build_packages
minutes, no GPU"] + q -- "Launch file / new node" --> l["airstack test -m liveliness --sim msairsim --num-robots 1"] + q -- "Sensor or bridge" --> s["airstack test -m sensors --sim isaacsim --num-robots 1"] + q -- "Controller / planner" --> a["airstack test -m autonomy --sim msairsim --trajectory-types Circle"] + u --> pr["Push branch, open PR"] + b --> pr + l --> pr + s --> pr + a --> pr + pr --> ci["Full suite runs on the ephemeral GPU pod"] + ci --> rep["Read the metrics comment"] + rep --> iter["/pytest with a narrowed mark to confirm a fix"] +``` + +Practical rules that follow from how the system is built: + +- **Reproduce CI locally with the same command.** `airstack test` and CI both call `pytest tests/` with the same flags. If a run fails in CI, copy the resolved command from the acknowledgment comment and run it on any GPU box — including an [interactive OSMO dev pod](../../../tutorials/airstack_on_osmo.md) if you do not have a local GPU. +- **Narrow before you re-run.** A `/pytest` with no args re-runs everything. `/pytest -m autonomy --sim msairsim --trajectory-types Circle` re-runs the one chain you are fixing, in a fraction of the time. +- **Never trust a green launch test against a stale build.** This is why `build_packages` is auto-prepended; keep it that way when writing your own `/pytest` line. +- **Read `summary.txt` before the raw log.** It groups each flight chain with per-phase wall times and status, so the failing phase is obvious without scrolling a 40-minute log. +- **Treat the metrics diff as a review artifact.** A PR that turns a metric red needs an explanation in the thread, even when every test passed. +- **Bump `VERSION` in `.env` when image content changes.** [`check-version-increment.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/check-version-increment.yml) gates the PR on a strictly-greater semver, and merging that bump is what triggers the release build below. + +--- + +## The release path + +`system-tests.yml` is not the only workflow on the ephemeral runners. +[`docker-build.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/docker-build.yml) +also requests `runs-on: [self-hosted, airstack-ephemeral]` and therefore gets +the same per-job pod treatment. + +```mermaid +flowchart LR + pr["PR merged to main or develop"] --> chk{".env VERSION changed?"} + chk -- no --> stop["No build"] + chk -- yes --> pod["Ephemeral OSMO pod"] + pod --> plan["Per-service fingerprint plan"] + plan --> retag["imagetools retag unchanged"] + plan --> build["compose build changed only"] + retag --> sign["cosign sign — keyless, GitHub OIDC"] + build --> sign + sign --> verify["cosign verify against the workflow identity"] +``` + +Signing is keyless via GitHub's OIDC token, and the same job immediately +verifies each published digest against the expected certificate identity, so a +published image that was not built by this workflow fails the check. Retagged +images keep the previous digest (and therefore an existing signature still +covers that digest; the job re-signs the same digest under the new tags’ refs). + +| Workflow | Runner | Purpose | +|---|---|---| +| `system-tests.yml` | Ephemeral OSMO GPU pod | Full test suite + metrics report | +| `docker-build.yml` | Ephemeral OSMO GPU pod | Retag or rebuild, push, and sign compose images | +| `check-version-increment.yml` | `ubuntu-latest` | Semver gate on `.env` `VERSION=` | +| `deploy_docs_from_*.yaml` | `ubuntu-latest` | Versioned MkDocs publish via `mike` | + +--- + +## Security model + +| Concern | How the design handles it | +|---|---| +| Cross-job state pollution | Fresh pod per job with an empty Docker cache; destroyed within ~30 s of completion | +| Fork PRs executing arbitrary code on a GPU node | `head.repo.full_name == github.repository` guard on the `pull_request` path, and an explicit fork check plus `author_association` gate on the `/pytest` path | +| Long-lived GitHub PAT on a worker | The PAT lives only on the orchestrator; workers get a single-use JIT config bound to one registration | +| Credentials tied to a person | OSMO auth uses a non-personal service-account token scoped to the CI pool | +| Privileged container is root-equivalent | Accepted deliberately — docker-in-docker is required — but bounded to a one-shot pod, on a dedicated pool, running only same-repo code | +| Orchestrator compromise blast radius | Systemd hardening: `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome=read-only`, `PrivateTmp`, with a single `ReadWritePaths` for state | +| Leaked pods after a crash | Name-prefix orphan sweep plus a `max_job_minutes` straggler ceiling | + +--- + +## Troubleshooting + +A failed run can break at the orchestrator, at the OSMO pod, at the runner, or +in the tests themselves, and each layer has a different inspection path. Work +down the list. + +| Symptom | Layer | First thing to check | +|---|---|---| +| Job sits `queued` forever, no pod appears | Orchestrator | `journalctl -u airstack-orchestrator.service --since '30 min ago'` — look for `submitted workflow for job ` | +| `find_queued_jobs failed: 401` | Orchestrator | GitHub PAT expired or lost a scope; rotate it | +| `osmo login failed` / auth error | Orchestrator | OSMO service-account token expired (default 31 days); mint a new one and restart the service | +| `osmo workflow submit failed ... privileged` | OSMO pool | The pool's platform lacks **Privileged Mode Allowed** | +| Job queued but never claimed | Labels | `runs-on` labels must be a superset of `runner_labels` in `config.yaml` | +| `dockerd did not become ready` | Pod | Not actually privileged; check the platform, then `osmo workflow logs "$WF" --task runner` | +| `nvidia-smi unavailable` | Pod | GPU not requested or the toolkit is not configured on the node | +| `Cannot connect to the Docker daemon` mid-test | Pod | Inner dockerd crashed — `osmo workflow exec "$WF" runner`, then read `/var/log/dockerd.log` | +| `No space left on device` | Pod | Bump `storage` in `config.yaml`; Isaac assets plus all images are large | +| Runner registered, then pytest failed | Tests | A real test failure — the GitHub Actions log and `summary.txt` are canonical | +| Metrics report job failed with no test failures | Report | A metric regressed past the 20 % threshold; read the diff table | + +To map a GitHub job to its pod: + +```bash +JOB_ID=73286176852 # from the GitHub Actions URL +WF=$(sudo jq -r ".jobs[\"$JOB_ID\"].workflow_id" /var/lib/airstack-orchestrator/state.json) + +osmo workflow query "$WF" --verbose +osmo workflow events "$WF" --task runner # scheduling, image pull, eviction +osmo workflow logs "$WF" --task runner # dockerd, run.sh, and the job itself +osmo workflow exec "$WF" runner # break-glass shell, while RUNNING +``` + +Full runbook, including credential rotation and worker-side diagnostics: +[CI/CD Orchestrator](../../../../tests/ci-cd-orchestrator.md). + +--- + +## File map + +| Path | Role | +|---|---| +| [`.github/workflows/system-tests.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/system-tests.yml) | The test workflow: triggers, arg parsing, image prep, pytest, artifact, metrics report | +| [`.github/orchestrator/orchestrator.py`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/orchestrator.py) | The spawn and reap loops, GitHub polling, JIT minting, OSMO CLI plumbing | +| [`.github/orchestrator/runner-workflow.yaml.j2`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-workflow.yaml.j2) | Per-job OSMO workflow template | +| [`.github/orchestrator/runner.Dockerfile`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner.Dockerfile) | Prebaked worker image | +| [`.github/orchestrator/runner-entrypoint.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-entrypoint.sh) | dockerd bring-up, GPU check, single-job runner | +| [`.github/orchestrator/config.example.yaml`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/config.example.yaml) | Every tunable: pool, platform, resources, limits, poll intervals | +| [`.github/orchestrator/setup.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/setup.sh) | One-time orchestrator host install | +| [`tests/conftest.py`](https://github.com/castacks/AirStack/blob/main/tests/conftest.py) | `airstack_env` fixture, collection order, `MetricsRecorder` | +| [`tests/parse_metrics.py`](https://github.com/castacks/AirStack/blob/main/tests/parse_metrics.py) | Report generation and the regression gate | +| [`tests/run_summary.py`](https://github.com/castacks/AirStack/blob/main/tests/run_summary.py) | `summary.txt` generation | + +## See also + +- [System Tests](../../../../tests/README.md) — marks, fixtures, metrics, and every CLI flag. +- [Unit Testing](unit_testing.md) — the co-location and proxy pattern for package-level tests. +- [End-to-End Testing](end_to_end_testing.md) — the fixed-trajectory benchmark in depth. +- [CI/CD Orchestrator](../../../../tests/ci-cd-orchestrator.md) — admin setup, rotation, and break-glass procedures. +- [AirStack on OSMO](../../../tutorials/airstack_on_osmo.md) — interactive GPU dev pods on the same pool. diff --git a/docs/development/intermediate/testing/end_to_end_testing.md b/docs/development/intermediate/testing/end_to_end_testing.md index 6cddf945b..794355ec9 100644 --- a/docs/development/intermediate/testing/end_to_end_testing.md +++ b/docs/development/intermediate/testing/end_to_end_testing.md @@ -64,7 +64,7 @@ Each run sweeps: | Parameter | CLI flag | Default | | --------- | -------- | ------- | -| Simulator | `--sim` | `msairsim,isaacsim` | +| Simulator | `--sim` | `isaacsim` (`msairsim` opt-in) | | Robot count | `--num-robots` | `1,3` | | Repeat count | `--stress-iterations` | `1` | | Trajectory type | `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | @@ -320,7 +320,7 @@ airstack test -m autonomy \ | Option | Default | Description | | ------ | ------- | ----------- | -| `--sim` | `msairsim,isaacsim` | Comma-separated sim targets | +| `--sim` | `isaacsim` | Comma-separated sim targets (`msairsim` opt-in) | | `--num-robots` | `1,3` | Comma-separated robot counts | | `--stress-iterations` | `1` | Repeat count per `(sim, num_robots)` | | `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | Trajectory sweep | @@ -431,7 +431,7 @@ Action server: `/{robot_name}/tasks/fixed_trajectory` — see also [Tasks and Ta | PX4 ready timeout | Sim not running, GPU issue | Check `nvidia-smi`, Isaac `omni_pass.env` | | `trajectory_success = 0` | Tracker stall or timeout | Check trajectory_controller logs; rebuild the workspace (`-m build_packages`) | | Cross-track error >> 5 m | Wrong tracker params or frame bug | Compare launch params; check world-frame transform | -| Tests run for hours | Default `--sim` and `--num-robots` sweep | Pin `--sim isaacsim --num-robots 1 --stress-iterations 1` | +| Tests run for hours | Default `--num-robots 1,3` (and `--sim msairsim` if opted in) | Pin `--sim isaacsim --num-robots 1 --stress-iterations 1` | | Unknown mark warning `autonomy` | Mark not in `pytest.ini` | Harmless; filter still works | --- diff --git a/docs/development/intermediate/testing/index.md b/docs/development/intermediate/testing/index.md index c23195477..07c5fb887 100644 --- a/docs/development/intermediate/testing/index.md +++ b/docs/development/intermediate/testing/index.md @@ -87,4 +87,4 @@ airstack test -m "build_packages or autonomy" \ - [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, co-located tests, CI workflow - [Testing frameworks](testing_frameworks.md) — `colcon test`, rostest patterns - [Integration testing](integration_testing.md) -- [CI/CD](ci_cd.md) — pipeline overview +- [CI/CD Pipeline on OSMO](ci_cd.md) — how CI runs the full stack on ephemeral GPU pods: architecture, triggers, what each mark catches, and the metrics regression gate diff --git a/gcs/docker/gcs-base-docker-compose.yaml b/gcs/docker/gcs-base-docker-compose.yaml index c8ac5200e..8894bcb9f 100644 --- a/gcs/docker/gcs-base-docker-compose.yaml +++ b/gcs/docker/gcs-base-docker-compose.yaml @@ -7,8 +7,10 @@ services: dockerfile: docker/Dockerfile.gcs tags: - &gcs_image ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:v${VERSION}_gcs + - &gcs_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_gcs cache_from: - *gcs_image + - *gcs_cache command: > bash -c " ssh service restart; diff --git a/mkdocs.yml b/mkdocs.yml index a36d4c75f..4b77be708 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -70,6 +70,7 @@ nav: - Unit Testing: docs/development/intermediate/testing/unit_testing.md - System Tests: tests/README.md - End-to-End Testing: docs/development/intermediate/testing/end_to_end_testing.md + - CI/CD Pipeline: docs/development/intermediate/testing/ci_cd.md - CI/CD Orchestrator: tests/ci-cd-orchestrator.md - Frame Conventions: docs/development/intermediate/frame_conventions.md - Docker Build Profiles: docs/development/intermediate/docker-build-profiles.md diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index 3ff968750..ad73100eb 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -168,6 +168,12 @@ RUN pip3 install --break-system-packages --ignore-installed \ kornia \ typeguard==2.13.3 +# Keep pytest < 8.1. ROS Jazzy launch_testing still implements +# pytest_pycollect_makemodule(path=...), which pluggy rejects after pytest 8.1 +# removed the py.path hook argument (PluginValidationError on colcon test). +RUN python3 -m pip install --no-cache-dir --break-system-packages \ + "pytest>=7.4,<8.1" + # Install MACVO Python dependencies (skipped if SKIP_MACVO=true) RUN if [ "${SKIP_MACVO}" != "true" ]; then \ pip3 install --break-system-packages \ diff --git a/robot/docker/docker-compose.yaml b/robot/docker/docker-compose.yaml index 31b3e368a..cc2bbb6c2 100644 --- a/robot/docker/docker-compose.yaml +++ b/robot/docker/docker-compose.yaml @@ -17,8 +17,13 @@ services: ROS_DISTRO: jazzy tags: - *desktop_image + # Floating tag republished by docker-build.yml on main/develop. The + # versioned tag above never exists yet on a PR (check-version-increment + # forces VERSION up), so it can only ever be a cache miss. + - &desktop_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-x86-64_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *desktop_image + - *desktop_cache # we use tmux sd-keys so that the session stays alive environment: - ROBOT_NAME_SOURCE=container_name # see .bashrc @@ -126,8 +131,10 @@ services: ROS_DISTRO: jazzy tags: - *voxl_image + - &voxl_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-voxl_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *voxl_image + - *voxl_cache environment: - ROBOT_NAME_SOURCE=hostname # see .bashrc - AUTOLAUNCH=${AUTOLAUNCH:-true} @@ -158,8 +165,10 @@ services: DUSTYNV_IMAGE: dustynv/ros:jazzy-ros-base-r36.4.0-cu128-24.04 tags: - *l4t_stack_base_image + - &l4t_stack_base_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-l4t-stack-base_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *l4t_stack_base_image + - *l4t_stack_base_cache # =================================================================================================================== # for running on an NVIDIA jetson (linux for tegra) device @@ -184,9 +193,12 @@ services: ROS_DISTRO: jazzy tags: - *l4t_image + - &l4t_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-l4t_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *l4t_image + - *l4t_cache - *l4t_stack_base_image + - *l4t_stack_base_cache # we use tmux send-keys so that the session stays alive ipc: host command: > @@ -246,8 +258,12 @@ services: L4T_MINOR: 4 L4T_PATCH: 0 IMAGE_NAME: dustynv/ros:jazzy-desktop-r36.4.0-cu128-24.04 + tags: + - *zed_l4t_image + - &zed_l4t_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_zed-l4t-36-4-0_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *zed_l4t_image + - *zed_l4t_cache command: > bash -c "ssh service restart; tmux new -d -s zed_driver && diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg index 55f87f13c..7e02f4b35 100644 --- a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg @@ -10,3 +10,13 @@ python_classes = Test* python_functions = test_* markers = unit: Hermetic unit tests (no ROS stack required) + linter: ament copyright/flake8/pep257 (run separately, not via colcon test) + copyright: ament_copyright + flake8: ament_flake8 + pep257: ament_pep257 +# colcon test / PYTEST_ADDOPTS -m is dropped by ament pytest. Ignore linter +# modules here so only unit tests run. +addopts = + --ignore=test/test_copyright.py + --ignore=test/test_flake8.py + --ignore=test/test_pep257.py diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py new file mode 100644 index 000000000..c985d5923 --- /dev/null +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py @@ -0,0 +1,7 @@ +# Skip ament linter modules during pytest/colcon test. +# PYTEST_ADDOPTS -m is not forwarded by ament pytest; collect_ignore is. +collect_ignore = [ + "test_copyright.py", + "test_flake8.py", + "test_pep257.py", +] diff --git a/simulation/isaac-sim/docker/Dockerfile.isaac-ros b/simulation/isaac-sim/docker/Dockerfile.isaac-ros index 92ac63fdf..69bbd8117 100644 --- a/simulation/isaac-sim/docker/Dockerfile.isaac-ros +++ b/simulation/isaac-sim/docker/Dockerfile.isaac-ros @@ -154,11 +154,23 @@ RUN sed -i \ 's|param set-default IMU_INTEG_RATE 250|param set-default IMU_INTEG_RATE ${PX4_IMU_INTEG_RATE:-250}|' \ /isaac-sim/PX4-Autopilot/ROMFS/px4fmu_common/init.d-posix/px4-rc.simulator -# install px4 dependencies and build -# LD_LIBRARY_PATH= so apt/openssl use the system libcrypto, not isaac-sim's older -# bundled one (which breaks the ca-certificates postinst). Cleared for this step only. -RUN cd PX4-Autopilot && \ - LD_LIBRARY_PATH= DEBIAN_FRONTEND=noninteractive ./Tools/setup/ubuntu.sh +# Install PX4 host deps and build SITL. +# Match ms-airsim: skip NuttX + Gazebo — Isaac Sim is the simulator, and those +# toolchains are heavy. PX4's ubuntu.sh still apt-installs +# software-properties-common whenever /.dockerenv is present; on the nvcr.io +# Isaac base that package's configure step races with a half-configured +# ca-certificates/launchpadlib chain and fails with dpkg exit 100. Reconfigure +# ca-certificates first and drop software-properties-common from the script +# (add-apt-repository is unused on the --no-nuttx/--no-sim-tools path). +RUN apt-get update && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ca-certificates && \ + update-ca-certificates && \ + dpkg --configure -a || true && \ + sed -i '/software-properties-common/d' PX4-Autopilot/Tools/setup/ubuntu.sh && \ + cd PX4-Autopilot && \ + DEBIAN_FRONTEND=noninteractive ./Tools/setup/ubuntu.sh --no-nuttx --no-sim-tools && \ + rm -rf /var/lib/apt/lists/* + # build px4 RUN cd PX4-Autopilot && \ make px4_sitl diff --git a/simulation/isaac-sim/docker/docker-compose.yaml b/simulation/isaac-sim/docker/docker-compose.yaml index dfd699aa4..50a9df16d 100644 --- a/simulation/isaac-sim/docker/docker-compose.yaml +++ b/simulation/isaac-sim/docker/docker-compose.yaml @@ -8,8 +8,13 @@ services: dockerfile: docker/Dockerfile.isaac-ros tags: - *image_tag + # Floating tag republished by docker-build.yml on main/develop. The + # versioned tag above never exists yet on a PR (check-version-increment + # forces VERSION up), so it can only ever be a cache miss. + - &cache_tag ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_isaac-sim cache_from: - *image_tag + - *cache_tag container_name: isaac-sim entrypoint: "" command: > diff --git a/simulation/ms-airsim/docker/docker-compose.yaml b/simulation/ms-airsim/docker/docker-compose.yaml index 3a4035374..9b49c9098 100644 --- a/simulation/ms-airsim/docker/docker-compose.yaml +++ b/simulation/ms-airsim/docker/docker-compose.yaml @@ -8,8 +8,10 @@ services: dockerfile: Dockerfile tags: - *ms_airsim_image + - &ms_airsim_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_ms-airsim cache_from: - *ms_airsim_image + - *ms_airsim_cache container_name: ms-airsim entrypoint: "" command: /root/entrypoint.sh diff --git a/tests/README.md b/tests/README.md index 6172d1e98..558c3fe15 100644 --- a/tests/README.md +++ b/tests/README.md @@ -221,7 +221,7 @@ pytest tests/ -m sensors \ | Option | Default | Description | |--------|---------|-------------| -| `--sim` | `msairsim,isaacsim` | Comma-separated sim targets | +| `--sim` | `isaacsim` | Comma-separated sim targets (`msairsim` opt-in) | | `--num-robots` | `1,3` | Comma-separated robot counts | | `--stress-iterations` | `3` | Up/down cycles per (sim, num_robots) config | | `--stable-duration` | `120` | Seconds ``test_stable`` / ``test_sensor_streams_stable`` poll for | @@ -529,6 +529,11 @@ Regressions are flagged with :red_circle:, improvements with :green_circle:. ## CI/CD Integration +!!! note "Full pipeline guide" + For the end-to-end picture — architecture diagrams, job lifecycle, trigger + reference, what each mark catches, and how to fold CI into your development + loop — see **[CI/CD Pipeline on OSMO](../docs/development/intermediate/testing/ci_cd.md)**. + ### Workflow: `system-tests.yml` [`.github/workflows/system-tests.yml`](../../../../.github/workflows/system-tests.yml) runs on: @@ -549,7 +554,7 @@ Regressions are flagged with :red_circle:, improvements with :green_circle:. #### Jobs -**`run-tests`** runs on a freshly-spawned ephemeral OpenStack instance (`[self-hosted, airstack-ephemeral]`). The instance is provisioned per-job by the orchestrator described below and destroyed once the job completes. It installs dependencies, runs pytest, and uploads `tests/results/` as an artifact named `test-results--` with 90-day retention. +**`run-tests`** runs on a freshly-spawned ephemeral OSMO pod (`[self-hosted, airstack-ephemeral]`). The pod is submitted per-job by the orchestrator described below and destroyed once the job completes. It installs dependencies, runs pytest, and uploads `tests/results/` as an artifact named `test-results--` with 90-day retention. **`report`** runs on `ubuntu-latest` after `run-tests` (even if it failed). It: @@ -565,9 +570,9 @@ The workflow uses [`dawidd6/action-download-artifact@v6`](https://github.com/daw --- -## CI/CD Orchestrator (OpenStack-backed ephemeral runners) +## CI/CD Orchestrator (OSMO-backed ephemeral runners) -AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they execute on **truly ephemeral OpenStack instances** spawned per-job by an orchestrator. Each test job gets a fresh VM that is destroyed once the job completes — no Docker layer carryover, no leaked containers, no shared host state. +AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they execute on **truly ephemeral [NVIDIA OSMO](https://nvidia.github.io/OSMO/) pods** submitted per-job by an orchestrator. Each test job gets a fresh GPU pod that is destroyed once the job completes — no Docker layer carryover, no leaked containers, no shared host state. (This replaced an OpenStack-Nova backend; the GitHub side and the per-job-destroy model are unchanged.) ### Architecture @@ -576,19 +581,19 @@ AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they │ Orchestrator VM (airstack-ci-cd-orchestrator) │ │ • polls GitHub for queued workflow_jobs │ │ • mints single-use JIT runner tokens │ -│ • spawns / reaps ephemeral instances via OpenStack Nova │ -│ • holds the GitHub PAT and OpenStack application credential│ +│ • submits / reaps ephemeral OSMO workflows via osmo CLI │ +│ • holds the GitHub PAT and OSMO service-account token │ └────────────┬───────────────────────────────────┬─────────────┘ │ │ ▼ ▼ ┌──────────────────────────────┐ ┌────────────────────────────────┐ │ Ephemeral worker (per job) │ │ GitHub Actions queue │ -│ Image: Ubuntu-24.04-GPU- │ │ workflow_job status=queued │ -│ Headless │ │ labels: [self-hosted, │ -│ cloud-init bootstraps Docker │ │ airstack-ephemeral] │ -│ + nvidia-container-toolkit + │ └────────────────────────────────┘ -│ GH Actions runner; runs ONE │ -│ job, then is destroyed. │ +│ Prebaked airstack-ci-runner │ │ workflow_job status=queued │ +│ image: Docker + nvidia CTK + │ │ labels: [self-hosted, │ +│ GH runner. Privileged pod │ │ airstack-ephemeral] │ +│ starts dockerd, runs ONE │ └────────────────────────────────┘ +│ job (JIT), then the pod │ +│ is destroyed. │ └──────────────────────────────┘ ``` @@ -596,21 +601,22 @@ AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they | Concern | Mitigation | |---------|------------| -| Cross-job state pollution (Docker cache, dangling networks, leftover artifacts) | Each job runs on a fresh VM. Spent VM is destroyed within ~30 s of job completion. | +| Cross-job state pollution (Docker cache, dangling networks, leftover artifacts) | Each job runs on a fresh OSMO pod, destroyed within ~30 s of job completion. | | Fork PRs executing arbitrary code | Workflow's `if: github.event.pull_request.head.repo.full_name == github.repository` — fork PRs skipped. | -| Runner running as root | The runner runs as the unprivileged `ubuntu` user inside an instance whose only purpose is one job. | -| Docker socket gives root-equivalent access | Bounded to a single one-shot VM. The orchestrator host doesn't expose Docker at all. | +| Runner runs privileged (root) for docker-in-docker | The pod is privileged (needed to run `airstack up`/compose), but it is single-use, scoped to the dedicated CI pool, and only same-repo code ever reaches it. | +| Docker socket gives root-equivalent access | Bounded to a single one-shot pod. The orchestrator host doesn't expose Docker at all. | | Long-lived PAT on the runner host | The PAT lives only on the orchestrator. Workers receive a single-use **JIT runner config** — a base64 token bound to one runner registration. | -| Persistent OpenStack creds tied to a user password | Orchestrator authenticates with an **application credential** (revocable, scoped) instead of `openrc.sh`. | +| Persistent creds tied to a personal account | Orchestrator authenticates with a shared, non-personal **OSMO service-account token** (revocable, scoped to the CI pool), not an individual's login. | ### Setup -The orchestrator service code, cloud-init template, systemd unit, and full setup runbook live in [`.github/orchestrator/`](../../../../.github/orchestrator/). See [`.github/orchestrator/README.md`](ci-cd-orchestrator.md) for: +The orchestrator service code, OSMO runner-workflow template, runner image, systemd unit, and full setup runbook live in [`.github/orchestrator/`](../../../../.github/orchestrator/). See [`.github/orchestrator/README.md`](ci-cd-orchestrator.md) for: -- creating the OpenStack application credential and `clouds.yaml` -- staging the GitHub PAT -- running `setup.sh` on the orchestrator VM -- filling in flavor / network / keypair / security-group in `/etc/airstack-orchestrator/config.yaml` +- obtaining the OSMO service-account token and a dedicated CI GPU pool (with privileged mode enabled) +- building and pushing the runner image (`runner.Dockerfile`) +- staging the GitHub PAT and the OSMO token +- running `setup.sh` on the orchestrator host (installs the `osmo` CLI) +- filling in osmo_url / pool / platform / runner_image / resources in `/etc/airstack-orchestrator/config.yaml` - enabling and verifying the `airstack-orchestrator.service` systemd unit ### Runner labels diff --git a/tests/colcon_unit_test_packages.yaml b/tests/colcon_unit_test_packages.yaml index 65cdd38cc..6d96da1e5 100644 --- a/tests/colcon_unit_test_packages.yaml +++ b/tests/colcon_unit_test_packages.yaml @@ -11,5 +11,7 @@ robot: packages: - natnet_ros2 - lidar_point_cloud_filter - # Skips ament_copyright / flake8 / pep257 on Python packages; run linters separately. - pytest_args: "-m not linter" + # Linter skip lives in lidar_point_cloud_filter setup.cfg + test/conftest.py. + # ament pytest does not honor PYTEST_ADDOPTS -m. + # launch_testing is skipped via PYTEST_DISABLE_PLUGIN_AUTOLOAD in the test. + pytest_args: [] diff --git a/tests/conftest.py b/tests/conftest.py index 62644eb98..203e9d611 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -21,8 +21,9 @@ # ── pytest config / hooks ────────────────────────────────────────────────── def pytest_addoption(parser): - parser.addoption("--sim", default="msairsim,isaacsim", - help="Comma-separated sim targets: msairsim, isaacsim") + parser.addoption("--sim", default="isaacsim", + help="Comma-separated sim targets: isaacsim, msairsim. " + "Default isaacsim; pass --sim msairsim to opt in.") parser.addoption("--num-robots", default="1,3", help="Comma-separated robot counts, e.g. 1,3") parser.addoption("--stress-iterations", type=int, default=1, @@ -67,6 +68,9 @@ def pytest_addoption(parser): parser.addoption("--waypoint-timeout", default="120", help="Per-waypoint time budget (s, odometry clock) in " "test_waypoint_flight. Default: 120") + parser.addoption("--no-image-build", action="store_true", default=False, + help="CI flag: skip image-build in system-tests.yml. " + "Ignored by pytest itself.") def pytest_configure(config): diff --git a/tests/harness/__init__.py b/tests/harness/__init__.py index 05efe2144..d3fcf4e32 100644 --- a/tests/harness/__init__.py +++ b/tests/harness/__init__.py @@ -26,6 +26,7 @@ AIRSTACK_ROOT, COLCON_UNIT_TEST_PACKAGES_YAML, colcon_test_robot_command, + format_pytest_addopts, load_colcon_unit_test_config, repo_path, unit_test_dirs, @@ -44,7 +45,7 @@ __all__ = [ # discovery "AIRSTACK_ROOT", "COLCON_UNIT_TEST_PACKAGES_YAML", "repo_path", - "colcon_test_robot_command", "load_colcon_unit_test_config", + "colcon_test_robot_command", "format_pytest_addopts", "load_colcon_unit_test_config", "unit_test_dirs", "unit_test_files", # session "logger", diff --git a/tests/harness/commands.py b/tests/harness/commands.py index 9b6bc2fb9..7c593e4de 100644 --- a/tests/harness/commands.py +++ b/tests/harness/commands.py @@ -55,8 +55,12 @@ def _run_teed(cmd_list, timeout, log_name=None, env=None, cwd=None): return result -def docker_exec(container, cmd, timeout=60, log_name=None): - full_cmd = ["docker", "exec", container, "bash", "-c", cmd] +def docker_exec(container, cmd, timeout=60, log_name=None, env=None): + full_cmd = ["docker", "exec"] + if env: + for key, value in env.items(): + full_cmd.extend(["-e", f"{key}={value}"]) + full_cmd.extend([container, "bash", "-c", cmd]) return _run_teed(full_cmd, timeout=timeout, log_name=log_name) diff --git a/tests/harness/discovery.py b/tests/harness/discovery.py index c0178a92d..28e03737a 100644 --- a/tests/harness/discovery.py +++ b/tests/harness/discovery.py @@ -3,9 +3,11 @@ Driven by ``tests/colcon_unit_test_packages.yaml``. ``conftest.pytest_configure`` adds ``unit_test_files()`` to the pytest run, and ``pytest_itemcollected`` marks each of those items ``unit`` via ``_is_unit_item``. ament lint tests are excluded here — they run under -``colcon test`` (see ``colcon_test_robot_command``'s ``-m not linter``). +``colcon test`` (linter skip is in package pytest config; see +``colcon_test_robot_command``). """ import os +import shlex from pathlib import Path import yaml @@ -43,20 +45,42 @@ def load_colcon_unit_test_config(workspace="robot"): raise ValueError( f"'{workspace}.packages' is empty in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" ) - return packages, cfg.get("pytest_args", "") + raw_args = cfg.get("pytest_args", []) + if isinstance(raw_args, str): + pytest_args = shlex.split(raw_args) if raw_args else [] + elif isinstance(raw_args, list): + pytest_args = [str(a) for a in raw_args] + else: + raise TypeError( + f"'{workspace}.pytest_args' must be a list or string in " + f"{COLCON_UNIT_TEST_PACKAGES_YAML.name}, got {type(raw_args).__name__}" + ) + return packages, pytest_args def colcon_test_robot_command(workspace="robot"): - """Shell command for colcon test over unit-test packages (robot workspace).""" - packages, pytest_args = load_colcon_unit_test_config(workspace) + """Shell command for colcon test over unit-test packages (robot workspace). + + Pytest flags from the YAML are *not* put on this command. colcon's + ``--pytest-args`` is a single nargs='*' option (last occurrence wins), + and nesting those tokens through ``bash -ic`` also breaks quoting. + Pass them as ``PYTEST_ADDOPTS`` via ``docker_exec(..., env=...)``. + """ + packages, _ = load_colcon_unit_test_config(workspace) pkg_list = " ".join(packages) - cmd = ( + return ( f"colcon test --packages-select {pkg_list} " "--event-handlers console_direct+ --return-code-on-test-failure" ) - if pytest_args: - cmd += f' --pytest-args "{pytest_args}"' - return cmd + + +def format_pytest_addopts(pytest_args): + """Build a PYTEST_ADDOPTS value that pytest will shlex-split back to tokens. + + Do not name this pytest_*: conftest functions with that prefix are treated + as pytest hooks and fail collection (exit code 3). + """ + return " ".join(shlex.quote(a) for a in pytest_args) # Each listed package resolves to its /test dir via these per-workspace globs. @@ -67,7 +91,7 @@ def colcon_test_robot_command(workspace="robot"): # ament lint tests ship in every ROS package's test/ dir and import ament_* at # module load (unavailable outside the built workspace). Skip them here — they run -# under `colcon test` instead (see colcon_test_robot_command's `-m not linter`). +# under `colcon test` instead (package pytest config skips ament linters). _LINTER_TEST_FILENAMES = { "test_copyright.py", "test_flake8.py", "test_pep257.py", "test_pep8.py", "test_xmllint.py", "test_lint_cmake.py", diff --git a/tests/system/test_build_packages.py b/tests/system/test_build_packages.py index 5c54cfee3..aa84b0464 100644 --- a/tests/system/test_build_packages.py +++ b/tests/system/test_build_packages.py @@ -1,10 +1,12 @@ +import shlex from pathlib import Path import pytest from conftest import (AIRSTACK_ROOT, airstack_cmd, colcon_test_robot_command, - docker_exec, load_colcon_unit_test_config, logger, - read_log_tail, wait_for_container) + docker_exec, format_pytest_addopts, + load_colcon_unit_test_config, logger, read_log_tail, + wait_for_container) def _warn_if_prebuilt(*ws_paths): @@ -52,7 +54,7 @@ def test_colcon_test_robot(self): Package list and pytest args come from tests/colcon_unit_test_packages.yaml. Workspace-wide ament linter tests are not gated here. """ - packages, _ = load_colcon_unit_test_config("robot") + packages, pytest_args = load_colcon_unit_test_config("robot") try: result = airstack_cmd("up", "robot-desktop", env_overrides={"AUTOLAUNCH": "false", "DISPLAY": ""}, @@ -69,10 +71,18 @@ def test_colcon_test_robot(self): ) assert build.returncode == 0, f"colcon build (with testing) failed:\n{read_log_tail()}" + # PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 skips launch_testing before + # pytest 8.1+ validates its removed path= hook. Linter skip is in + # the package setup.cfg / test/conftest.py (ament pytest ignores + # PYTEST_ADDOPTS -m). + exec_env = {"PYTEST_DISABLE_PLUGIN_AUTOLOAD": "1"} + if pytest_args: + exec_env["PYTEST_ADDOPTS"] = format_pytest_addopts(pytest_args) test = docker_exec( container, - f"bash -ic '{colcon_test_robot_command('robot')}'", + f"bash -ic {shlex.quote(colcon_test_robot_command('robot'))}", timeout=300, + env=exec_env, ) assert test.returncode == 0, ( f"colcon test failed (packages: {', '.join(packages)}):\n{read_log_tail()}"