Skip to content

ci: land OSMO ephemeral runners and system-test harness on develop - #382

Open
pvkumara wants to merge 19 commits into
developfrom
ci/osmo-orchestrator
Open

ci: land OSMO ephemeral runners and system-test harness on develop#382
pvkumara wants to merge 19 commits into
developfrom
ci/osmo-orchestrator

Conversation

@pvkumara

@pvkumara pvkumara commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What features did you add and/or bugs did you address?

  • Which GitHub issue does this address?

  • Additional description if not fully described in the GitHub issue

  • Please add videos and images to demonstrate the feature. Please upload videos to somewhere persistent (e.g. YouTube or Vimeo) for archival purposes.

How did you implement it?

  • Algorithm details, design decisions, engineering notes, and any other relevant information about the implementation should be included

How do you run and use it?

  • What commands and button presses do you use to manually launch the stack to use your new feature?

  • Write a detailed procedure with EXACT BASH COMMANDS so that another maintainer can replicate and understand the benefits of your feature, and reproduce the videos and images you added above.

Testing with PyTest

  • What pytests did you add to ensure the feature is reliable and robust? What metrics are used?

  • What's the exact command to run the pytests that test your feature? i.e. airstack test -m ...

  • What are the expected results of the tests? What should a maintainer look at to understand whether the test succeeded?

Documentation

  • Was mkdocs.yml updated? (y/n)

  • Do the docs have sufficient scope such that a newcomer can easily reproduce and use your feature?

  • Is there sufficient visual media?

Versioning

pvkumara5 and others added 19 commits July 26, 2026 20:54
@pvkumara5 @cursoragent
ci(orchestrator): migrate ephemeral CI runners from OpenStack to NVID…
ee1c269 
…IA OSMO

Replace the OpenStack-Nova spawn/reap backend with OSMO workflow submission. The GitHub side is unchanged (self-hosted/airstack-ephemeral labels, single-use JIT runner tokens, same-repo fork guard) and the one-job-per-worker destroy-after model is preserved; only the spawn target moved from creating a Nova VM to submitting an OSMO workflow.

orchestrator.py: submit/query/cancel/list via the osmo CLI, job_id -> workflow_id state, re-login-on-auth-failure, orphan sweep via osmo workflow list; drop floating-IP/boot-volume/placement/keypair/security-group logic.

runner.Dockerfile + runner-entrypoint.sh + runner-workflow.yaml.j2: prebaked privileged docker-in-docker + GPU GitHub runner image/task (replaces cloud-init.yaml.j2).

config.example.yaml, setup.sh, airstack-orchestrator.service, requirements.txt: OSMO service-account token auth, install the osmo CLI, drop openstacksdk. Docs (AGENTS.md, tests/README.md, orchestrator README) updated to OSMO.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
ci(orchestrator): pin AirLab OSMO JSON keys and runner image path
88d8c59 
Resolve uuid/live name after submit (OSMO returns name-only + suffix),
default config to the Keycloak-backed airstack pool and Harbor runner
image, and add scripts to build/push airstack-ci-runner on OSMO DinD.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
docs(ci): document the OSMO-backed CI/CD pipeline
165d011 
Fills in the empty ci_cd.md stub with an end-to-end guide to how CI runs
the full AirStack stack on ephemeral OSMO GPU pods: architecture and job
lifecycle diagrams, runner pod anatomy, the three trigger paths, what
each pytest mark catches, the metrics regression gate, the security
model, and layer-by-layer troubleshooting.

Adds the page to the mkdocs nav (it was previously unreachable) and
cross-links it from tests/README.md and the testing index.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): repair Docker builds on OSMO ephemeral runners
f56810d 
Every build_docker and build_packages test failed on the OSMO backend
because the inner dockerd kept its data-root on the pod's overlayfs
rootfs. Linux rejects a directory on overlayfs as an overlay upperdir,
so image pulls still succeeded -- containerd unpacks layers with plain
writes -- while every build step needing a real mount died with
"mount source: overlay ... err: invalid argument", surfacing as
unrelated-looking apt-get and WORKDIR failures.

runner-entrypoint.sh now picks a storage backend by attempting a real
overlay mount rather than trusting the filesystem type, preferring a
loopback ext4 data-root (real overlay2, sparse, dies with the pod) and
falling back to a pod-mounted filesystem, fuse-overlayfs, then vfs.
vfs is a last resort only: it copies the whole filesystem per layer and
would exhaust the storage request on the sim images.

Also bumps the GitHub Actions runner to 2.336.0, since 2.334.0 stops
being able to run jobs on 2026-08-10.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): seed PR Docker builds from a floating cache tag
4583165 
Versioned cache_from entries always miss on PRs because VERSION is forced
up; add a stable cache_* tag published only by docker-build.yml so system
tests can reuse layers without writing the shared cache.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
ci(docker-build): retag unchanged images on VERSION bump
5550b76 
Skip full compose rebuilds when a service's content fingerprint matches
the previous versioned image label; registry-retag instead and only
rebuild services whose Docker inputs changed.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): parse quoted .env values before inline comments
b093327 
docker_image_plan was feeding NUM_ROBOTS with a trailing comment into
compose config, which broke strconv.Atoi for deploy.replicas.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
ci(docker-build): build/push services sequentially
56a60c7 
Publish successful images even when a sibling (e.g. isaac-sim) fails, and
still cosign whatever was retagged or pushed in the same run.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
chore: bump VERSION to 0.19.0-alpha.8 for retag validation
624dec7 
Seeded gcs/ms-airsim/robot images carry content-fingerprint labels; this
bump should registry-retag those digests without rebuilding.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): unblock isaac-sim PX4 apt and robot colcon pytest
d18efe9 
Isaac's PX4 ubuntu.sh fails dpkg configure on the NVIDIA base; pre-fix
ca-certificates, drop software-properties-common, and skip NuttX/Gazebo
like ms-airsim. Pin pytest<8.1 and disable launch_testing for colcon
unit tests so ROS Jazzy's outdated pytest hook no longer aborts CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): pass colcon --pytest-args as separate tokens
81f7910 
A single quoted blob made pytest treat "-p no:launch_testing" as part of
the -m expression, which broke lidar_point_cloud_filter colcon tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): quote colcon pytest args through bash -ic
959380d 
Nested single quotes around 'not linter' terminated the outer bash -ic
string early, so pytest saw 'not' as a path. Use shlex.quote for the
whole command and list-form pytest_args in the YAML.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): pass colcon pytest flags via PYTEST_ADDOPTS
9667722 
colcon --pytest-args is a single nargs='*' option, so repeating it
dropped -p and pytest treated no:launch_testing as a file path.
Set PYTEST_ADDOPTS with docker exec -e instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): rename helper so pytest does not treat it as a hook
8acd7a5 
conftest functions named pytest_* are registered as hooks.
pytest_addopts_env caused PluginValidationError and exit code 3.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
ci: skip image-build for build_packages reruns
241d86a 
Pull and retag cache_* images instead of baking isaac/airsim on every
colcon/pytest iteration. /pytest --no-image-build does the same for
other marks. compose up --no-build when AIRSTACK_NO_IMAGE_BUILD=1.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): disable pytest plugin autoload for colcon tests
865eb0d 
-p no:launch_testing is applied after setuptools entrypoints load, so
pytest 8.1+ still crashes on launch_testing's path= hook. Set
PYTEST_DISABLE_PLUGIN_AUTOLOAD so cache_* robot images (unpinned pytest)
can run lidar tests without a rebuild.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
fix(ci): skip lidar ament linters in package pytest config
40f526f 
PYTEST_ADDOPTS -m not linter never reached ament pytest, so copyright /
flake8 / pep257 still ran after the unit tests passed. Ignore those
modules in setup.cfg and collect_ignore.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
ci: default system tests to isaacsim only
f937b50 
PR-open and bare /pytest were sweeping both sims. Default --sim to
isaacsim; msairsim is opt-in via --sim msairsim.

Co-authored-by: Cursor <cursoragent@cursor.com>
@pvkumara5 @cursoragent
Merge origin/develop into ci/osmo-orchestrator
95a8d87 
Bring the OSMO CI branch onto current develop (harness split, waypoint
flight, notebooks, robot-deployment fixes). Keep Isaac PX4 apt fix,
isaacsim-only --sim default, and colcon pytest env passing.
Bump VERSION to 0.19.0-alpha.13.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pvkumara @pvkumara5