Skip to content

Latest commit

 

History

History
294 lines (255 loc) · 19.3 KB

AGENTS.md

File metadata and controls

294 lines (255 loc) · 19.3 KB

AGENTS.md

This file provides guidance to AI coding assistants (Claude Code, Copilot, etc.) when working with code in this repository.

Sourced Assertions

Verify load-bearing factual claims — code, tooling, third-party behaviour — against the code, the repo docs, or a web search before stating them. Flag anything unverified, or a deduction.

Badge in chat output only (never in commit messages, code comments, or committed files), inline right after the claim, with a path:line cite for code/doc:

  • ✅ code / ✅ doc path:line — verified in the code / repo docs
  • ✅ web URL — verified online (bare URL, outside the code span; pin to a commit SHA, not a branch)
  • 💭 deduction — inferred from verified facts
  • ⚠️ unverified — unverifiable, or from training data

Skip badges on restatements, tool output, and descriptions of your own next step.

Project Overview

virtkit — a rootless microVM toolkit shipped as static-musl binaries (vk + the embedded vk-agent, plus the optional vk-registry central server, the vk-runnerctl runner throttle, the experimental vk-hub fleet hub and local web UI, and the experimental vk-gitlab runner that has the hub run GitLab jobs). It boots OCI/Docker images as fast microVMs on its embedded libkrun VMM, gives them a shared LAN with egress over ordinary host sockets (no tap, no bridge, no CAP_NET_ADMIN, no root), and drives commands into them over vsock. The same codebase powers local compose-service VMs and a GitLab custom executor. See README.md for the full feature tour.

Architecture

A Cargo workspace (Cargo.toml, edition 2024) with fifteen crates:

  • vk-core/ — the shared host↔guest library: the wire protocol (messages, framing, addr, net, status, fleetctl), the formats both sides speak (atop, oomkills), plus the runtime helpers both sides build on (exec, forward, pty, dockerignore). Deliberately free of guest-only concerns and of russh, so the host links none of that.
  • vk-driver/ — the host driver (depends on vk-core): image building/conversion (OCI → ext4/initramfs), the compose service runner + control plane, the GitLab executor, the userspace L2 network switch (ARP/DHCP/DNS + transparent TCP/UDP egress via ipstack), and the libkrun VMM (vmm/libkrun_sys; the pinned guest kernel and vk-agent are embedded so vk runs self-contained).
  • vk-agent/ — the guest PID 1 / agent (depends on vk-core): brings a systemd-less guest up (mounts, networking, hostname, virtio-fs, optional SSH) and serves an exec channel over vsock so the host can run commands inside the VM.
  • vk-registry/ — the content-addressed OCI store (lib) plus a standalone server (bin). vk-driver links the lib for its in-process build cache. The vk-registry binary is a central OCI-distribution daemon meant to be shared by every runner: it serves the store over HTTP(S), fronts upstream registries as a pull-through cache (caching only digest-addressed content), coordinates build-once via a lease/heartbeat lock API (/lock), and is a backend for the task build cache. See its DESIGN.md.
  • vk-selfupdate/ — <tool> update for the binaries a user installs by hand: resolve a GitHub release, check the download against the digest published beside it, smoke-test it, and rename it over the running binary. Parameterized by tool so every one of them passes the same gates; the caller supplies only which asset it is and what version it was built as. vk and the vk-registry binary are the callers.
  • vk-fs/ — filesystem objects created private and published whole: mode asked for at creation rather than left to the process-wide umask, rename into place so a name never leads nowhere, a directory resolved once and worked through its descriptor, and a name acted on only where it cannot have become another user's. A leaf with no dependencies but libc and anyhow, so every crate here can use it — vk-runnerctl included.
  • vk-hub-proto/ — what vk tells other programs on its host, such as vk-hub: the list of the host's VMs vk workloads prints; and what vk node and vk-hub exchange in fleet mode: enrollment, the session's messages, protocol version negotiation, and the payloads each side signs or verifies; and the hub's client API and the job spec a producer such as vk-gitlab submits through it; and what vk-registry and vk-hub exchange when the hub manages a registry, and vk-gitlab when it enrolls as a relay. Types and pure functions only — no runtime, transport or crypto.
  • vk-peer/ — what a peer of vk-hub needs to join it and hold a session: its ed25519 identity, enrollment, TLS and the session's opening handshake, release keys, and the pieces of a self-update on trial (download under the peer's signature, publish by rename, fexecve with alarm(2) armed). vk node uses it; each peer keeps its own messages and state.
  • vk-oidc/ — the OIDC relying party vk-registry and vk-hub sign people in with: discovery checked against the configured issuer, the authorization URL with state and PKCE, the code exchange and UserInfo, and the checks on an [oidc] table's issuer and client-secret file. Each caller keeps its own routes, cookies and sessions. Its fake-idp feature is an in-process provider for their tests.
  • vk-hub/ — the hub (bin, experimental). vk-hub serve is the fleet hub: it enrolls vk nodes with single-use tokens, pins each node's ed25519 key, holds their WebSocket sessions and keeps their inventory and heartbeats in a redb database, along with the desired state and commands — drain, quarantine, update, reset — it steers them with; it holds the vk releases they update to, and rolls a release out a wave at a time. vk-hub serve also serves a web UI of the fleet when ui_addr is set; vk-hub local serves the same UI for the VMs of the machine it runs on, read from vk workloads --watch. Both sign people in with single-use links and keep an audit log. Built like the vk-registry server (hyper, rustls on ring); operators reach it through a private unix socket and the web UI. See docs/fleet-design.md and docs/fleet-prototype.md.
  • vk-gitlab/ — the GitLab runner for a fleet (lib + bin, experimental): gitlab-runner's GitLab-facing side, reimplemented from gitlab-runner v19.5.0 (MIT, see NOTICE). It asks GitLab for jobs only once the hub has reserved room, translates each into vk-hub-proto's job spec, submits it through the hub's client API, and copies the job's output and outcome back into GitLab's trace and state; a state file per runner lets a restarted daemon pick its jobs up again. Its tests run against an in-process fake GitLab and fake hub, with fixtures ported from gitlab-runner's own suite. See docs/gitlab-dispatch.md.
  • vk-runnerctl/ — the only component that runs as root, and deliberately the smallest: it sets gitlab-runner's concurrent from a number unprivileged vk leaves in a file, clamped into a range only root can configure. It takes no arguments and no paths from its caller, so granting it NOPASSWD grants nothing else; all the policy lives in vk. Its config edit is also a library, for unprivileged vk to apply to a runner config it owns.
  • vk-tpm/ — the TPM 2.0 engine in Rust behind libkrun's TPM CRB device (Windows guests' tpm=on): command processing, sessions and parameter encryption, hierarchies and dictionary-attack protection, PCRs, hash sequences, objects and keys (RSA, ECC P-256, primaries derived as libtpms derives them), contexts, persistent objects, NV indices, policies, attestation, duplication, EK provisioning, versioned state; crypto from RustCrypto crates only. See docs/tpm-design.md.
  • vk-uefi-vars/ — the UEFI variable service of Windows guests, on the host behind libkrun's uefi-vars device: the variables in an edk2 variable store image, GetVariable, GetNextVariableName, SetVariable and QueryVariableInfo as edk2's SMM variable driver answers them, Secure Boot's authenticated writes (PKCS#7, signature lists), private authenticated variables, variable policies and locks; crypto from RustCrypto crates only. See docs/uefi-vars-design.md.
  • vk-testkit/ — helpers the end-to-end tests (tests/*.sh) run on the host, so a test needs nothing of it but a shell and vk: git-repo writes a bare repository git's dumb HTTP transport can clone. Test-only, never shipped.

libkrun is vendored (its own cargo workspace, locally patched) under third_party/libkrun — see its VENDOR.md for the patch list. Its devices crate depends on vk-tpm and vk-uefi-vars by path, so its Cargo.lock locks their dependencies too: after changing them, run cargo update -p vk-tpm (or -p vk-uefi-vars) in third_party/libkrun as well. CI lints and tests that workspace separately (quality.yml's libkrun job); locally, point ./dev.sh at it with --manifest-path third_party/libkrun/Cargo.toml, e.g. ./dev.sh test --manifest-path third_party/libkrun/Cargo.toml -p krun-devices --lib cmos.

The guest kernel is a vanilla Linux vmlinux built from a vendored config fragment (kernel/); it is pinned and built separately from the binaries.

Development Environment

The release artifacts are built reproducibly inside a pinned devcontainer image (.devcontainer/Dockerfile: the nixos/nix base by digest, the toolchain from .devcontainer/nix/flake.nix pinned by flake.lock). Nix runs only inside that image at build time — no Nix on the host. A musl cross gcc compiles the C our deps vendor (ring, zstd, jemalloc) and links the static-musl binaries; no system C libraries are linked. Release build scripts can use Docker or vk, so no local Rust setup is needed. The fast edit loop below is deliberately vk-only and never invokes Docker.

./build-kernel.sh [--no-cache]      # guest kernel vmlinux -> dist/ (vk or Docker; slow) — run first
./build-firmware.sh [--docker|--no-cache]  # UEFI firmware CLOUDHV.fd -> dist/ (optional; embedded by build.sh)
./build.sh                          # static-musl binaries -> dist/ (vk or Docker)
./build.sh --fast                   # same, but the debug profile -> much faster iteration
./build.sh --only=vk-hub            # compile just these binaries (comma list), not assembled
./build.sh --assemble               # append dist/{vk-agent,vmlinux,CLOUDHV*.fd} to dist/vk-bare -> dist/vk + vk.sha256 + build-info.txt
./dev.sh check                      # type/borrow checking, whole workspace, every target
./dev.sh clippy                     # the lints CI gates on, same scope
./dev.sh fmt [--check]              # rustfmt the workspace with the pinned toolchain
./dev.sh test                       # the whole test suite, doctests included
./dev.sh check -p vk-core           # the same, narrowed to one affected crate
./dev.sh test -p vk-core --lib …    # one module's unit tests (see below)
./dev.sh build -p vk-testkit        # debug binary of a test helper (not `vk`: see build.sh)
./dev.sh shell                      # interactive shell in that same VM
./audit.sh [--deny warnings]        # cargo-audit against the committed Cargo.lock
./sweep.sh [--time 15]              # cargo-sweep stale target/ artifacts (default --installed)
./update.sh                         # bump the pinned Rust toolchain + re-lock the flake
./update-kernel.sh [--lts|--stable] # bump the pinned guest kernel (defaults to LTS)
./update-ci-tools.sh [--latest]     # re-pin vk-hub's built-in CI tools (vk-hub/ci-tools/)

When you need a runnable binary, use ./build.sh --fast (alias --debug). It builds the unoptimized debug profile — a much faster compile that still produces the same static-musl vk with the kernel + agent embedded. The output is not stripped and not reproducible, so it is for iterating only, never a release artifact (it cannot combine with --bootstrap-check). Use a plain ./build.sh for anything you ship or compare bytes against.

Every local (non-release) build also gets line-tables-only debuginfo from [profile.dev], which keeps file:line in panics and backtraces while dropping the bulky per-variable DWARF; dependencies get none at all. When you need a debugger, build with cargo's --profile debugging (./dev.sh check|test accept it too) — build.sh offers only the release and dev profiles.

build.sh compiles dist/vk-bare, then assemble.sh appends dist/vk-agent and dist/vmlinux to produce dist/vk. When present, dist/CLOUDHV.fd and its CLOUDHV_VARS*.fd templates join the payload, read at runtime by vk-driver/src/embed.rs. Both ./build.sh and ./build.sh --fast embed dist/vmlinux and fail when it is absent, so ./build-kernel.sh comes first in a fresh checkout; --no-kernel builds a vk without the embedded kernel instead (it then needs --kernel at runtime, and is not shippable). Only the assembling needs the kernel: ./build.sh --only=vk,vk-agent compiles without it.

Fast edit/check/test loop

During iterative edits, do not run cargo build or release builds: ./dev.sh build is for a non-shipped helper such as vk-testkit, and a runnable vk comes from ./build.sh --fast. Start with ./dev.sh check -p <affected-crate> for type and borrow checking. Then run only the test target and module affected by the change, for example:

./dev.sh test -p vk-core --lib dockerignore::tests
./dev.sh test -p vk-core --test exec disconnect_kills_remote_process
./dev.sh test -p vk-driver --bin vk atop_view::tests

./dev.sh clippy -p <crate> runs CI's Clippy gate for one crate. It defaults to -- -D warnings, and explicit -- lint flags replace that default.

With no arguments, each mode covers the whole workspace. check and clippy add --all-targets, so test code receives the same coverage as CI's Clippy run; fmt adds --all; test keeps Cargo's defaults to include doctests. Run the broader commands before committing. Keep the narrow forms for the edit loop itself — the wide ones take minutes where a scoped run takes seconds.

dev.sh is Docker-free: on first use it boots the pinned build image as a shared vk development VM; later invocations use vk exec, avoiding another image build and boot. It reuses target/ — its RUSTFLAGS match build.sh's exactly, so it shares dependency artifacts with ./build.sh --fast — and rejects optimized invocations. The VM powers itself off after half an hour with no cargo command or open shell, so a forgotten one stops holding memory; ./dev.sh stop ends it immediately. Starting a VM also deletes the stopped VMs of this repository's checkouts that no longer exist. A vk and a flock on PATH are required and there is deliberately no Docker fallback. VK_DEV_CPUS and VK_DEV_MEM size the VM, VK_DEV_IDLE_SECS sets that idle window (0 keeps the VM until it is stopped). Use ./build.sh --fast only when an executable is actually needed for runtime testing. ./dev.sh shell opens an interactive shell in that VM (same user, directory and cargo environment) and holds the VM for as long as the shell runs, for the odd cargo or toolchain command the modes above refuse. That VM boots with nested virtualization where the host allows it and the vk on PATH can ask for it, so the shell can also run what it builds: ./dist/vk run … works inside it, as does ./build.sh --fast --use-virtkit=./dist, their images and boot scratch redirected under /var/tmp/vk on the guest's own disk rather than the RAM-backed /tmp. To search the tree from inside the VM, reach for ugrep and bfs; the image's grep and find are busybox applets.

Cargo commands (pinned toolchain)

The toolchain is pinned in rust-toolchain.toml (musl target, clippy + rustfmt). Run cargo directly if you have it, or inside the devcontainer image to match CI exactly (clippy compiles the workspace, so it needs build.sh's writable cargo home — see .github/workflows/quality.yml). These are the CI-parity commands, run to verify a change; the edit loop above is what to use while iterating.

cargo build --release --workspace
cargo test --workspace                              # tests, e.g. vk-core/tests/exec.rs
cargo fmt --all                                     # prefer ./dev.sh fmt: pinned rustfmt
cargo clippy --workspace --all-targets --locked -- -D warnings

Code Quality Config

  • Rust: rustfmt + clippy, pinned via rust-toolchain.toml (edition 2024). CI runs cargo fmt --check and cargo clippy ... -D warnings.
  • Shell: Bash, set -euo pipefail. Scripts that also run inside the build image (e.g. audit.sh under CI) must stay POSIX-compatible — assume only sh there.
  • Dependency audit: cargo-audit with the RUSTSEC ignore list in .cargo/audit.toml (each entry documented with rationale + residual risk).

CI

  • GitHub Actions (.github/workflows/): ci.yml (lint + test + audit + build on push/PR), release.yml (gate pushes to release, including tests/release-e2e.sh against the built vk, then fast-forward main, push the v* tag and publish the GitHub release; see docs/releasing.md), with reusable quality.yml (lint + test + audit), build.yml (the kernel, the firmware and each binary in parallel jobs, then the job that assembles vk) and rebuild.yml. A release is attested by rebuilding each binary from scratch in a vk microVM and checking it matches the Docker build byte for byte (the per-binary form of build.sh --bootstrap-check).
  • End-to-end suite (tests/): VK=./dist/vk tests/release-e2e.sh runs it against a local build — it needs KVM, e2fsprogs and network — and naming scripts narrows the run. Every other tests/*.sh is picked up by its glob, so adding one gates the next release with no registration step — except windows-*.sh (run with ISO_DIR) and gitlab-e2e.sh, which boots a GitLab CE of its own to test vk-gitlab (run with E2E_GITLAB=1; see docs/gitlab-e2e.md). Helpers live in subdirectories (tests/fleet/, tests/gitlab/), out of the glob's reach.

Reproducibility is load-bearing: the binaries are baked into microVM images. Keep builds byte-deterministic (pinned toolchain/base image, SOURCE_DATE_EPOCH, path remapping). Do not break the pinning when changing build inputs.

Commit Messages

See .agents/commit-message-guidelines.md for the format, scope list, body rules, and changelog rules. In short: one concern per commit, independently buildable; single-line imperative summary (no trailing period) with an optional scope: prefix (e.g. ci:, build-kernel.sh:); a wrapped body only when the diff does not speak for itself, kept high-level. A user-visible change updates CHANGELOG.md in the same commit, pitched even higher-level than the message.

Code Review

Code review is expected on the production branch (main): one concern per commit, every commit independently buildable, and every changed line auditable at a glance. Review against the conventions in .agents/coding-guidelines.md and the message rules in .agents/commit-message-guidelines.md.

Coding Conventions

See .agents/coding-guidelines.md for general conventions, formatting requirements, and per-language guidelines (Rust, Shell).