Skip to content

Latest commit

 

History

History
804 lines (701 loc) · 57.5 KB

gitlab-dispatch.md

File metadata and controls

804 lines (701 loc) · 57.5 KB

GitLab dispatch: the contract

Status: experimental; vk-gitlab, vk-hub and vk node implement it. vk-hub-proto defines the wire types: client for the hub's client API, dispatch for session job messages, and job for the job spec. Serde tests pin their JSON. See GitLab jobs for the design and rationale.

Three parties take part:

  • vk-gitlab — a daemon that is gitlab-runner's GitLab-facing side: it holds the runner tokens, asks GitLab for jobs, writes their traces and states, and submits each job to the hub as a job spec.
  • vk-hub — reserves capacity on nodes, places jobs, relays and stores their output, and reports how they ended.
  • vk node — runs each job in microVMs, stage by stage, and talks to GitLab directly for the job's own transfers: the clone, dependency artifacts, artifact uploads.

Tokens

Credential Held by Used for
runner authentication token (glrt-…) vk-gitlab alone POST /api/v4/jobs/request, /runners/verify
relay key (ed25519, <state_dir>/hub/key.pk8) vk-gitlab alone its session with the hub, which authenticates its client API requests
CI_JOB_TOKEN (the job response's token) vk-gitlab; the job's node, in the spec the daemon: trace, state; the node: clone, artifact upload
dependency tokens (dependencies[].token) the job's node downloading that dependency's artifacts
registry credentials (credentials) the job's node pulling the job's images
node key the node its session, nothing else

gitlab-runner sends the job token, not the runner token, on every per-job endpoint — PUT /jobs/:id, PATCH /jobs/:id/trace, POST /jobs/:id/artifacts — so a node holding the job token holds nothing the job itself does not. The hub sees a spec's secrets in transit and keeps them in memory only, until the node has accepted the job; its records and audit log hold the spec redacted (CiJob::redacted).

Job flow

  1. vk-gitlab queues a reservation request for the runner's placement (POST /v1/reservations, queue: true) and fetches its ticket each time the hub tells it the ticket changed, until the hub grants it: the hub has offered one envelope to a node and the node accepted.
  2. It takes a slot under concurrent and the runner's limit, and claims the grant (POST /v1/reservations/<id>/claim).
  3. It asks GitLab for a job (POST /api/v4/jobs/request), renewing the reservation while the request is outstanding, and reports the admission's outcome to the hub. On 204 it gives the slot and the reservation back and queues again.
  4. On 201 it writes the job to its state file, translates the response into a spec and submits it on the reservation under its admission (POST /v1/jobs).
  5. The hub sends the node start; the node journals the job, converts the reservation into the job's ledger entry and answers accepted.
  6. Seeing the job accepted, vk-gitlab sends GitLab PUT /jobs/:id state=running — the commit of a job taken under two_phase_job_commit. GitLab marks the job running before it answers the request, so the job is the fleet's to run or fail from then on (see Failures).
  7. The node runs the stages, streaming masked output to the hub, which stores it and acks. Told of more output, vk-gitlab reads it from its GitLab offset and patches the trace.
  8. The node uploads the artifacts to GitLab itself, then sends its result once its output is all acked.
  9. vk-gitlab patches the rest of the trace, sends the final PUT /jobs/:id, settles the job with the hub (POST /v1/jobs/<id>/settle) and drops it from its state file.

Admission and reservations

GitLab hands a runner a job only when the runner asks, and a job it has handed over cannot move to another runner. So the daemon holds capacity before it asks, never after, and the hub decides fleet-wide which runner's request gets the room a node frees, using the operator's capacity policy. See capacity arbitration for configuration and monitoring:

  • Queued, not held. Each of a runner's request_concurrency request loops queues a reservation request and follows its ticket without holding a slot under concurrent or limit. Unclaimed tickets end with the session of the connection that queued them (a relay at protocol 2 to 4: with its session, or after 120 seconds without a poll). Only claimed tickets survive a hub restart. Fetching a lost ticket returns 410, and the loop queues again.
  • Arbitrated. The hub offers room to the waiting tickets and the queued jobs of each placement together: a job at its place_within_secs deadline first, then a job holding its reservation, then production work other than probes, then production probes, then normal work — except that after production_probe_burst production probes (default 1) since normal work was last granted room, normal work is owed the next turn ahead of production probes. A hub starts owing it that turn, so after a restart normal work goes first. Within each, the lane holding the least for its weight goes first, the oldest first. Shares are not set aside: room an idle lane leaves goes to the others. A ticket waits while its lane holds its max_slots, its relay its concurrent or its runner its limit — counting its tickets offered, granted or claimed and its jobs, those its heartbeat counts without an admission included — while its runner backs off, and until the relay's current session has sent a heartbeat listing every request it holds. A job submitted without an admission waits on its lane's max_slots too.
  • Production headroom. The capacity policy accepts at most 64 headroom entries. Each reserves slots envelopes of its placement (pool, envelope) for production work. Normal work must leave the fleet that much room, or all remaining room if already short. Entries are checked independently; identical placements use their greatest slot count. An entry with no room on a node does not block work there.
  • Bounded discovery. A request of a runner that took no job in the last busy_secs is a probe, GitLab likely having nothing for it: at most probe_limit probes are out fleet-wide, one per runner. An empty or error outcome backs the runner off for backoff_initial_secs, doubled with each consecutive one up to backoff_max_secs, ±25%; a job outcome clears it. These are capacity policy fields (vk-hub capacity-policy); defaults 60, 1, 1 and 60.
  • Then a reservation. The hub offers the ticket's envelope to a node, least loaded first (Placement). The node decides it through its admission ledger at once — a reservation never queues, and is refused while an earlier job waits — and counts it as granted memory and disk while it lasts. A node that refuses or does not answer within 5 seconds is skipped until every fitting node was tried; they are tried again 2 seconds later. An acceptance grants the ticket: {admission, reservation, node, envelope, lease_secs}, the lease what remains of the node's.
  • Then a claim. The daemon takes a slot — giving the grant back as released when another loop took it meanwhile — lists the admission in its heartbeat's request_admissions, and claims the grant. A grant not claimed before its lease runs out is released. The hub stores a claimed admission, so it outlives a hub restart.
  • Then the job request. The reservation's lease covers the request: a lease of lease_secs (default 90) against a job request the daemon times out at 60 seconds, renewed every lease_secs / 3 while the request is outstanding. A request in flight is never aborted: GitLab may already have assigned the job.
  • Then the outcome. The daemon reports what came of the admission: job, empty (GitLab answered 204), error (the request failed) or released (given back without asking GitLab), retrying until the hub takes it or answers 400, 404, 409 or 410. Any outcome but job ends the admission and releases its reservation. A job's admission moves to the heartbeat's job_admissions and is charged until the job is settled.
  • Then the job. The job is submitted on the reservation at once, naming its admission. The node sizes the job by its own rules — MICROVM_CPUS, MICROVM_MEM, the script's # vk: hint, history — and returns what the job does not need to the ledger; a job needing more than its reservation waits for the rest in the ledger, as an executor job does. The node refuses offers until it is admitted, so smaller reservations cannot take its room. It fails after the node's wait_timeout_secs.

Heartbeats reconcile claims whose outcome was lost. A claimed admission that a heartbeat of the claiming session, received at least 10 seconds after the claim, lists in neither list is reclaimed and its reservation released. After a reconnect or a hub restart the relay's first heartbeat of the new session decides: an admission listed as a job's is one, one not listed is reclaimed.

Idle reservations are bounded: a runner holds at most request_concurrency tickets waiting, offered or granted, and a claimed grant lasts one request plus a renewal. A reservation that ends mid-request — its node lost, or the node's clock ran its lease out — leaves a job that arrives meanwhile without one; the hub then queues it under its admission and places it within its place_within_secs (vk-gitlab asks for 300 seconds), and failing that ends it no_capacity, which the daemon reports as runner_system_failure. That window is the only way a job taken from GitLab waits for capacity.

The lease is a duration on the node's monotonic clock from its acceptance; no wall-clock time crosses the wire. The daemon counts it from when it sent its request, so it always believes the lease ends earlier than the node does.

Placement

A runner in vk-gitlab's configuration names what its jobs ask of a node; every runner of one vk-gitlab takes jobs from the one GitLab server its [gitlab] url names:

[gitlab]
url = "https://gitlab.example.com"

[[runners]]
name = "ci"
token_file = "/etc/vk-gitlab/runner-ci.token"   # glrt-…
envelope = { mem = "16G", cpus = 8, disk = "16G" }
request_concurrency = 1

The runner's lane on the hub selects its pools and nodes (capacity arbitration). A legacy pool in a [[runners]] table is ignored since 0.111, and legacy labels since 0.112, each with a startup warning to remove it. Accepting them lets unattended relay updates keep serving with an older config.

A runner's envelope is what the hub sets aside on a node and counts there for each of its jobs. The node admits the job's guest and its services, each service VM at its own size, so the envelope should cover both: a job whose services take it past its envelope waits on the node for the difference, and one whose guest and services together pass the node's whole budget is refused (Placed jobs on the node).

concurrent and each runner's request_concurrency are at most 128, and a daemon has at most 64 runners: vk-gitlab refuses a configuration past them, and the hub ends the session of a relay whose inventory lists more runners or a larger request_concurrency.

GitLab's job response does not carry the job's tags, and GitLab only gives a runner jobs whose tags it has, so placement is per runner: one runner per class of work, with the tags of that class registered in GitLab, its envelope here, and its lane on the hub. A pool is a set of nodes the hub's operator names (vk-hub nodes pools <id> <a,b>, or on the node's page), which a lane selects. The envelope stays in the daemon's configuration, which it reports to the hub; the hub sets the relay no ceiling on it.

Since 0.112, pools select nodes without labels. The hub ignores labels reported or requested by older relays and reported by older nodes, so once the hub is updated, relays and nodes update in any order. It rejects labels in API key placements and new headroom entries. At startup, it removes stored headroom labels, preserving each entry's pool and auditing the removal.

A node's concurrency ceiling (vk-hub nodes ceiling) and its own executor limit ([executor.schedule] max_concurrency, which the node reports) cap its placed work at the smaller limit. The hub offers and starts nothing on a node whose reservations and unfinished jobs reach the cap. The node enforces it independently, refusing ceiling or concurrency by its own count. Running jobs above a lowered ceiling carry on.

A host runs its own gitlab-runner with the vk executor or takes placed jobs, never both. A node that finds such a runner on its host says so in its report and refuses with runner every new offer and every start without a reservation (a reservation it already holds is renewed and started on); the hub places nothing on it and counts it in no capacity. A host that takes placed jobs sets [node] runner = "none" (unset gives none only where no gitlab-runner unit or binary shows), so the node drains, resets and updates it as one whose runner it supervises.

Among the nodes with room, the hub offers a reservation, or starts a job without one, on the least loaded first, then the one with the most room, then by node ID. A node's load is the largest of: its placed work against that work plus its room (the envelopes it still fits); the vCPUs of its placed work per CPU; and its heartbeat's 1-minute load average per CPU, left out for a node too old to report one. Work therefore spreads across nodes even at moderate load — concurrent builds, and the traffic they put on shared services such as the registry, are not stacked on the one node with slightly more room, a host busy with work of its own takes new work last, and losing a node loses fewer jobs. Memory, disk and the cap bound each node, and capacity (fits) is the sum of every node's room.

A job whose image its node builds from the checkout (dockerfile: or compose:, its own or a service's) goes first to a node that ran a job of the same project with the same such images within that node's image_cache_idle_secs, while that node is lightly loaded: its 1-minute load average, plus the vCPUs of jobs sent to it in the last minute and of this job, per CPU, below the hub's image_affinity_max_load (default 0.7), and without this job no more than image_affinity_max_extra_load (default 0.25) above the lowest such load among candidates. That node boots it in seconds where another spends a minute or two building the image. A job submitted on a reservation on a node without the image then starts on the warm node without it, and the hub releases the reservation. Under heavier load the order above holds.

Where a runner's work goes is its lane's pools in the capacity policy: pool names, and nodes as #node#<node ID> (at most 256), the union of whose members the lane may use. Room is granted only on those nodes, and the job submitted on it runs there; a queued job may move among them, image affinity included. A runner whose lane selects nothing gets no room, its requests and queued jobs waiting for its lane selects no pool or node. A queued job whose reservation is on a node its lane no longer allows has the reservation released and waits for an allowed node, ending at its place_within_secs deadline with its lane's allowed nodes as the reason when nodes had room but none was allowed (see capacity arbitration). A pool a relay's request or job names is ignored; an API key's names where its job goes, a pool or a node's #node#<node ID>, which the key must allow by name: a key limited to pools reaches a node only through those pools.

A relay job its lane no longer allows after it was submitted — the runner unmapped, its project no longer listed, or its branch no longer matching the project's branches — is held: it waits, keeping any reservation it was submitted on so it starts as soon as it is allowed again, or until its deadline, when it ends with no longer allowed by its lane as the reason. A job recorded by a hub from before capacity arbitration, without a scheduling identity, is not held.

Daemon ↔ hub: the client API

JSON over TLS 1.3, on the hub's node listener (addr), under /v1/. A relay at protocol 5, such as vk-gitlab of this release, holds one HTTP/2 connection to the hub, under the private ALPN ID vk-relay/1: its session (below) is a WebSocket stream of it (RFC 8441's extended CONNECT), and its requests are the others. Relays at 2 to 4, API keys, enrollments, release downloads, nodes and registries use HTTP/1.1; the hub never offers h2. On a plaintext loopback listener, a relay's connection opens with HTTP/2's preface instead. Past 4096 client requests in flight on the hub, a request is answered unavailable. Bodies are vk_hub_proto::client's. vk-gitlab is an enrolled relay (GitLab relay enrollment): vk-hub token create --kind gitlab | vk-gitlab join <hub-url> --token - [--ca FILE] pins its key on the hub and keeps it, the hub's URL, its CA and the relay's ID under <state_dir>/hub/ (0700); vk-gitlab leave deletes them. Both are refused while a state file holds a job.

Authentication. A relay's session authenticates its connection: its handshake signs the connection's TLS exporter with the relay's key, and the hub then serves the client API on that connection as that relay, unsigned, while the session is current. Before the session is up, the connection serves the session's CONNECT alone, and must reach it within the time an HTTP/1.1 handshake has; a second CONNECT, any other request, or one on an established connection carrying an authorization, vk-relay, vk-time or vk-signature header, is refused 403 forbidden. Once the session ends (lost, superseded, removed, or the hub handing over), the connection takes no new request (503 unavailable), finishes those it serves and closes. vk-gitlab sends a request made while it has no session once the next is up, and one left unanswered on a lost connection once more on the new one: every request is idempotent. A relay at 2 to 4 instead signs each request, on its own HTTP/1.1 connection: vk-relay carries its ID, vk-time the time, and vk-signature an ed25519 signature over client_request_message — the relay ID, method, path and query, the body's SHA-256, the time, and the TLS connection's exporter; the hub checks that the relay is enrolled and not removed, the time within DOWNLOAD_SKEW_SECS, and the signature. Either way, a TLS-terminating proxy in front of the hub breaks the binding. A relay is held to no pool or envelope ceiling: it already holds the runner tokens and sees every job's secrets.

Session. Besides the client API, vk-gitlab run holds a session with the hub at /v1/gitlab/session, as a node does, redialing it when lost: it reports its version, host, system ID, concurrent, its GitLab server's URL and the SHA-256 of its certificate's public key, and each runner's name, GitLab runner ID, placement, limit, request_concurrency and whether GitLab took its token; every heartbeat, each runner's job requests outstanding and jobs held, with the admission of each (request_admissions, job_admissions), and its last GitLab answer; and each runner's state (polling, waiting_capacity, server_pending, unhealthy, token_refused). A change to requests, jobs or states is sent at once, within 250 ms, besides the periodic heartbeat. The hub answers with its status of the server. A runner asks GitLab for jobs only while that status is approved: not before the first status, nor while the server is pending, rejected or the relay rebound, nor once the hub refused the relay for good. Jobs it already holds run either way.

The hub speaks protocol versions 1 to 5: 1 to 4 on HTTP/1.1, 5 alone on a relay's HTTP/2 connection. Version 2 adds request_concurrency and the admission lists, 3 updates, 4 runners added on the hub, and 5 the one connection and its notices: the hub tells the relay, over the session, when one of its tickets is granted or ends (ticket {ticket}), and when one of its jobs not settled reaches a new revision or more output (job {job, revision, output_len}). A notice carries no data: the relay fetches what changed with the requests below. vk-gitlab speaks 5 only. The hub takes new work only from a relay whose session is at version 2 or later: a version-1 relay keeps its session, and replays its submissions, follows, cancels and settles the jobs it holds, but its capacity and reservation requests and its new jobs are refused 403 forbidden. A hub older than vk-gitlab — one that negotiates HTTP/1.1, or refuses version 5 — is refused: the relay logs that the hub must be upgraded, asks GitLab for no jobs, and dials again every minute or so. Upgrade the hub first, then the relays: a relay whose [hub] allow lists update is updated from the hub's Updates page or vk-hub relays update, its running jobs resumed by the release (docs/gitlab-relay-enrollment.md, "Updates").

Request Body Answer
POST /v1/capacity CapacityRequest 200 Capacity
POST /v1/reservations ReservationRequest 201 ReservationGrant, or 202 ReservationTicket when queued
GET /v1/reservations/tickets/<ticket>[?wait=<secs>] — 201 ReservationGrant, or 202 ReservationTicket while queued
DELETE /v1/reservations/tickets/<ticket> — 204
POST /v1/reservations/<id>/claim — 204
POST /v1/reservations/<admission>/outcome OutcomeRequest 204
POST /v1/reservations/<id>/renew RenewRequest 200 ReservationGrant
DELETE /v1/reservations/<id> — 204
POST /v1/jobs JobSubmission 201 JobView
GET /v1/jobs/<id>?after=<revision>[&wait=<secs>] — 200 JobView
GET /v1/jobs/<id>/output?offset=<n>[&wait=<secs>] — 200 bytes
POST /v1/jobs/<id>/cancel CancelRequest 202 JobView
POST /v1/jobs/<id>/settle — 204
POST /v1/clients ClientHello 204 (API keys of other clients; a relay is refused)

Notices, not long polls. A relay at 5 follows a ticket or a job by fetching it when a notice names it, once when it starts following it, once on each new session (which may have missed a notice), and after 60 seconds without one. It keeps the latest notice per ticket and job, since one can arrive before the answer that names its object. The hub refuses wait, and a capacity request's wait_secs, on a relay's connection with 400 invalid. vk-gitlab keeps at most 4 output reads and 4 job submissions in flight on its connection, so its session never waits behind more bulk than the connection's flow-control windows hold.

Long polls, for API keys and relays at 2 to 4. wait on the GETs, and wait_secs in a capacity request, hold a request for at most 60 seconds: a ticket until it is granted, a capacity answer until its revision passes after, a job view until its revision passes after, an output read until there are bytes past offset or the output is complete. A long poll that times out answers as a plain request would.

Capacity. {runner, placement, after, wait_secs} → {revision, fits}. fits counts envelopes, not nodes: a node with room for two counts two. Advisory — the hub's count, from heartbeats less the reservations and starting jobs they do not show yet, of how many envelopes of the placement its ready nodes could take, each node at most what its cap leaves past the reservations and jobs it holds — and no longer asked by vk-gitlab, which queues instead.

Reservations. {runner, queue, request_id, placement, lease_secs, wait_secs} → {admission, reservation, node, envelope, lease_secs}. The request joins the hub's queue, whose offers filter the nodes of the runner's lane, or of the pool an API key names, on readiness and headroom. With queue, the hub answers 201 with a grant if one is ready at once, else 202 {ticket}; without, it waits up to wait_secs (at least 5 seconds) for a grant, then answers 503 no_capacity. A relay names its runner. A runner may have request_concurrency tickets waiting, offered or granted, a principal 256 unended, the hub 4096; past them, and while a relay's session is not connected, the hub answers 503 unavailable. A grant's admission is what its outcome reports on and its job's submission names.

GET on a ticket answers 202 while it waits and 201 with the grant, its lease_secs what is left of the lease, once a node accepted; 410 reservation_gone for a ticket that ended — cancelled, refused, ended with its session, unpolled (any but a relay at 5's), or granted and not claimed in time — or that the hub does not know. DELETE cancels it, releasing an offered or granted reservation; it answers 204 for a ticket the hub does not know, and 409 once claimed. Both answer 404 for another principal's.

claim takes a grant for a GitLab request: 204, again for one already claimed; 410 for no such grant, or once its reservation or lease is gone. outcome takes {admission, outcome}, outcome one of job, empty, error, released: 204, again for the same outcome; 400 for a body naming another admission than the path; 404 for an admission the hub does not know or of another principal; 409 for an outcome the admission's state does not allow — anything but released before the claim, error or released after its job, another once it ended.

renew extends a lease from now and answers the lease the node granted; a reservation that lapsed, was released or whose node was lost answers 410 reservation_gone. DELETE releases it, and answers 204 for one already gone.

Jobs. {runner, admission, request_id, placement, reservation, place_within_secs, spec} → JobView. A job under an admission must match its runner, lane, placement and reservation (403 forbidden), and an admission takes one job (409 conflict); a job on a queued reservation must name its admission. A new job refused with a 4xx ends its admission as empty. The hub starts the job on the reservation's node — unless a lightly loaded node holds the job's built image (placement), when it releases the reservation and starts the job there; with no reservation, or one that is gone, it places the job as it places a reservation, for up to place_within_secs. A node that refuses the start leaves the job free to place again; a start the node may have taken is never placed twice — see failures. The spec is at most 512 KiB serialized (413 too_large).

JobView is {id, revision, state, node, stage, output_len, cancel, result}: state is queued, starting, running or finished, and result is present once finished, with the node's usage when it sent one (see Result). revision moves with every change but the output's length.

Output. The answer's body is the output from offset, at most 1 MiB, with vk-output-offset (its first byte's offset), vk-output-length (the output's length so far) and vk-output-complete: true when the job has ended and the body reaches its end. An offset past the end answers 416 with vk-output-length. The hub keeps a job's whole output until it is settled, so the daemon can read from any offset — whatever GitLab says it holds.

Cancellation. {mode}: graceful or immediate. A later immediate overrides a graceful; cancelling a finished job changes nothing and answers its view.

Settle. The daemon has delivered the outcome to GitLab; the hub drops the job's output and keeps its record.

Idempotency and retries. request_id — 16 random bytes, hex — makes a create idempotent for a day: the same ID with the same body answers the first answer again, or 410 not_found once the hub has dropped that job from its history; with another body, 409 conflict. A queued request retried answers its ticket, or the ticket's grant. Renew, release, claim, cancel and settle are idempotent by nature, as are a ticket's cancel and an outcome repeated. The daemon retries transport errors and every retryable code with backoff from 1 to 30 seconds and jitter, honouring retry_after_secs, and never retries another 4xx — except job cancels and settles. While handling a job, it retries a cancel or settle after any failure, including a refused relay, with the same backoff until accepted, 400 or 404. Cancel delivery runs alongside the trace; an immediate cancel replaces a graceful one. A settle refused with 409 is retried until the job ends. The job keeps its slot during settlement retries because the hub still charges its admission. When the daemon stops taking jobs for shutdown or update, a pending settle is left to its next start, which cancels and settles reported jobs and resumes running ones.

Status code Meaning Retry
400 invalid unparseable body, malformed ID, a job without a project, an outcome naming another admission than its path; wait or wait_secs on a relay's HTTP/2 connection no
401 unauthorized a bearer key missing or bad; for a relay at 2 to 4, no signature, a bad one or its time too far off, or a relay the hub does not know or removed no
403 forbidden a job of another GitLab server than the relay's, of a relay told rebound, or an API key's GitLab job; a relay below protocol 2 asking for new work; a runner without a lane, or a project or branch its lane does not allow; a job that does not match its admission; on a relay's HTTP/2 connection, a request before its session, a second session, or a request carrying a key or signature no
403 approval_pending a job of a project, or of a server, an operator has yet to approve on the hub no: the job fails
403 project_rejected a job of a project, or of a server, an operator rejected no: the job fails
404 not_found no such job, reservation, grant or admission of this relay's; another principal's ticket no
409 conflict request_id reused with another body; an outcome the admission's state does not allow; an admission already used by a job; a claimed ticket cancelled; a settle of a job not finished no (a settle: yes)
410 reservation_gone lapsed, released, or its node lost; a ticket that ended or the hub does not know reserve again
410 not_found a create retried after its job left the hub's history no
413 too_large spec over 512 KiB no
500 internal — yes
503 no_capacity no node accepted in time yes, after retry_after_secs
503 unavailable hub busy or starting; the reservation queue full; the relay's session not connected, or the session of the connection a request came on ended; an admission being updated yes, after retry_after_secs

An error body is {error, code, retry_after_secs?, approval_url?}. Unknown codes read as other; unknown fields are ignored. When the hub serves a web UI, approval_pending and project_rejected include approval_url: the page where an operator decides on the project's approval, <ui_url>/gitlab/<instance>/projects/<project ID>, or, for its server, GitLab › Overview, <ui_url>/gitlab, which lists the servers pending.

A job refused approval_pending or project_rejected is not held: approval is a human step. vk-gitlab releases its reservation and fails it in GitLab, uncommitted, as runner_configuration_error (script_failure where GitLab does not know it), its trace ending with instructions and a link, for example:

ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub: https://hub.example.com:8444/gitlab/main/projects/12
Retry this job once an operator has approved it there.

Without approval_url (an older hub or one without a UI), the trace uses the enrolled hub URL:

ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub https://hub.example.com.
Retry this job once an operator has approved it (GitLab > Pending).

GitLab's retry: when: runner_system_failure does not retry that reason, so a pending project does not loop. The hub keeps the refusal as the answer to the submission's request_id, so it counts once.

Hub ↔ node: protocol version 3

Version 3 (JOBS) adds one message type each way to the existing session, {"type": "job", "kind": …}, whose payloads are vk_hub_proto::dispatch's. A job message in a session below 3 is a protocol error. PROTOCOL stays at 1 to 2; each side that implements version 3 negotiates from a range of its own reaching it: the hub accepts 1 to 3, so a node offering 3 gets it, and one offering at most 2 is steered and never offered a job.

Hub → node Answer Meaning
offer {reservation, envelope, lease_secs} offer_reply set the envelope aside
renew {reservation, lease_secs} lease extend the lease from now
release {reservation} lease gone give it back
start {job, reservation?, envelope, spec} job run a job
output_ack {job, offset} — output stored durably up to offset
cancel {job, mode} job, then result stop a job
recorded {job} — result stored; stop repeating it
Node → hub Meaning
held {reservations, jobs} everything the node holds, once per session after its report
offer_reply {reservation, reply} accepted {lease_secs} or refused {reason, message?}
lease {reservation, state} held {remaining_secs} or gone {why}: expired, released, started, unknown
job {job, state} accepted, refused {reason, message?}, running {stage, activity?, guest? {cpus, mem_mib}}, finished
output {job, offset, data} output bytes from offset, base64, at most 256 KiB
result {job, result} how the job ended; repeated until recorded
egress {job, report} from version 6: a running job's egress report so far, when it changes and after a reconnect

A running state's guest is the size of the job's VM, once its prepare has sized it; a hub older than it ignores it.

An egress report is cumulative, with a revision: the hub keeps the highest, so one sent again replaces the counts rather than adding to them, and the result's report, the last, has the highest. The node takes it at each stage's start and every 5 seconds, at most once every 2, and sends it when it changed; a session below version 6 gets the result's alone.

Refusal reasons are memory, disk, cpus, not_ready (draining, drained, quarantined, in maintenance), policy, no_reservation, invalid, ceiling (the node's placed jobs not finished and reservations held reach the hub's ceiling, as the node last applied it), concurrency (the same, at the node's own max_concurrency) and runner (the host runs its own gitlab-runner with the vk executor); one the hub does not know reads as other, as ceiling, concurrency and runner do on a hub older than them.

Reservations. A node answers an offer at once from its ledger: granted, or refused with the resource that is short. It holds the entry in the ledger itself, with no job process behind it, and drops it when the lease runs out on its monotonic clock, telling the hub lease gone expired. A lease is at most 600 seconds. A drain refuses new offers and lets held reservations lapse; a quarantine releases them.

Start. The node journals the job before answering accepted, so a start redelivered after a reconnect is answered from the journal and not run twice. A start naming a reservation the node does not hold is admitted like any other ask, without waiting: refused no_reservation when it does not fit.

Output. Offsets count bytes of the output as GitLab will hold it: masked, with section markers, timestamped (unless the job sets FF_TIMESTAMPS false), cut at the trace limit. A stamp is when the node read the line (its start, for a line that spans reads), however late the daemon reads it from the hub. The node sends at most 4 MiB past the last ack, keeps every byte past it on disk, and after a reconnect resends from the offset the hub acked. The hub appends each chunk to the job's output file and acks once it is synced; a chunk overlapping what it holds is trimmed, one leaving a gap is a protocol error.

Result. {failure?, exit_code?, message?, output_len, artifacts, usage?}, sent once every byte up to output_len is acked. artifacts gives each upload's outcome: uploaded, skipped, too_large or failed. usage is what the job used on the node, {wall_ms, cpu_ms?, peak_mem_bytes?, cpus?, mem_mib?}:

  • wall_ms: from the driver's start to the job's end, cleanup included;
  • cpu_ms and peak_mem_bytes: the job's VM supervisor and every process under it — the VMM and its vCPUs, service VMs, the switch, the forwards — read from /proc after the last stage and before cleanup, using the same measurement as the executor's job resource usage trace. CPU time is user plus system, guest execution included. Peak memory is each process's VmHWM summed: an upper bound where they did not peak together. Host-side cache and artifact transfers, and a host checkout, are not counted. Both are absent when no VM was up to read;
  • cpus and mem_mib: the guest's size, from the job's MICROVM_CPUS and MICROVM_MEM clamped by the node's ceilings.

usage is optional, as are all its fields except wall_ms.

Reconnects. After its report, a version-3 node sends held. The hub releases each reservation it does not know, cancels immediate each job it has given up on, and answers each job it follows with output_ack at the offset it holds.

Daemon ↔ GitLab

vk-gitlab speaks GitLab's runner API as gitlab-runner 19.5 does (network/gitlab.go, network/trace.go):

  • Runners. POST /api/v4/runners/verify at start with the glrt- token and a system_id of the daemon's own, persisted.
  • Job requests. POST /api/v4/jobs/request with RUNNER-TOKEN, info and last_update — the X-GitLab-Last-Update of the previous answer, which lets GitLab hold the request until its queue changes. 201 is a job, 204 none, 403 a token GitLab refuses (the runner stops), 429 and 503 back off as Retry-After says.
  • Features. info.features advertises variables, image, services, artifacts, cache, fallback_cache_keys, upload_multiple_artifacts, upload_raw_artifacts, refspecs, masking, raw_variables, artifacts_exclude, multi_build_steps, trace_reset, trace_checksum, trace_size, cancelable, cancel_gracefully, return_exit_code, service_variables and two_phase_job_commit. Not session, terminal, proxy, shared, vault_secrets, service_multiple_aliases (a node's executor takes one alias per service), image_executor_opts, service_executor_opts, native_steps_integration or job_inputs. info.executor is vk, info.shell bash.
  • Commit. PUT /jobs/:id state=running once the node has accepted the job. A job that cannot be placed is failed instead (runner_system_failure).
  • Trace. PATCH /jobs/:id/trace?debug_trace=<CI_DEBUG_TRACE> with JOB-TOKEN and Content-Range: <start>-<end> (inclusive, no unit), at most 1 MiB a patch, every X-GitLab-Trace-Update-Interval seconds (3 by default, capped at 15 minutes). 202 moves the offset on; 416 moves it to the end of the response's Range: 0-<n>, which is also how a restarted daemon finds where to resume; 404 or 403 abort the job. The trace preserves the node's output byte for byte. Errors for jobs the daemon refuses or cannot decode, submit or place use gitlab-runner's logger timestamp format (stream 00, stdout), unless the job's FF_TIMESTAMPS is false. The state file preserves this flag across restarts.
  • Keep-alive. With no new output for 30 seconds, PUT /jobs/:id state=running with the trace's crc32:<hex> checksum and byte size.
  • Final update. Once the hub reports the job finished and its output complete and every byte is patched: PUT /jobs/:id with state success or failed, failure_reason, exit_code and output {checksum, bytesize}. 202 and 412 are asked again after the update interval, with backoff up to an hour, as gitlab-runner's final update does.
  • Cancellation. Every PATCH and PUT answer carries Job-Status. canceling asks the hub for a graceful cancel; canceled or failed, or a 403, an immediate one, after which the daemon writes nothing more for the job.

The daemon keeps one state file per runner, 0600, recording each job it has taken — GitLab job ID, job token, hub job ID, request_ids — before it submits or commits it. A restarted daemon resumes each from the hub's view and GitLab's trace offset; a job it had not yet submitted is submitted with the same request_id.

The state file is <state_dir>/<runner name>.json. state_dir defaults to /var/lib/vk-gitlab and is created 0700; the daemon refuses to start when group or others can access it. A daemon run as an unprivileged service user cannot create the default, so create it for that user before the first start: install -d -m 0700 -o <user> -g <group> /var/lib/vk-gitlab. A record stays until the hub has settled its job, including a job already reported to GitLab or abandoned before submission. A runner's name defaults to its token's short form, so a runner whose token may change needs an explicit name, or the new token starts an empty file and the old one's jobs are never resumed; the daemon warns at start about state files no configured runner owns.

Hub jobs and reservations belong to the relay that created them, and the hub answers not_found for them to any other. The state file records the relay's ID, and a daemon enrolled as another relay refuses to start while the file holds jobs, as it does a file of a vk-gitlab from before relays. Before enrolling again (join --replace), stop the daemon (SIGTERM) and let its running jobs finish within shutdown_timeout; join and leave are refused while a job is still recorded, until it ends or the file is removed, which gives the job up to GitLab's timeout.

The daemon does not rotate runner tokens. It verifies the token at start and daily, warns from a week before GitLab's token_expires_at, and logs an error once it has expired. A verify GitLab gives no answer to is retried, backing off up to five minutes. On a fleet, a runner asks for jobs only once its session has sent the hub an inventory listing it verified, which it sends without coalescing: the hub refuses an unverified runner's requests.

Failures

A job GitLab handed over cannot be given back. GitLab ignores two_phase_job_commit: it marks the job running before it answers the job request (Ci::RegisterJobService#assign_runner!), PUT /jobs/:id takes only running, success and failed, and a running job with no update for 30 minutes is dropped as no_updates_running. So a job the fleet cannot run is failed, never left uncommitted.

The hub never retries a GitLab job. It places a job again only while no node can have started it: after a refused start, or when the reservation it was sent to is gone and no start went out. A start whose answer was lost waits for the node's held to say whether it took it.

How it ended JobResult.failure GitLab failure_reason Older GitLab
success — — —
a step's non-zero exit script script_failure
past timeout_secs timeout job_execution_timeout
image or service image not pulled or built image_pull image_pull_failure runner_system_failure
unsupported job, refused by node policy configuration runner_configuration_error script_failure
clone, artifact or cache transfer failed external_dependency runner_external_dependency_failure runner_system_failure
VM, agent or host failure system runner_system_failure
node stopped or drained hard mid-job interrupted runner_interrupted unknown_failure
canceled canceled unknown_failure: GitLab keeps the job canceled
not placed within place_within_secs no_capacity runner_system_failure
node lost lost runner_system_failure

"Older GitLab" applies when the job's features.failure_reasons lacks the reason; the fallback follows gitlab-runner's common/failure_reason_mapper.go (job::gitlab_failure_reason). GitLab's retry: rules then decide whether the job runs again, on whichever runner asks.

A node is lost for a job once it has been unreachable for job_lost_after (default 5 minutes) past the hub's unreachable threshold. The hub ends the job lost; the daemon reports it; when the node comes back, its held names the job and the hub cancels it immediate, and its later uploads fail on a token GitLab has invalidated. A node out of touch for less keeps running the job and buffering its output, which the trace catches up on.

A hub restart keeps job records, outputs and offsets, and loses specs and reservations: a job not yet accepted by a node ends lost, and a reservation the hub no longer knows is released when its node reports it. A daemon restart loses nothing it had written to its state file.

Artifacts and caches

Artifact uploads go from the node to GitLab, with the job token, as gitlab-runner's artifacts-uploader does from inside the build environment. Relaying them through the hub and the daemon would put every byte through two more hops, through a session that carries control and telemetry only and whose messages are capped at 1 MiB, for no gain in trust: the node already holds the token that authorizes the upload. The cost is that the node, not the daemon, sees GitLab's answer, so it reports each upload's outcome in the result and in the trace. The node follows gitlab-runner's upload rules: POST /jobs/:id/artifacts multipart with the archive in field file, query artifact_format, artifact_type and expire_in, a 307's Location followed, 413 not retried, 503 retried after Retry-After. Unlike gitlab-runner, it refuses a 307 from https to http and sends the job token after a 307 only to GitLab's own origin.

Dependency artifacts come from GitLab to the node: GET /jobs/<dependency id>/artifacts with that dependency's token.

Caches go to the registry the node already uses. A cache is an OCI artifact with one tar+zstd layer, in repository ci-cache/<gitlab host>[/<gitlab path>]/<project id>, tagged with the sha256 of its key and the protection of the job's ref, so a protected and an unprotected job never share one, as with gitlab-runner. The last archive written wins; content addressing makes an unchanged cache's upload a blob probe. Each request to the registry is retried as vk's other registry transfers are — a lost or refused connection for up to 90 seconds, a 503 or 429 for up to three minutes — so a registry's update does not lose a job its cache. A request that runs into CACHE_REQUEST_TIMEOUT is not retried, and like a registry still unavailable after its retries, leaves a warning and skips the step's other cache requests. Caches are not shared with gitlab-runner's cache.zips: a project moving to the fleet starts cold.

A job with a registry token. When the hub mints a token for the job's registry (step 4), the job's start carries it, with the repository its ci-cache namespace resolves to. The node then keeps the job's caches in exactly that repository, an absolute path on the registry beside the node's own [registry] repo prefix, with the job's token, each tagged with the sha256 of its key alone: the repository names the protection. It skips the caches, with a warning, when the repository is not a repository name or the token has expired. A job whose start names no cache repository keeps today's caches with the node's credential, except under [registry] guest_credential = "job", where it runs without caches, with a warning in its trace, rather than fall back to the node's credential. The token is masked in the job's trace, and dropped from its journaled start once its result is written. The node reports its [registry] base URL in its inventory, which the hub matches against its registries to mint the token for it.

Under guest_credential = "job" the guest's registry proxy (registry.vk) runs for every job and injects the job's token, never the node's credential. A job with no token gets a proxy that refuses everything; so does every job a local gitlab-runner gives the node, which under "job" reaches the registry only by its own means. Under "job-or-node", which a hub names to nodes taking tokens of their own, placed jobs are treated as under "job", and a local gitlab-runner's job gets the node's guest token instead (registry hub, 4d).

With a token, the proxy forwards the lock API (/lock/) and plain files (/dav/files/) as well as /v2/; the registry enforces the token's grants. Under /dav/files/ only GET, HEAD, PROPFIND, PUT, MKCOL, DELETE and OPTIONS pass: MOVE and COPY would name the proxy in their Destination. The job's steps find the proxy in VK_REGISTRY (http://registry.vk) and their grants in VK_REGISTRY_NAMESPACES, space-separated <prefix>=<rights> with r, w and l for read, write and lock. The proxy needs net.mode = "switch", the only network that serves registry.vk; under any other, a job gets neither the proxy nor these variables.

Archiving and extracting run in the guest, in vk-agent, on the job's tree as the job left it, with the bytes streamed over vsock; the node does the network transfers. No archiver needs to be in the job's image. On the node, an archive is converted as it streams; only what a format needs whole touches the job's disk — a downloaded zip, a cache layer, an artifact's archive before its upload — each capped at 10 GiB, so a job cannot fill the disk other jobs share. A request to the cache's registry is bounded by CACHE_REQUEST_TIMEOUT (minutes, 10 by default), as gitlab-runner bounds its cache transfers.

The job spec

JobSpec is tagged by kind; gitlab_ci is the only kind. vk-gitlab translates GitLab's job response into a CiJob, normalizing what the node would otherwise interpret: defaults are filled in, cache keys are expanded and sanitized (cache/cachekey), aliases are split, empty formats become zip. What only concerns the conversation with GitLab stays with the daemon.

Every field of the response, from gitlab-runner 19.5's common/spec/spec.go:

GitLab field In the spec Notes
id job.id
token token
allow_git_fetch sources.allow_fetch
job_info.name, .stage, .pipeline_id, .project_id, .project_name, .project_full_path, .namespace_id, .root_namespace_id, .user_id job.* project_full_path as project_path
job_info.organization_id, .instance_id, .instance_uuid, .scoped_user_id — ignored: already among the job's variables where the job needs them
job_info.time_in_queue_seconds, .project_jobs_running_on_instance_runners_count, .queue_size, .queue_depth — daemon only: queue metrics
git_info.repo_url sources.repo_url credentials stripped; the node adds the job token through a credential helper
git_info.repo_object_format, .ref, .sha, .before_sha, .ref_type, .refspecs, .depth, .protected sources.*
runner_info.timeout timeout_secs
runner_info.uuid — daemon only: logs
inputs — job_inputs is not advertised
variables[] key, value, public, file, masked, raw variables the daemon appends the runner's own (CI_RUNNER_VERSION, …); the node adds the environment's (CI_BUILDS_DIR, CI_PROJECT_DIR, CI_JOB_STATUS, CI_SERVER_TLS_CA_FILE, …)
steps[] name, script, timeout, when, allow_failure steps
image name, alias, command, entrypoint, ports, variables, pull_policy image resolved under the node's image rules
image.executor_opts.docker.platform, .docker.user, .kubernetes.user image.platform, image.user image_executor_opts is not advertised; when GitLab sends them anyway, these are honoured
services[] services as image; run as a compose group beside the job VM, on one node
artifacts[] name, untracked, paths, exclude, when, artifact_type, artifact_format, expire_in artifacts sent as written; before archiving, the node expands name, paths and exclude from the job's variables and the steps' $GITLAB_ENV, covering $NAME, ${NAME}, ${#NAME} and -, :-, +, :+ of the uploader's shell.Expand. Any other expression is kept as written, with a warning; variables only the image's environment sets expand to nothing
cache[] key, untracked, policy, paths, when, fallback_keys caches keys expanded against the job's and the runner's variables, as gitlab-runner expands them; variables only the node sets (CI_PROJECT_DIR, CI_CONCURRENT_ID, …) expand to nothing
credentials[] of type registry registry_credentials other types are ignored, as by gitlab-runner
dependencies[] id, token, name, artifacts_file dependencies
features.trace_sections, .token_mask_prefixes trace trace.limit_bytes is the runner's output_limit, 4 MiB by default
features.failure_reasons — daemon only: failure mapping
features.tracing — ignored: no OpenTelemetry export
secrets — vault_secrets is not advertised; a job with secrets fails runner_configuration_error
hooks[] hooks pre_get_sources_script and post_get_sources_script, run in the guest around the checkout
run — native_steps_integration is not advertised; a job with run fails runner_configuration_error
policy_options — the daemon writes gitlab-runner's "Job triggered by policy" line to the trace
suspend_options — ignored: no suspended environments
the runner's tls_ca_file server_ca_pem its contents, when set, for CI_SERVER_TLS_CA_FILE; without it CI_SERVER_TLS_CA_FILE is not set

server_url is the runner's configured URL; job.runner_id the ID /runners/verify returned.

Stages

The node runs gitlab-runner's stage order (common/build.go): prepare_executor (admission, image, boot, services), prepare_script, get_sources, restore_cache, download_artifacts, each step_<name>, after_script, archive_cache or archive_cache_on_failure, upload_artifacts_on_success or _on_failure, then cleanup. get_sources uses the host-side checkout the executor already has, honouring GIT_STRATEGY, GIT_DEPTH, GIT_SUBMODULE_STRATEGY, GIT_CHECKOUT and GIT_CLEAN_FLAGS; the other stages reuse the executor's VM, exec and cleanup code. Step scripts are generated as gitlab-runner's bash shell generates them (shells/abstract.go, shells/bash.go), and output is masked and timestamped as its common/buildlogger masks and stamps it: each stream on its own — the node's lines and vk gitlab prepare as stream 00, vk gitlab run as 01, stdout O and stderr E apart — masked before it is stamped, the stamps counted in the trace limit. FF_TIMESTAMPS is read from the job's variables only, on by default as in gitlab-runner 19.5; a runner's [runners.feature_flags] has no equivalent here. Those ports keep gitlab-runner's MIT notice.

Compatibility fixtures

Ported from gitlab-runner v19.5.0 (MIT; the notice is in NOTICE), as tests of the vk-gitlab crate unless noted:

Upstream Ported as
network/gitlab_test.go getRequestJobResponse a JobResponse JSON fixture; parse it and translate it to a CiJob, checking every field the table above maps
common/support.go GetSuccessfulBuild, GetRemoteSuccessfulMultistepBuild, GetFailedBuild, getBuildResponse, getStepsBuildResponse JobResponse fixtures for multi-step, failing and run jobs
common/network_test.go Test_Image_ExecutorOptions_UnmarshalJSON, TestFeaturesInfo_JSONMarshaling executor options and the info.features body
network/gitlab_test.go TestGitLabClient_RequestJob, TestSetLastUpdate, …_TransmitsTwoPhaseJobCommit, TestVerifyRunner, TestTokenIsCreatedRunnerToken request, long-poll and verify against a fake GitLab
TestUpdateJob, TestUpdateJobAsKeepAlive state updates and Job-Status handling
TestPatchTrace, TestRangeMismatchPatchTrace, TestPatchTraceContentRangeAndLength, TestPatchTraceContentRangeHeaderValues, TestPatchTraceUrlParams, TestUpdateIntervalHeaderHandling, TestAbortedPatchTrace, TestJobFailedStatePatchTrace trace patching, offsets and intervals
network/trace_test.go (TestJobFinishTraceUpdateRetry, TestJobDelayedTraceProcessingWithRejection, TestJobMaxTracePatchSize, TestCancelingJobIncrementalUpdate, TestJobChecksum, TestJobBytesize) the trace loop and final update
checkTestArtifactsUploadHandlerContent, TestArtifactsUpload, TestArtifactsDownload upload and download expectations, in vk-driver's node job tests
common/failure_reason_mapper_test.go vk-hub-proto job::tests (built)
cache/cachekey tests key sanitizing, in vk-gitlab
helpers/archives zip tests, helpers/trace masker tests archive and masking, in vk-driver

Open questions

  • Registry scopes. A cache write needs a token scoped to the project's cache repository; vk-registry grants per-account read and write, not per-repository scopes, so caches start with the node's own registry credential.
  • Secrets through the hub. The hub sees a spec's secrets in memory. Encrypting them to the reserved node's pinned key would keep them from the hub, at the cost of the hub no longer placing a job on another node than the one reserved.