Status: experimental; vk-gitlab, vk-hub and vk node implement it. vk-hub-proto
defines the wire types: client for the hub's client API, dispatch for session job
messages, and job for the job spec. Serde tests pin their JSON. See
GitLab jobs for the design and rationale.
Three parties take part:
vk-gitlab— a daemon that is gitlab-runner's GitLab-facing side: it holds the runner tokens, asks GitLab for jobs, writes their traces and states, and submits each job to the hub as a job spec.vk-hub— reserves capacity on nodes, places jobs, relays and stores their output, and reports how they ended.vk node— runs each job in microVMs, stage by stage, and talks to GitLab directly for the job's own transfers: the clone, dependency artifacts, artifact uploads.
| Credential | Held by | Used for |
|---|---|---|
runner authentication token (glrt-…) |
vk-gitlab alone |
POST /api/v4/jobs/request, /runners/verify |
relay key (ed25519, <state_dir>/hub/key.pk8) |
vk-gitlab alone |
its session with the hub, which authenticates its client API requests |
CI_JOB_TOKEN (the job response's token) |
vk-gitlab; the job's node, in the spec |
the daemon: trace, state; the node: clone, artifact upload |
dependency tokens (dependencies[].token) |
the job's node | downloading that dependency's artifacts |
registry credentials (credentials) |
the job's node | pulling the job's images |
| node key | the node | its session, nothing else |
gitlab-runner sends the job token, not the runner token, on every per-job endpoint —
PUT /jobs/:id, PATCH /jobs/:id/trace, POST /jobs/:id/artifacts — so a node holding
the job token holds nothing the job itself does not. The hub sees a spec's secrets in
transit and keeps them in memory only, until the node has accepted the job; its records
and audit log hold the spec redacted (CiJob::redacted).
vk-gitlabqueues a reservation request for the runner's placement (POST /v1/reservations,queue: true) and fetches its ticket each time the hub tells it the ticket changed, until the hub grants it: the hub has offered one envelope to a node and the node accepted.- It takes a slot under
concurrentand the runner'slimit, and claims the grant (POST /v1/reservations/<id>/claim). - It asks GitLab for a job (
POST /api/v4/jobs/request), renewing the reservation while the request is outstanding, and reports the admission's outcome to the hub. On 204 it gives the slot and the reservation back and queues again. - On 201 it writes the job to its state file, translates the response into a spec and
submits it on the reservation under its admission (
POST /v1/jobs). - The hub sends the node
start; the node journals the job, converts the reservation into the job's ledger entry and answersaccepted. - Seeing the job accepted,
vk-gitlabsends GitLabPUT /jobs/:idstate=running— the commit of a job taken undertwo_phase_job_commit. GitLab marks the job running before it answers the request, so the job is the fleet's to run or fail from then on (see Failures). - The node runs the stages, streaming masked output to the hub, which stores it and acks.
Told of more output,
vk-gitlabreads it from its GitLab offset and patches the trace. - The node uploads the artifacts to GitLab itself, then sends its result once its output is all acked.
vk-gitlabpatches the rest of the trace, sends the finalPUT /jobs/:id, settles the job with the hub (POST /v1/jobs/<id>/settle) and drops it from its state file.
GitLab hands a runner a job only when the runner asks, and a job it has handed over cannot move to another runner. So the daemon holds capacity before it asks, never after, and the hub decides fleet-wide which runner's request gets the room a node frees, using the operator's capacity policy. See capacity arbitration for configuration and monitoring:
- Queued, not held. Each of a runner's
request_concurrencyrequest loops queues a reservation request and follows its ticket without holding a slot underconcurrentorlimit. Unclaimed tickets end with the session of the connection that queued them (a relay at protocol 2 to 4: with its session, or after 120 seconds without a poll). Only claimed tickets survive a hub restart. Fetching a lost ticket returns 410, and the loop queues again. - Arbitrated. The hub offers room to the waiting tickets and the queued jobs of each
placement together: a job at its
place_within_secsdeadline first, then a job holding its reservation, then production work other than probes, then production probes, then normal work — except that afterproduction_probe_burstproduction probes (default 1) since normal work was last granted room, normal work is owed the next turn ahead of production probes. A hub starts owing it that turn, so after a restart normal work goes first. Within each, the lane holding the least for itsweightgoes first, the oldest first. Shares are not set aside: room an idle lane leaves goes to the others. A ticket waits while its lane holds itsmax_slots, its relay itsconcurrentor its runner itslimit— counting its tickets offered, granted or claimed and its jobs, those its heartbeat counts without an admission included — while its runner backs off, and until the relay's current session has sent a heartbeat listing every request it holds. A job submitted without an admission waits on its lane'smax_slotstoo. - Production headroom. The capacity policy accepts at most 64
headroomentries. Each reservesslotsenvelopes of its placement (pool, envelope) for production work. Normal work must leave the fleet that much room, or all remaining room if already short. Entries are checked independently; identical placements use their greatest slot count. An entry with no room on a node does not block work there. - Bounded discovery. A request of a runner that took no job in the last
busy_secsis a probe, GitLab likely having nothing for it: at mostprobe_limitprobes are out fleet-wide, one per runner. Anemptyorerroroutcome backs the runner off forbackoff_initial_secs, doubled with each consecutive one up tobackoff_max_secs, ±25%; ajoboutcome clears it. These are capacity policy fields (vk-hub capacity-policy); defaults 60, 1, 1 and 60. - Then a reservation. The hub offers the ticket's envelope to a node, least loaded first
(Placement). The node decides it through its admission ledger at once — a
reservation never queues, and is refused while an earlier job waits — and counts it as
granted memory and disk while it lasts. A node that refuses or does not answer within 5 seconds is skipped until
every fitting node was tried; they are tried again 2 seconds later. An acceptance grants
the ticket:
{admission, reservation, node, envelope, lease_secs}, the lease what remains of the node's. - Then a claim. The daemon takes a slot — giving the grant back as
releasedwhen another loop took it meanwhile — lists the admission in its heartbeat'srequest_admissions, and claims the grant. A grant not claimed before its lease runs out is released. The hub stores a claimed admission, so it outlives a hub restart. - Then the job request. The reservation's lease covers the request: a lease of
lease_secs(default 90) against a job request the daemon times out at 60 seconds, renewed everylease_secs / 3while the request is outstanding. A request in flight is never aborted: GitLab may already have assigned the job. - Then the outcome. The daemon reports what came of the admission:
job,empty(GitLab answered 204),error(the request failed) orreleased(given back without asking GitLab), retrying until the hub takes it or answers 400, 404, 409 or 410. Any outcome butjobends the admission and releases its reservation. A job's admission moves to the heartbeat'sjob_admissionsand is charged until the job is settled. - Then the job. The job is submitted on the reservation at once, naming its admission. The
node sizes the job by its own rules —
MICROVM_CPUS,MICROVM_MEM, the script's# vk:hint, history — and returns what the job does not need to the ledger; a job needing more than its reservation waits for the rest in the ledger, as an executor job does. The node refuses offers until it is admitted, so smaller reservations cannot take its room. It fails after the node'swait_timeout_secs.
Heartbeats reconcile claims whose outcome was lost. A claimed admission that a heartbeat of the claiming session, received at least 10 seconds after the claim, lists in neither list is reclaimed and its reservation released. After a reconnect or a hub restart the relay's first heartbeat of the new session decides: an admission listed as a job's is one, one not listed is reclaimed.
Idle reservations are bounded: a runner holds at most request_concurrency tickets waiting,
offered or granted, and a claimed grant lasts one request plus a renewal. A reservation that
ends mid-request — its node lost, or the node's clock ran its lease out — leaves a job that
arrives meanwhile without one; the hub then queues it under its admission and places it
within its place_within_secs (vk-gitlab asks for 300 seconds), and failing that ends it
no_capacity, which the daemon reports as runner_system_failure. That window is the only
way a job taken from GitLab waits for capacity.
The lease is a duration on the node's monotonic clock from its acceptance; no wall-clock time crosses the wire. The daemon counts it from when it sent its request, so it always believes the lease ends earlier than the node does.
A runner in vk-gitlab's configuration names what its jobs ask of a node; every runner of one
vk-gitlab takes jobs from the one GitLab server its [gitlab] url names:
[gitlab]
url = "https://gitlab.example.com"
[[runners]]
name = "ci"
token_file = "/etc/vk-gitlab/runner-ci.token" # glrt-…
envelope = { mem = "16G", cpus = 8, disk = "16G" }
request_concurrency = 1The runner's lane on the hub selects its pools and nodes
(capacity arbitration). A legacy pool in a [[runners]] table
is ignored since 0.111, and legacy labels since 0.112, each with a startup warning to remove
it. Accepting them lets unattended relay updates keep serving with an older config.
A runner's envelope is what the hub sets aside on a node and counts there for each of its
jobs. The node admits the job's guest and its services, each service VM at its own size, so
the envelope should cover both: a job whose services take it past its envelope waits on the
node for the difference, and one whose guest and services together pass the node's whole
budget is refused (Placed jobs on the node).
concurrent and each runner's request_concurrency are at most 128, and a daemon has at most
64 runners: vk-gitlab refuses a configuration past them, and the hub ends the session of a
relay whose inventory lists more runners or a larger request_concurrency.
GitLab's job response does not carry the job's tags, and GitLab only gives a runner jobs
whose tags it has, so placement is per runner: one runner per class of work, with the tags
of that class registered in GitLab, its envelope here, and its lane on the hub. A pool is a
set of nodes the hub's operator names (vk-hub nodes pools <id> <a,b>, or on the node's page),
which a lane selects. The envelope stays in the daemon's configuration, which it reports to the
hub; the hub sets the relay no ceiling on it.
Since 0.112, pools select nodes without labels. The hub ignores labels reported or requested
by older relays and reported by older nodes, so once the hub is updated, relays and nodes
update in any order. It rejects labels in API key placements and new
headroom entries. At startup, it removes stored headroom labels, preserving each entry's
pool and auditing the removal.
A node's concurrency ceiling (vk-hub nodes ceiling) and its own executor limit
([executor.schedule] max_concurrency, which the node reports) cap its placed work at the
smaller limit. The hub offers and starts nothing on a node whose reservations and unfinished
jobs reach the cap. The node enforces it independently, refusing ceiling or concurrency
by its own count. Running jobs above a lowered ceiling carry on.
A host runs its own gitlab-runner with the vk executor or takes placed jobs, never both. A node
that finds such a runner on its host says so in its report and refuses with runner every new
offer and every start without a reservation (a reservation it already holds is renewed and
started on); the hub places nothing on it and counts it in no capacity.
A host that takes placed jobs sets [node] runner = "none" (unset gives none only where no
gitlab-runner unit or binary shows), so the node drains, resets and updates it as one whose
runner it supervises.
Among the nodes with room, the hub offers a reservation, or starts a job without one, on the
least loaded first, then the one with the most room, then by node ID. A node's load is the
largest of: its placed work against that work plus its room (the envelopes it still fits);
the vCPUs of its placed work per CPU; and its heartbeat's 1-minute load average per CPU, left
out for a node too old to report one. Work therefore spreads across nodes even at moderate
load — concurrent builds, and the traffic they put on shared services such as the registry,
are not stacked on the one node with slightly more room, a host busy with work of its own
takes new work last, and losing a node loses fewer jobs. Memory, disk and the cap
bound each node, and capacity (fits) is the sum of every node's room.
A job whose image its node builds from the checkout (dockerfile: or compose:, its own or a
service's) goes first to a node that ran a job of the same project with the same such images
within that node's image_cache_idle_secs, while that node is lightly loaded: its 1-minute
load average, plus the vCPUs of jobs sent to it in the last minute and of this job, per CPU,
below the hub's image_affinity_max_load (default 0.7), and without this job no more than
image_affinity_max_extra_load (default 0.25) above the lowest such load among candidates.
That node boots it in seconds where another spends a minute or two building the image. A job
submitted on a reservation on a node without the image then starts on the warm node without it,
and the hub releases the reservation. Under heavier load the order above holds.
Where a runner's work goes is its lane's pools in the capacity policy: pool names, and nodes
as #node#<node ID> (at most 256), the union of whose members the lane may use. Room is granted
only on those nodes, and the job submitted on it runs there; a queued job may move among them,
image affinity included. A runner whose lane selects nothing gets no room, its requests and
queued jobs waiting for its lane selects no pool or node. A queued job whose reservation is on
a node its lane no longer allows has the reservation released and waits for an allowed node,
ending at its place_within_secs deadline with its lane's allowed nodes as the reason when
nodes had room but none was allowed (see capacity arbitration). A
pool a relay's request or job names is ignored; an API key's names where its job goes, a pool
or a node's #node#<node ID>, which the key must allow by name: a key limited to pools reaches
a node only through those pools.
A relay job its lane no longer allows after it was submitted — the runner unmapped, its project
no longer listed, or its branch no longer matching the project's branches — is held: it waits,
keeping any reservation it was submitted on so it starts as soon as it is allowed again, or
until its deadline, when it ends with no longer allowed by its lane as the reason. A job
recorded by a hub from before capacity arbitration, without a scheduling identity, is not held.
JSON over TLS 1.3, on the hub's node listener (addr), under /v1/. A relay at protocol 5,
such as vk-gitlab of this release, holds one HTTP/2 connection to the hub, under the
private ALPN ID vk-relay/1: its session (below) is a WebSocket stream of it (RFC 8441's
extended CONNECT), and its requests are the others. Relays at 2 to 4, API keys, enrollments,
release downloads, nodes and registries use HTTP/1.1; the hub never offers h2. On a
plaintext loopback listener, a relay's connection opens with HTTP/2's preface instead. Past
4096 client requests in flight on the hub, a request is answered unavailable. Bodies are
vk_hub_proto::client's. vk-gitlab is an enrolled relay (GitLab relay
enrollment):
vk-hub token create --kind gitlab | vk-gitlab join <hub-url> --token - [--ca FILE] pins its
key on the hub and keeps it, the hub's URL, its CA and the relay's ID under <state_dir>/hub/
(0700); vk-gitlab leave deletes them. Both are refused while a state file holds a job.
Authentication. A relay's session authenticates its connection: its handshake signs the
connection's TLS exporter with the relay's key, and the hub then serves the client API on that
connection as that relay, unsigned, while the session is current. Before the session is up,
the connection serves the session's CONNECT alone, and must reach it within the time an
HTTP/1.1 handshake has; a second CONNECT, any other request, or one on an established
connection carrying an authorization, vk-relay, vk-time or vk-signature header, is
refused 403 forbidden. Once the session ends (lost, superseded, removed, or the hub handing
over), the connection takes no new request (503 unavailable), finishes those it serves and
closes. vk-gitlab sends a request made while it has no session once the next is up, and one
left unanswered on a lost connection once more on the new one: every request is idempotent. A
relay at 2 to 4 instead signs each request, on its own HTTP/1.1 connection: vk-relay carries
its ID, vk-time the time, and vk-signature an ed25519 signature over
client_request_message — the relay ID, method, path and query, the body's SHA-256, the time,
and the TLS connection's exporter; the hub checks that the relay is enrolled and not removed,
the time within DOWNLOAD_SKEW_SECS, and the signature. Either way, a TLS-terminating proxy in
front of the hub breaks the binding. A relay is held to no pool or envelope ceiling: it already
holds the runner tokens and sees every job's secrets.
Session. Besides the client API, vk-gitlab run holds a session with the hub at
/v1/gitlab/session, as a node does, redialing it when lost: it reports its version, host,
system ID, concurrent, its GitLab server's URL and the SHA-256 of its certificate's public
key, and each runner's name, GitLab runner ID, placement, limit, request_concurrency and
whether GitLab took its token; every heartbeat, each runner's job requests outstanding and jobs
held, with the admission of each (request_admissions, job_admissions), and its last GitLab
answer; and each runner's state (polling, waiting_capacity, server_pending,
unhealthy, token_refused). A change to requests, jobs or states is sent at once, within
250 ms, besides the periodic heartbeat. The hub answers with its status of the server. A runner
asks GitLab for jobs only while that status is approved: not before the first status, nor
while the server is pending, rejected or the relay rebound, nor once the hub refused the
relay for good. Jobs it already holds run either way.
The hub speaks protocol versions 1 to 5: 1 to 4 on HTTP/1.1, 5 alone on a relay's HTTP/2
connection. Version 2 adds request_concurrency and the admission lists, 3 updates, 4 runners
added on the hub, and 5 the one connection and its notices: the hub tells the relay, over
the session, when one of its tickets is granted or ends (ticket {ticket}), and when one of
its jobs not settled reaches a new revision or more output (job {job, revision, output_len}). A notice carries no data: the relay fetches what changed with the requests
below. vk-gitlab speaks 5 only. The hub takes new work only from a relay whose session is at
version 2 or later: a version-1 relay keeps its session, and replays its submissions, follows,
cancels and settles the jobs it holds, but its capacity and reservation requests and its new
jobs are refused 403 forbidden. A hub older than vk-gitlab — one that negotiates HTTP/1.1,
or refuses version 5 — is refused: the relay logs that the hub must be upgraded, asks GitLab
for no jobs, and dials again every minute or so. Upgrade the hub first, then the relays: a
relay whose [hub] allow lists update is updated from the hub's Updates page or vk-hub relays update, its running jobs resumed by the release
(docs/gitlab-relay-enrollment.md, "Updates").
| Request | Body | Answer |
|---|---|---|
POST /v1/capacity |
CapacityRequest |
200 Capacity |
POST /v1/reservations |
ReservationRequest |
201 ReservationGrant, or 202 ReservationTicket when queued |
GET /v1/reservations/tickets/<ticket>[?wait=<secs>] |
— | 201 ReservationGrant, or 202 ReservationTicket while queued |
DELETE /v1/reservations/tickets/<ticket> |
— | 204 |
POST /v1/reservations/<id>/claim |
— | 204 |
POST /v1/reservations/<admission>/outcome |
OutcomeRequest |
204 |
POST /v1/reservations/<id>/renew |
RenewRequest |
200 ReservationGrant |
DELETE /v1/reservations/<id> |
— | 204 |
POST /v1/jobs |
JobSubmission |
201 JobView |
GET /v1/jobs/<id>?after=<revision>[&wait=<secs>] |
— | 200 JobView |
GET /v1/jobs/<id>/output?offset=<n>[&wait=<secs>] |
— | 200 bytes |
POST /v1/jobs/<id>/cancel |
CancelRequest |
202 JobView |
POST /v1/jobs/<id>/settle |
— | 204 |
POST /v1/clients |
ClientHello |
204 (API keys of other clients; a relay is refused) |
Notices, not long polls. A relay at 5 follows a ticket or a job by fetching it when a
notice names it, once when it starts following it, once on each new session (which may have
missed a notice), and after 60 seconds without one. It keeps the latest notice per ticket and
job, since one can arrive before the answer that names its object. The hub refuses wait, and
a capacity request's wait_secs, on a relay's connection with 400 invalid. vk-gitlab
keeps at most 4 output reads and 4 job submissions in flight on its connection, so its session
never waits behind more bulk than the connection's flow-control windows hold.
Long polls, for API keys and relays at 2 to 4. wait on the GETs, and wait_secs in a
capacity request, hold a request for at most 60 seconds: a ticket until it is granted, a
capacity answer until its revision passes after, a job view until its revision passes
after, an output read until there are bytes past offset or the output is complete. A long
poll that times out answers as a plain request would.
Capacity. {runner, placement, after, wait_secs} → {revision, fits}. fits counts
envelopes, not nodes: a node with room for two counts two. Advisory — the hub's count, from
heartbeats less the reservations and starting jobs they do not show yet, of how many envelopes
of the placement its ready nodes could take, each node at most what its cap leaves past the
reservations and jobs it holds — and no longer asked by vk-gitlab, which queues instead.
Reservations. {runner, queue, request_id, placement, lease_secs, wait_secs} →
{admission, reservation, node, envelope, lease_secs}. The request joins the hub's
queue, whose offers filter the nodes of the runner's lane, or of
the pool an API key names, on readiness and headroom. With queue, the hub answers 201
with a grant if one is ready at once, else 202 {ticket}; without, it waits up to wait_secs
(at least 5 seconds) for a grant, then answers 503 no_capacity. A relay names its runner. A
runner may have request_concurrency tickets waiting, offered or granted, a principal 256
unended, the hub 4096; past them, and while a relay's session is not connected, the hub answers
503 unavailable. A grant's admission is what its outcome reports on and its job's submission
names.
GET on a ticket answers 202 while it waits and 201 with the grant, its lease_secs what is
left of the lease, once a node accepted; 410 reservation_gone for a ticket that ended —
cancelled, refused, ended with its session, unpolled (any but a relay at 5's), or granted and
not claimed in time — or that the hub does not know. DELETE cancels it, releasing an offered
or granted reservation; it answers 204 for a ticket the hub does not know, and 409 once
claimed. Both answer 404 for another principal's.
claim takes a grant for a GitLab request: 204, again for one already claimed; 410 for no such
grant, or once its reservation or lease is gone. outcome takes {admission, outcome},
outcome one of job, empty, error, released: 204, again for the same outcome; 400 for a
body naming another admission than the path; 404 for an admission the hub does not know or of
another principal; 409 for an outcome the admission's state does not allow — anything but
released before the claim, error or released after its job, another once it ended.
renew extends a lease from now and answers the lease the node granted; a reservation that
lapsed, was released or whose node was lost answers 410 reservation_gone. DELETE releases
it, and answers 204 for one already gone.
Jobs. {runner, admission, request_id, placement, reservation, place_within_secs, spec} →
JobView. A job under an admission must match its runner, lane, placement and reservation
(403 forbidden), and an admission takes one job (409 conflict); a job on a queued
reservation must name its admission. A new job refused with a 4xx ends its admission as
empty. The hub starts the job on the reservation's node — unless a lightly loaded node holds
the job's built image (placement), when it releases the reservation and starts
the job there; with no reservation, or one that is gone, it places the job as it places a
reservation, for up to place_within_secs. A node that refuses the start leaves the job free
to place again; a start the node may have taken is never placed twice — see
failures. The spec is at most 512 KiB serialized (413 too_large).
JobView is {id, revision, state, node, stage, output_len, cancel, result}: state is
queued, starting, running or finished, and result is present once finished, with the
node's usage when it sent one (see Result).
revision moves with every change but the output's length.
Output. The answer's body is the output from offset, at most 1 MiB, with
vk-output-offset (its first byte's offset), vk-output-length (the output's length so
far) and vk-output-complete: true when the job has ended and the body reaches its end. An
offset past the end answers 416 with vk-output-length. The hub keeps a job's whole
output until it is settled, so the daemon can read from any offset — whatever GitLab says it
holds.
Cancellation. {mode}: graceful or immediate. A later immediate overrides a graceful;
cancelling a finished job changes nothing and answers its view.
Settle. The daemon has delivered the outcome to GitLab; the hub drops the job's output and keeps its record.
Idempotency and retries. request_id — 16 random bytes, hex — makes a create
idempotent for a day: the same ID with the same body answers the first answer again, or 410
not_found once the hub has dropped that job from its history; with another body, 409
conflict. A queued request retried answers its ticket, or the ticket's grant. Renew,
release, claim, cancel and settle are idempotent by nature, as are a ticket's cancel and an
outcome repeated.
The daemon retries transport errors and every retryable code with backoff from 1 to 30
seconds and jitter, honouring retry_after_secs, and never retries another 4xx — except
job cancels and settles. While handling a job, it retries a cancel or settle after any
failure, including a refused relay, with the same backoff until accepted, 400 or 404.
Cancel delivery runs alongside the trace; an immediate cancel replaces a graceful one.
A settle refused with 409 is retried until the job ends. The job keeps its slot during
settlement retries because the hub still charges its admission. When the daemon stops
taking jobs for shutdown or update, a pending settle is left to its next start, which
cancels and settles reported jobs and resumes running ones.
| Status | code |
Meaning | Retry |
|---|---|---|---|
| 400 | invalid |
unparseable body, malformed ID, a job without a project, an outcome naming another admission than its path; wait or wait_secs on a relay's HTTP/2 connection |
no |
| 401 | unauthorized |
a bearer key missing or bad; for a relay at 2 to 4, no signature, a bad one or its time too far off, or a relay the hub does not know or removed | no |
| 403 | forbidden |
a job of another GitLab server than the relay's, of a relay told rebound, or an API key's GitLab job; a relay below protocol 2 asking for new work; a runner without a lane, or a project or branch its lane does not allow; a job that does not match its admission; on a relay's HTTP/2 connection, a request before its session, a second session, or a request carrying a key or signature |
no |
| 403 | approval_pending |
a job of a project, or of a server, an operator has yet to approve on the hub | no: the job fails |
| 403 | project_rejected |
a job of a project, or of a server, an operator rejected | no: the job fails |
| 404 | not_found |
no such job, reservation, grant or admission of this relay's; another principal's ticket | no |
| 409 | conflict |
request_id reused with another body; an outcome the admission's state does not allow; an admission already used by a job; a claimed ticket cancelled; a settle of a job not finished |
no (a settle: yes) |
| 410 | reservation_gone |
lapsed, released, or its node lost; a ticket that ended or the hub does not know | reserve again |
| 410 | not_found |
a create retried after its job left the hub's history | no |
| 413 | too_large |
spec over 512 KiB | no |
| 500 | internal |
— | yes |
| 503 | no_capacity |
no node accepted in time | yes, after retry_after_secs |
| 503 | unavailable |
hub busy or starting; the reservation queue full; the relay's session not connected, or the session of the connection a request came on ended; an admission being updated | yes, after retry_after_secs |
An error body is {error, code, retry_after_secs?, approval_url?}. Unknown codes read as other;
unknown fields are ignored. When the hub serves a web UI, approval_pending and
project_rejected include approval_url: the page where an operator decides on the project's
approval, <ui_url>/gitlab/<instance>/projects/<project ID>, or, for its server, GitLab ›
Overview, <ui_url>/gitlab, which lists the servers pending.
A job refused approval_pending or project_rejected is not held: approval is a human step.
vk-gitlab releases its reservation and fails it in GitLab, uncommitted, as
runner_configuration_error (script_failure where GitLab does not know it), its trace ending
with instructions and a link, for example:
ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub: https://hub.example.com:8444/gitlab/main/projects/12
Retry this job once an operator has approved it there.
Without approval_url (an older hub or one without a UI), the trace uses the enrolled hub URL:
ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub https://hub.example.com.
Retry this job once an operator has approved it (GitLab > Pending).
GitLab's retry: when: runner_system_failure does not retry that reason, so a pending project
does not loop. The hub keeps the refusal as the answer to the submission's request_id, so it
counts once.
Version 3 (JOBS) adds one message type each way to the existing session, {"type": "job", "kind": …}, whose payloads are vk_hub_proto::dispatch's. A job message in a session below 3
is a protocol error. PROTOCOL stays at 1 to 2; each side that implements version 3 negotiates
from a range of its own reaching it: the hub accepts 1 to 3, so a node offering 3 gets it, and
one offering at most 2 is steered and never offered a job.
| Hub → node | Answer | Meaning |
|---|---|---|
offer {reservation, envelope, lease_secs} |
offer_reply |
set the envelope aside |
renew {reservation, lease_secs} |
lease |
extend the lease from now |
release {reservation} |
lease gone |
give it back |
start {job, reservation?, envelope, spec} |
job |
run a job |
output_ack {job, offset} |
— | output stored durably up to offset |
cancel {job, mode} |
job, then result |
stop a job |
recorded {job} |
— | result stored; stop repeating it |
| Node → hub | Meaning |
|---|---|
held {reservations, jobs} |
everything the node holds, once per session after its report |
offer_reply {reservation, reply} |
accepted {lease_secs} or refused {reason, message?} |
lease {reservation, state} |
held {remaining_secs} or gone {why}: expired, released, started, unknown |
job {job, state} |
accepted, refused {reason, message?}, running {stage, activity?, guest? {cpus, mem_mib}}, finished |
output {job, offset, data} |
output bytes from offset, base64, at most 256 KiB |
result {job, result} |
how the job ended; repeated until recorded |
egress {job, report} |
from version 6: a running job's egress report so far, when it changes and after a reconnect |
A running state's guest is the size of the job's VM, once its prepare has sized it; a hub
older than it ignores it.
An egress report is cumulative, with a revision: the hub keeps the highest, so one sent again
replaces the counts rather than adding to them, and the result's report, the last, has the
highest. The node takes it at each stage's start and every 5 seconds, at most once every 2, and
sends it when it changed; a session below version 6 gets the result's alone.
Refusal reasons are memory, disk, cpus, not_ready (draining, drained, quarantined, in
maintenance), policy, no_reservation, invalid, ceiling (the node's placed jobs not
finished and reservations held reach the hub's ceiling, as the node last applied it),
concurrency (the same, at the node's own max_concurrency) and runner (the host runs its own
gitlab-runner with the vk executor); one the hub does not know reads as other, as ceiling,
concurrency and runner do on a hub older than them.
Reservations. A node answers an offer at once from its ledger: granted, or refused with
the resource that is short. It holds the entry in the ledger itself, with no job process
behind it, and drops it when the lease runs out on its monotonic clock, telling the hub
lease gone expired. A lease is at most 600 seconds. A drain refuses new offers and lets
held reservations lapse; a quarantine releases them.
Start. The node journals the job before answering accepted, so a start redelivered after
a reconnect is answered from the journal and not run twice. A start naming a reservation the
node does not hold is admitted like any other ask, without waiting: refused no_reservation
when it does not fit.
Output. Offsets count bytes of the output as GitLab will hold it: masked, with section
markers, timestamped (unless the job sets FF_TIMESTAMPS false), cut at the trace limit. A
stamp is when the node read the line (its start, for a line that spans reads), however late
the daemon reads it from the hub. The node sends at most 4 MiB past the last ack, keeps every
byte past it on disk, and after a reconnect resends from the offset the hub acked. The hub
appends each chunk to the job's output file and acks once it is synced; a chunk overlapping
what it holds is trimmed, one leaving a gap is a protocol error.
Result. {failure?, exit_code?, message?, output_len, artifacts, usage?}, sent once every
byte up to output_len is acked. artifacts gives each upload's outcome: uploaded, skipped,
too_large or failed. usage is what the job used on the node, {wall_ms, cpu_ms?, peak_mem_bytes?, cpus?, mem_mib?}:
wall_ms: from the driver's start to the job's end, cleanup included;cpu_msandpeak_mem_bytes: the job's VM supervisor and every process under it — the VMM and its vCPUs, service VMs, the switch, the forwards — read from/procafter the last stage and before cleanup, using the same measurement as the executor'sjob resource usagetrace. CPU time is user plus system, guest execution included. Peak memory is each process'sVmHWMsummed: an upper bound where they did not peak together. Host-side cache and artifact transfers, and a host checkout, are not counted. Both are absent when no VM was up to read;cpusandmem_mib: the guest's size, from the job'sMICROVM_CPUSandMICROVM_MEMclamped by the node's ceilings.
usage is optional, as are all its fields except wall_ms.
Reconnects. After its report, a version-3 node sends held. The hub releases each
reservation it does not know, cancels immediate each job it has given up on, and answers
each job it follows with output_ack at the offset it holds.
vk-gitlab speaks GitLab's runner API as gitlab-runner 19.5 does
(network/gitlab.go, network/trace.go):
- Runners.
POST /api/v4/runners/verifyat start with theglrt-token and asystem_idof the daemon's own, persisted. - Job requests.
POST /api/v4/jobs/requestwithRUNNER-TOKEN,infoandlast_update— theX-GitLab-Last-Updateof the previous answer, which lets GitLab hold the request until its queue changes. 201 is a job, 204 none, 403 a token GitLab refuses (the runner stops), 429 and 503 back off asRetry-Aftersays. - Features.
info.featuresadvertisesvariables,image,services,artifacts,cache,fallback_cache_keys,upload_multiple_artifacts,upload_raw_artifacts,refspecs,masking,raw_variables,artifacts_exclude,multi_build_steps,trace_reset,trace_checksum,trace_size,cancelable,cancel_gracefully,return_exit_code,service_variablesandtwo_phase_job_commit. Notsession,terminal,proxy,shared,vault_secrets,service_multiple_aliases(a node's executor takes one alias per service),image_executor_opts,service_executor_opts,native_steps_integrationorjob_inputs.info.executorisvk,info.shellbash. - Commit.
PUT /jobs/:idstate=runningonce the node has accepted the job. A job that cannot be placed is failed instead (runner_system_failure). - Trace.
PATCH /jobs/:id/trace?debug_trace=<CI_DEBUG_TRACE>withJOB-TOKENandContent-Range: <start>-<end>(inclusive, no unit), at most 1 MiB a patch, everyX-GitLab-Trace-Update-Intervalseconds (3 by default, capped at 15 minutes). 202 moves the offset on; 416 moves it to the end of the response'sRange: 0-<n>, which is also how a restarted daemon finds where to resume; 404 or 403 abort the job. The trace preserves the node's output byte for byte. Errors for jobs the daemon refuses or cannot decode, submit or place use gitlab-runner's logger timestamp format (stream00, stdout), unless the job'sFF_TIMESTAMPSis false. The state file preserves this flag across restarts. - Keep-alive. With no new output for 30 seconds,
PUT /jobs/:idstate=runningwith the trace'scrc32:<hex>checksum and byte size. - Final update. Once the hub reports the job finished and its output complete and every
byte is patched:
PUT /jobs/:idwithstatesuccessorfailed,failure_reason,exit_codeandoutput {checksum, bytesize}. 202 and 412 are asked again after the update interval, with backoff up to an hour, as gitlab-runner's final update does. - Cancellation. Every PATCH and PUT answer carries
Job-Status.cancelingasks the hub for a graceful cancel;canceledorfailed, or a 403, an immediate one, after which the daemon writes nothing more for the job.
The daemon keeps one state file per runner, 0600, recording each job it has taken — GitLab
job ID, job token, hub job ID, request_ids — before it submits or commits it. A restarted
daemon resumes each from the hub's view and GitLab's trace offset; a job it had not yet
submitted is submitted with the same request_id.
The state file is <state_dir>/<runner name>.json. state_dir defaults to
/var/lib/vk-gitlab and is created 0700; the daemon refuses to start when group or others
can access it. A daemon run as an unprivileged service user cannot create the default, so
create it for that user before the first start:
install -d -m 0700 -o <user> -g <group> /var/lib/vk-gitlab. A record stays until the hub
has settled its job, including a job already reported to GitLab or abandoned before
submission. A runner's name defaults to its token's short form, so a runner whose token may
change needs an explicit name, or the new token starts an empty file and the old one's jobs
are never resumed; the daemon warns at start about state files no configured runner owns.
Hub jobs and reservations belong to the relay that created them, and the hub answers
not_found for them to any other. The state file records the relay's ID, and a daemon
enrolled as another relay refuses to start while the file holds jobs, as it does a file of a
vk-gitlab from before relays. Before enrolling again (join --replace), stop the daemon
(SIGTERM) and let its running jobs finish within shutdown_timeout; join and leave are
refused while a job is still recorded, until it ends or the file is removed, which gives the
job up to GitLab's timeout.
The daemon does not rotate runner tokens. It verifies the token at start and daily, warns
from a week before GitLab's token_expires_at, and logs an error once it has expired. A
verify GitLab gives no answer to is retried, backing off up to five minutes. On a fleet, a
runner asks for jobs only once its session has sent the hub an inventory listing it verified,
which it sends without coalescing: the hub refuses an unverified runner's requests.
A job GitLab handed over cannot be given back. GitLab ignores two_phase_job_commit: it marks
the job running before it answers the job request (Ci::RegisterJobService#assign_runner!),
PUT /jobs/:id takes only running, success and failed, and a running job with no update
for 30 minutes is dropped as no_updates_running. So a job the fleet cannot run is failed,
never left uncommitted.
The hub never retries a GitLab job. It places a job again only while no node can have
started it: after a refused start, or when the reservation it was sent to is gone and no
start went out. A start whose answer was lost waits for the node's held to say whether it
took it.
| How it ended | JobResult.failure |
GitLab failure_reason |
Older GitLab |
|---|---|---|---|
| success | — | — | — |
| a step's non-zero exit | script |
script_failure |
|
past timeout_secs |
timeout |
job_execution_timeout |
|
| image or service image not pulled or built | image_pull |
image_pull_failure |
runner_system_failure |
| unsupported job, refused by node policy | configuration |
runner_configuration_error |
script_failure |
| clone, artifact or cache transfer failed | external_dependency |
runner_external_dependency_failure |
runner_system_failure |
| VM, agent or host failure | system |
runner_system_failure |
|
| node stopped or drained hard mid-job | interrupted |
runner_interrupted |
unknown_failure |
| canceled | canceled |
unknown_failure: GitLab keeps the job canceled |
|
not placed within place_within_secs |
no_capacity |
runner_system_failure |
|
| node lost | lost |
runner_system_failure |
"Older GitLab" applies when the job's features.failure_reasons lacks the reason; the
fallback follows gitlab-runner's common/failure_reason_mapper.go
(job::gitlab_failure_reason). GitLab's retry: rules then decide whether the job runs
again, on whichever runner asks.
A node is lost for a job once it has been unreachable for job_lost_after (default 5
minutes) past the hub's unreachable threshold. The hub ends the job lost; the daemon
reports it; when the node comes back, its held names the job and the hub cancels it
immediate, and its later uploads fail on a token GitLab has invalidated. A node out of
touch for less keeps running the job and buffering its output, which the trace catches up
on.
A hub restart keeps job records, outputs and offsets, and loses specs and reservations:
a job not yet accepted by a node ends lost, and a reservation the hub no longer knows is
released when its node reports it. A daemon restart loses nothing it had written to its
state file.
Artifact uploads go from the node to GitLab, with the job token, as gitlab-runner's
artifacts-uploader does from inside the build environment. Relaying them through the hub
and the daemon would put every byte through two more hops, through a session that carries
control and telemetry only and whose messages are capped at 1 MiB, for no gain in trust: the
node already holds the token that authorizes the upload. The cost is that the node, not the
daemon, sees GitLab's answer, so it reports each upload's outcome in the result and in the
trace. The node follows gitlab-runner's upload rules: POST /jobs/:id/artifacts multipart
with the archive in field file, query artifact_format, artifact_type and expire_in,
a 307's Location followed, 413 not retried, 503 retried after Retry-After. Unlike
gitlab-runner, it refuses a 307 from https to http and sends the job token after a 307 only
to GitLab's own origin.
Dependency artifacts come from GitLab to the node: GET /jobs/<dependency id>/artifacts
with that dependency's token.
Caches go to the registry the node already uses. A cache is an OCI artifact with one
tar+zstd layer, in repository ci-cache/<gitlab host>[/<gitlab path>]/<project id>,
tagged with the sha256 of its key and the protection of the job's ref, so a protected and an
unprotected job never share one, as with gitlab-runner. The last archive written wins; content addressing
makes an unchanged cache's upload a blob probe. Each request to the registry is retried as vk's
other registry transfers are — a lost or refused connection for up to 90 seconds, a 503 or 429
for up to three minutes — so a registry's update does not lose a job its cache. A request that
runs into CACHE_REQUEST_TIMEOUT is not retried, and like a registry still unavailable after
its retries, leaves a warning and skips the step's other cache requests. Caches are not shared
with gitlab-runner's cache.zips: a project moving to the fleet starts cold.
A job with a registry token. When the hub mints a token for the job's registry
(step 4), the job's start carries it,
with the repository its ci-cache namespace resolves to. The node then keeps the job's
caches in exactly that repository, an absolute path on the registry beside the node's own
[registry] repo prefix, with the job's token, each tagged with the sha256 of its key alone:
the repository names the protection. It skips the caches, with a warning, when the
repository is not a repository name or the token has expired. A job whose start names no
cache repository keeps today's caches with the node's credential, except under [registry] guest_credential = "job", where it runs without caches, with a warning in its trace,
rather than fall back to the node's credential. The token is masked in the job's trace, and
dropped from its journaled start once its result is written. The node reports its
[registry] base URL in its inventory, which the hub matches against its registries to
mint the token for it.
Under guest_credential = "job" the guest's registry proxy (registry.vk) runs for every
job and injects the job's token, never the node's credential. A job with no token gets a
proxy that refuses everything; so does every job a local gitlab-runner gives the node, which
under "job" reaches the registry only by its own means. Under "job-or-node", which a hub
names to nodes taking tokens of their own, placed jobs are treated as under "job", and a
local gitlab-runner's job gets the node's guest token instead
(registry hub, 4d).
With a token, the proxy forwards the lock API (/lock/) and plain files (/dav/files/) as
well as /v2/; the registry enforces the token's grants. Under /dav/files/ only GET,
HEAD, PROPFIND, PUT, MKCOL, DELETE and OPTIONS pass: MOVE and COPY would name the proxy in
their Destination. The job's steps find the proxy in VK_REGISTRY (http://registry.vk)
and their grants in VK_REGISTRY_NAMESPACES, space-separated <prefix>=<rights> with r,
w and l for read, write and lock. The proxy needs net.mode = "switch", the only network
that serves registry.vk; under any other, a job gets neither the proxy nor these variables.
Archiving and extracting run in the guest, in vk-agent, on the job's tree as the job left
it, with the bytes streamed over vsock; the node does the network transfers. No archiver
needs to be in the job's image. On the node, an archive is converted as it streams; only
what a format needs whole touches the job's disk — a downloaded zip, a cache layer, an
artifact's archive before its upload — each capped at 10 GiB, so a job cannot fill the disk
other jobs share. A request to the cache's registry is bounded by CACHE_REQUEST_TIMEOUT
(minutes, 10 by default), as gitlab-runner bounds its cache transfers.
JobSpec is tagged by kind; gitlab_ci is the only kind. vk-gitlab translates
GitLab's job response into a CiJob, normalizing what the node would otherwise interpret:
defaults are filled in, cache keys are expanded and sanitized (cache/cachekey), aliases are
split, empty formats become zip. What only concerns the conversation with GitLab stays with
the daemon.
Every field of the response, from gitlab-runner 19.5's common/spec/spec.go:
| GitLab field | In the spec | Notes |
|---|---|---|
id |
job.id |
|
token |
token |
|
allow_git_fetch |
sources.allow_fetch |
|
job_info.name, .stage, .pipeline_id, .project_id, .project_name, .project_full_path, .namespace_id, .root_namespace_id, .user_id |
job.* |
project_full_path as project_path |
job_info.organization_id, .instance_id, .instance_uuid, .scoped_user_id |
— | ignored: already among the job's variables where the job needs them |
job_info.time_in_queue_seconds, .project_jobs_running_on_instance_runners_count, .queue_size, .queue_depth |
— | daemon only: queue metrics |
git_info.repo_url |
sources.repo_url |
credentials stripped; the node adds the job token through a credential helper |
git_info.repo_object_format, .ref, .sha, .before_sha, .ref_type, .refspecs, .depth, .protected |
sources.* |
|
runner_info.timeout |
timeout_secs |
|
runner_info.uuid |
— | daemon only: logs |
inputs |
— | job_inputs is not advertised |
variables[] key, value, public, file, masked, raw |
variables |
the daemon appends the runner's own (CI_RUNNER_VERSION, …); the node adds the environment's (CI_BUILDS_DIR, CI_PROJECT_DIR, CI_JOB_STATUS, CI_SERVER_TLS_CA_FILE, …) |
steps[] name, script, timeout, when, allow_failure |
steps |
|
image name, alias, command, entrypoint, ports, variables, pull_policy |
image |
resolved under the node's image rules |
image.executor_opts.docker.platform, .docker.user, .kubernetes.user |
image.platform, image.user |
image_executor_opts is not advertised; when GitLab sends them anyway, these are honoured |
services[] |
services |
as image; run as a compose group beside the job VM, on one node |
artifacts[] name, untracked, paths, exclude, when, artifact_type, artifact_format, expire_in |
artifacts |
sent as written; before archiving, the node expands name, paths and exclude from the job's variables and the steps' $GITLAB_ENV, covering $NAME, ${NAME}, ${#NAME} and -, :-, +, :+ of the uploader's shell.Expand. Any other expression is kept as written, with a warning; variables only the image's environment sets expand to nothing |
cache[] key, untracked, policy, paths, when, fallback_keys |
caches |
keys expanded against the job's and the runner's variables, as gitlab-runner expands them; variables only the node sets (CI_PROJECT_DIR, CI_CONCURRENT_ID, …) expand to nothing |
credentials[] of type registry |
registry_credentials |
other types are ignored, as by gitlab-runner |
dependencies[] id, token, name, artifacts_file |
dependencies |
|
features.trace_sections, .token_mask_prefixes |
trace |
trace.limit_bytes is the runner's output_limit, 4 MiB by default |
features.failure_reasons |
— | daemon only: failure mapping |
features.tracing |
— | ignored: no OpenTelemetry export |
secrets |
— | vault_secrets is not advertised; a job with secrets fails runner_configuration_error |
hooks[] |
hooks |
pre_get_sources_script and post_get_sources_script, run in the guest around the checkout |
run |
— | native_steps_integration is not advertised; a job with run fails runner_configuration_error |
policy_options |
— | the daemon writes gitlab-runner's "Job triggered by policy" line to the trace |
suspend_options |
— | ignored: no suspended environments |
the runner's tls_ca_file |
server_ca_pem |
its contents, when set, for CI_SERVER_TLS_CA_FILE; without it CI_SERVER_TLS_CA_FILE is not set |
server_url is the runner's configured URL; job.runner_id the ID /runners/verify
returned.
The node runs gitlab-runner's stage order (common/build.go): prepare_executor (admission,
image, boot, services), prepare_script, get_sources, restore_cache,
download_artifacts, each step_<name>, after_script, archive_cache or
archive_cache_on_failure, upload_artifacts_on_success or _on_failure, then cleanup.
get_sources uses the host-side checkout the executor already has, honouring GIT_STRATEGY,
GIT_DEPTH, GIT_SUBMODULE_STRATEGY, GIT_CHECKOUT and GIT_CLEAN_FLAGS; the other stages
reuse the executor's VM, exec and cleanup code. Step scripts are generated as gitlab-runner's
bash shell generates them (shells/abstract.go, shells/bash.go), and output is masked and
timestamped as its common/buildlogger masks and stamps it: each stream on its own — the
node's lines and vk gitlab prepare as stream 00, vk gitlab run as 01, stdout O and
stderr E apart — masked before it is stamped, the stamps counted in the trace limit.
FF_TIMESTAMPS is read from the job's variables only, on by default as in gitlab-runner 19.5;
a runner's [runners.feature_flags] has no equivalent here. Those ports keep gitlab-runner's
MIT notice.
Ported from gitlab-runner v19.5.0 (MIT; the notice is in NOTICE), as tests of the
vk-gitlab crate unless noted:
| Upstream | Ported as |
|---|---|
network/gitlab_test.go getRequestJobResponse |
a JobResponse JSON fixture; parse it and translate it to a CiJob, checking every field the table above maps |
common/support.go GetSuccessfulBuild, GetRemoteSuccessfulMultistepBuild, GetFailedBuild, getBuildResponse, getStepsBuildResponse |
JobResponse fixtures for multi-step, failing and run jobs |
common/network_test.go Test_Image_ExecutorOptions_UnmarshalJSON, TestFeaturesInfo_JSONMarshaling |
executor options and the info.features body |
network/gitlab_test.go TestGitLabClient_RequestJob, TestSetLastUpdate, …_TransmitsTwoPhaseJobCommit, TestVerifyRunner, TestTokenIsCreatedRunnerToken |
request, long-poll and verify against a fake GitLab |
TestUpdateJob, TestUpdateJobAsKeepAlive |
state updates and Job-Status handling |
TestPatchTrace, TestRangeMismatchPatchTrace, TestPatchTraceContentRangeAndLength, TestPatchTraceContentRangeHeaderValues, TestPatchTraceUrlParams, TestUpdateIntervalHeaderHandling, TestAbortedPatchTrace, TestJobFailedStatePatchTrace |
trace patching, offsets and intervals |
network/trace_test.go (TestJobFinishTraceUpdateRetry, TestJobDelayedTraceProcessingWithRejection, TestJobMaxTracePatchSize, TestCancelingJobIncrementalUpdate, TestJobChecksum, TestJobBytesize) |
the trace loop and final update |
checkTestArtifactsUploadHandlerContent, TestArtifactsUpload, TestArtifactsDownload |
upload and download expectations, in vk-driver's node job tests |
common/failure_reason_mapper_test.go |
vk-hub-proto job::tests (built) |
cache/cachekey tests |
key sanitizing, in vk-gitlab |
helpers/archives zip tests, helpers/trace masker tests |
archive and masking, in vk-driver |
- Registry scopes. A cache write needs a token scoped to the project's cache repository;
vk-registrygrants per-account read and write, not per-repository scopes, so caches start with the node's own registry credential. - Secrets through the hub. The hub sees a spec's secrets in memory. Encrypting them to the reserved node's pinned key would keep them from the hub, at the cost of the hub no longer placing a job on another node than the one reserved.