The Rust backend is workers/server; SQL migrations live in
workers/server/migrations. It owns Nix, Go, sccache, management, and scheduled
maintenance. Use the self-host infrastructure modules and
run migrations explicitly with separate migration credentials.
It connects directly to Postgres and S3-compatible storage, without Worker
service bindings, KV, Hyperdrive, or Durable Objects.
The root wrangler.jsonc deploys the forwarding-only gateway (snowstead).
API_ORIGIN selects the Rust origin; the gateway has no database, storage,
signing, authorization, or maintenance logic. pnpm deploy deploys only this
production gateway and does not run database migrations. Backend secrets belong
to the Rust runtime, not the gateway. Hosted reads retain the existing R2/CDN
origin for public objects; production gateway routes remain API-only.
Staging must set its own API_ORIGIN before deploying the gateway:
API_ORIGIN=https://staging-api.example.com scripts/deploy.sh stagingUse an independently provisioned staging backend, database, buckets, and secrets; never point it at production data. The populated benchmark database is not staging.
For local development, pnpm dev runs Rust on http://127.0.0.1:8787 with
disposable Postgres and S3 services and fresh local secrets, without .dev.vars.
Back up and restore-test the existing database before migration. Preserve the
original JWT_SECRET, ADMIN_SECRET, signing master key, database, buckets,
object keys, cache URLs, and CDN rules. Supply the existing master-key value as
Rust's SIGNING_MASTER_KEY; a new key makes sealed signing keys unreadable.
Legacy self-hosters must also preserve shared signing settings, and keep
AUTH_TOKEN with ALLOW_SHARED_AUTH_TOKEN=true until their clients migrate. Keep secret values out of source control and logs.
Validate the Rust backend against existing objects and clients before changing traffic. Retain a tested rollback path and legacy state, including KV, until all readers and writers have moved. Drain or reconcile outstanding uploads; removing old routes does not revoke existing multipart grants. Do not run old and new maintenance concurrently or re-enable the legacy bandwidth importer: it can overwrite Rust's origin counters. Review your installation's cutover record before any further traffic change. Removing the TypeScript source is not authorization to delete deployed resources or data, and does not prove exhaustive API parity.
Fresh databases use snowstead-server migrate. Existing installations must
follow the checked migration ledger: a Drizzle-only database must first use
the pre-baseline Rust release; the complete legacy SQLx history requires the
explicit migrate --rebaseline operator action. The current binary refuses to
infer or overwrite an unknown schema. Historical releases remain in Git;
source removal does not change this upgrade requirement.
Apply the additive 0001_optimistic_publication.sql migration before deploying
the new server. It adds bucket pointers, durable object ownership, completion
receipts, and the exact-key cleanup queue without removing legacy objects.
Drain old writers and maintenance before cutover. Upload sessions that wrote
straight to a NAR's final key, and the legacy-unresolved reservations cache
deletion once recorded for them, predate staged uploads. Maintenance neither
expires nor protects them, and the server refuses to start while any remain:
reconcile their provider operations and remove those rows first.
The additive 0014_lifecycle_constraints.sql migration names the persisted
state machines documented in workers/server/src/lifecycle.rs. Its CHECK
constraints are NOT VALID, so rows written by retired releases are not
scanned and cannot fail the deployment; new and updated rows are checked. It
also indexes upload sessions by cache and path, and enforces one active
signing key per cache unless the database already holds a cache with several,
which it reports with a warning instead.
After publishing immutable pointers, do not roll back to a server that cannot read them. Use a compatible reader release or a forward fix.
The hosted REQUIRE_CACHE_SIGNING_KEYS=true policy refuses cache creation
without a master key and refuses publication without a per-cache key.
Self-hosters may explicitly set it to "false" to use
SIGNING_PRIVATE_KEY for caches without their own key. Existing shared-key
caches need trust-overlap activation before publishing under hosted policy;
do not silently replace a key clients already trust.
This release replaces relayed multipart bodies with direct S3 UploadPart grants. It needs no additional schema migration, bucket, public route, or CLI credential. Keep the existing private-bucket S3 endpoint and credentials. Clients must be able to reach that endpoint over HTTPS. The storage credential must permit multipart creation, part upload, completion, abort, read, and delete. Do not replace the S3 API endpoint with a public bucket or CDN hostname.
Before activation:
- Build both release packages and run the real CLI proof below against disposable Postgres and S3. This checks the packaged binaries, not only debug builds.
- Verify direct part signing and completion against an isolated R2 test bucket. Local MinIO tests do not prove provider compatibility or production capacity. Never run the normal server with a disposable database and live buckets: its maintenance process can classify live objects as orphans.
- Stop new pushes and let current pushes finish, or cancel them through the existing authenticated API. Cancellation is an operator action, not part of deployment. Restart cancelled pushes after the upgrade.
- Run the candidate's read-only
snowstead-server check-upload-cutoverwith the runtime environment before activation. The command reports the count of active relay sessions containing uploaded parts and exits unsuccessfully until those sessions drain or are cancelled. It also counts upload sessions whose upload key is outsidestaging/andlegacy-unresolvedreservations, and exits unsuccessfully until an operator has resolved them. Server startup repeats the guard in case an old writer races the preflight. Do not bypass it by editing upload receipts or provider IDs.
To verify the release packages locally, from the repository root:
server=$(nix build --no-link --print-out-paths .#server)
cli=$(nix build --no-link --print-out-paths .#default)
SNOWSTEAD_TEST_SERVER="$server/bin/snowstead-server" \
SNOWSTEAD_TEST_CLI="$cli/bin/snowstead" \
nix develop --command env -u SNOWSTEAD_TEST_DATABASE_URL \
scripts/with-postgres.sh scripts/with-r2.sh \
node --test workers/server/test/cli.mjsThe proof uses default CLI concurrency, checks that multipart bytes bypass the API, verifies downloaded NARs against the local Nix store, and exercises restart and cutover refusal. It creates only disposable databases and buckets.
After activation, run a bounded CLI push before resuming bulk uploads. Verify completion and a signed download, not only a successful part PUT. Do not log presigned URLs: they grant temporary write access to a reserved part.
Before rolling back to a relay-only server, drain or abort direct-storage sessions with the new server. Old code cannot resume their provider uploads. Keep the database, published objects, and signing keys intact; prefer a forward fix if an upload or publication has an ambiguous result.
Use the Rust origin as the substituter URL, including for public caches.
Rust renders narinfo from current SQL publication and signing state, and
serves nix-cache-info after resolving the cache in SQL. Neither metadata
endpoint redirects to the CDN or reads mutable metadata from R2.
Public narinfo has a five-minute TTL and an ETag derived from the rendered
bytes; misses use no-store. Public cache-info has a one-hour TTL.
Private metadata requires authorization and uses private, no-cache.
Only public NAR bytes use the bucket hostname. With PUBLIC_CACHE_ORIGIN
configured, Rust validates publication and redirects to the committed bucket
pointer (or a validated legacy key). Without it, Rust streams public NARs.
Private NARs always stream through Rust after authorization, without reusable
presigned redirects. Hosted production keeps cache.snowstead.com as the
public NAR origin, not the substituter metadata origin.
For a self-hosted public bucket, use a dedicated custom domain on a zone in
the same account and a Cache Rule. Keep API and private reads on Rust.
Cloudflare does not cache .nar by default:
wrangler r2 bucket create snowstead-public
wrangler r2 bucket local-uploads enable snowstead-public
wrangler r2 bucket domain add snowstead-public --domain cache.example.com --zone-id <zone id>Cache Rule, on the zone:
- Expression:
(http.host eq "cache.example.com" and not starts_with(http.request.uri.path, "/_api/") and not starts_with(http.request.uri.path, "/private/")) - Cache eligibility: eligible; Edge TTL: use cache-control if present, bypass if not, and 60 seconds for status 404; Browser TTL: respect origin.
New NAR objects have unique immutable keys; SQL stores separate public/private bucket pointers while retaining logical content deduplication. Public NARs use a one-year immutable TTL. The bucket rule's one-minute negative TTL does not apply to Rust metadata misses.
Legacy CDN invalidation remains available. Purging needs the vars
CDN_ZONE_ID and PUBLIC_CACHE_ORIGIN and the
secret CDN_PURGE_TOKEN, an API token with Cache Purge on the zone;
without them nothing is purged. Automatic purges are saved under
cdn-purges/ in the private bucket before dispatch. Requests are batched
and paced, with bounded retries for network errors, 429s, and 5xx responses.
Rust's maintenance loop retries unfinished jobs and respects Retry-After.
Inspect the service logs for persistent failures, fix the configuration or
credentials, and the backlog resumes. No new bucket or database table is needed.
To purge by hand (an empty object purges the cache prefix, while
storePathHashes targets up to 500 narinfo URLs):
curl -X POST https://cache.example.com/_api/v1/admin/caches/main/purge \
-H "authorization: Bearer $ADMIN_SECRET" -d '{}' # or {"storePathHashes": [...]}
# An org API key can purge only its own public caches.
curl -X POST https://cache.example.com/_api/v1/orgs/ORG_ID/caches/main/purge \
-H "authorization: Bearer $ORG_API_KEY" -d '{}'The response { "accepted": true, "paths": "all" } means Cloudflare
acknowledged the request, not that eviction was verified everywhere.
Purges don't remove clients' local copies, Nix's negative cache, or shared
content-addressed NARs. This is cache invalidation, not erasure of public data.
Upgrading: old raw CDN narinfo/cache-info and client copies can remain stale, including former one-year TTLs. Switch substituters to the Rust origin. SQL withdrawal, slug changes, and signing activation do not erase those copies. A migration queues the legacy metadata objects of every current publication for asynchronous deletion, once; it does not rewrite mutable metadata or shorten existing edge/client TTLs. Plan explicit legacy CDN invalidation where needed. Purges do not clear client caches or revoke public bytes already shared.
Nix publication first records ownership of a unique destination key and protects its source. Storage copying runs outside SQL transactions and does not reserve retained capacity. A short SQL transaction then checks ownership, expiry, current policy, and signing configuration, and atomically adopts the bucket pointer, publishes the path, and saves the completion receipt. It does not check or enforce the storage limit. Replaying that receipt does not republish withdrawn paths. A stale attempt cannot adopt its object or overwrite published bytes.
These are short SQL-only decisions, not literal lock freedom. No lock spans
an organization: writers in one organization run concurrently, and each locks
only the rows its decision depends on. An upload, publication, or compiler
transfer holds its cache's row FOR KEY SHARE and its organization's row
FOR SHARE, so a policy change, a cache deletion, a pin, or a path deletion,
which take the cache row FOR UPDATE, runs wholly before or after it. Whoever
inserts or deletes a cache's published paths first takes that cache's row for
its totals, so publications into one cache commit one after another and
publications into different caches do not wait for each other. Requests for
one upload session exclude each other with pg_try_advisory_xact_lock, and
job writers lock their push job's row.
Every write transaction has a 100 ms lock_timeout. A writer that cannot get
a lock in that time, or at once for a NOWAIT lock or a held session lock, is
answered 503 with Retry-After: 1 and must retry. The connection pool is the
only other backpressure: a writer waits for a connection up to the pool's
acquisition deadline, then gets the same 503. Transaction startup finishes
even if its caller disconnects: cancelling SQLx 0.9 during BEGIN can
otherwise return an untracked transaction to the pool. No transaction spans
storage I/O. The CLI retries explicit session-creation 503 responses up to
four attempts, but does not replay a lost creation response. The Go helper
retries uploads marked 503 with Retry-After: 1 before opening its circuit
breaker.
Organization-wide limits are soft. Active upload sessions, staged bytes, compiler staging, and push job budgets are read without a lock, so a few concurrent requests can each pass a limit that only one of them fits under.
Capacity remains logical and shared across Nix, Go, and sccache, and counts committed retained entries separately from staging budgets. The storage limit is eventually consistent. Publication commits without checking it and wakes a background evictor, which also scans every ten minutes for organizations over their limit. The evictor removes least-recently-used entries in its own bounded transactions, at most 256 entries each, until the organization is at or under its limit. One evictor works on an organization at a time, across server instances: a second one finds the first at work and leaves it to it. A pass reads the limit and the usage it acts on under locks, so a limit that was raised, or an overage that someone else already deleted, is not evicted against once that change has been acknowledged. LRU ordering still uses the later of publication and access, with existing pin, push-job, and active-upload protections; an organization whose remaining entries are all protected stays over its limit until a protection ends. An organization can therefore exceed its limit briefly, and the most recent publication is evicted like any other entry when nothing older is evictable. Direct CDN downloads do not update access.
New uploads are held back when they start, at Nix session creation and at a Go
or sccache reservation. Once usage is at or above 110% of the limit they get
503 with Retry-After, which also wakes the evictor; the CLI retries it. At or
above the limit while pinned and job-protected paths alone fill it they are
refused with 413, which the CLI does not retry: delete retained paths.
Sessions and reservations that already exist continue, and a batch larger than
the whole limit is still refused with 413 before any member publishes.
Deletion, retention, and eviction withdraw SQL visibility atomically and queue known exact keys for asynchronous cleanup. Storage failure does not roll back the committed withdrawal. Maintenance retries idempotent deletion outside SQL transactions. Expiry revokes adoption of new attempts, not provider work; orphan sweeps find late writes. Stored NAR bytes are collected once no publication points at them and no publication attempt is preparing the same content; an upload session alone does not keep them. Collection locks only the rows it removes and skips any that a publication is adopting.
Snowstead implements the WebDAV subset used by unmodified sccache. Enable the
sccache protocol and issue a cache-scoped token. Nix and Go permissions do
not grant compiler access.
snowstead cache create compiler --protocols sccache --private
snowstead token create ci --caches compiler --perms sccache:read,sccache:write
export SCCACHE_WEBDAV_ENDPOINT=https://cache.snowstead.com/_api/v1/sccache/ORG_SLUG/compiler/
export SCCACHE_WEBDAV_TOKEN="$SNOWSTEAD_TOKEN"
export RUSTC_WRAPPER=sccache
export CARGO_INCREMENTAL=0
cargo build --release
sccache --show-statsUse a client built with WebDAV support. For CMake, set
-DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache, or
invoke sccache cc / sccache c++ directly. Stop an existing sccache daemon
before changing its endpoint or credentials. Do not set
SCCACHE_WEBDAV_KEY_PREFIX: the endpoint already identifies the namespace.
Clients can set SCCACHE_WEBDAV_DISABLE_CREATE_DIR=true to avoid directory
probes; older clients can use the virtual directories.
For untrusted pull requests, use SCCACHE_WEBDAV_RW_MODE=READ_ONLY and a token
with only sccache:read. Public caches allow anonymous reads. Give write
credentials only to protected jobs building reviewed code: a writer can poison
compiler outputs. Use separate caches for separate trust domains.
PUT requires Content-Length and exact-length validation, with a 64 MiB archive
limit. Archives expire seven days after publication. Duplicate writes preserve
the published bytes and do not extend retention. Storage counts toward shared
organization capacity; compiler staging also has a separate byte budget.
The endpoint supports a/b/c/<64-lowercase-hex-key>, .sccache_check, and
virtual directory PROPFIND/MKCOL, not general WebDAV listing, DELETE, MOVE, or COPY.
Org-management credentials can read statistics with
GET /_api/v1/orgs/:org/caches/:name/sccache and reset only compiler data with
POST /_api/v1/orgs/:org/caches/:name/sccache/reset. Reset leaves Nix and Go intact.
pnpm check exercises real Rust and C++ clients against Rust with disposable
Postgres/S3 and fresh local client caches.
The Rust server implements the hosted Nix, Go, sccache, and management APIs. The migration target is API compatibility with documented improvements, not identical behavior for every Worker input or storage state. Do not claim 1:1 parity from passing the two implementations' separate test suites.
The following differences are intentional:
- Integrity and authorization: Rust verifies compressed NAR content, declared hashes, and sizes before publication. It checks token revocation in SQL on each authentication and requires current publication/membership for NAR reads. It does not reproduce the Worker's corrupt-upload acceptance, revocation propagation gaps, or stale cached authorization.
- Upload recovery: failed compiler transfers do not commit their declared output. Pending attempts share a byte reservation; the first verified transfer adopted under the organization lock commits its actual output and size. A corrected retry may change either before adoption without cancelling another writer. Once adopted, Go outputs are immutable, and different-size sccache retries cannot publish an unfinished adopted object. Replays of published sccache entries still leave the original bytes unchanged.
- HTTP behavior: unsupported methods use Axum's 405 responses and GET routes support HEAD. Unmatched API routes return 404 without authenticating a nonexistent operation. Validation messages are not byte-for-byte copies of Worker errors. Compiler ETags identify the immutable entry, rather than exposing the storage provider's ETag. Clients must treat them as opaque.
- Management inputs: path deletion accepts RFC3339 timestamps with an
offset or complete
YYYY-MM-DDdates at midnight UTC. JavaScript's other date formats, includingYYYY-MM, are not supported. Audited management request bodies are limited to 1 MiB; oversized bodies currently return 400. - Storage and caching: Rust compiler reads access storage rather than the
Worker's Cache API and return
x-snowstead-cache: MISS. Expiration is checked after storage access and again before returning a response; transfers are not interrupted if their entry expires after response streaming starts. Direct public narinfo misses useno-store, not negative edge caching. Only public NAR delivery uses the configured CDN handoff. - Legacy metadata: narinfo and
nix-cache-infoare rendered from SQL. A migration queues the objects the earlier service stored for them; withdrawing a path, cache, or slug no longer queues per-path metadata cleanup. - Accounting and maintenance: origin downloads attempt immediate SQL bandwidth increments independently of optional telemetry; these count selected object bytes, not confirmed client receipt. The public importer updates public bytes separately, rather than replacing origin totals from sampled Worker analytics. Maintenance uses bounded cleanup batches, a token expiry grace period, and retention selection that skips caches with no work.
- Admission and tracing: removal of request/upload-count admission caps and optional tracing are deliberate changes. Protocol validation, retained storage limits, staging safeguards, and bounded buffers still apply.
Cloudflare Cache API/CDN behavior, all storage failure interleavings, and all maintenance recovery states have not been exhaustively compared. These differences are not a claim that every unlisted behavior has been proven equal.
The observability Cargo feature enables vendor-neutral OpenTelemetry and
tracing instrumentation. The standalone server logs locally; it does not
configure an exporter or read telemetry-provider credentials.
Embedding applications enable this feature on their snowstead-server
dependency and install their own tracing subscriber, providers, sampling,
exporters, and shutdown handling. Call observability::init_metrics(meter)
before constructing database pools. Install observability::PoolTimingLayer
with a filter for the sqlx::pool::acquire target to record acquisition waits.
Trace spans use the snowstead_observability target.
Spans cover HTTP response streaming, database queries and transaction waits, storage operations, multipart uploads, outbound requests, and maintenance. Operation names are static. Raw SQL, credentials, storage keys, and arbitrary error payloads are not added to these spans. Export policy belongs to the embedding application. Instrumentation compiles out without the feature; Hotpath profiling remains independently opt-in.
Configure the CLI with your gateway URL and admin secret:
snowstead login --token-stdin --url https://cache.example.com --name prod < admin-token-filenix.settings.substituters = [ "https://cache.example.com/main" ];Caches are public unless created private. Reading a private cache
(nix-cache-info, narinfo, NARs) needs a token with pull on it, as a
bearer token or as the password of HTTP Basic auth, which is what Nix sends
from a netrc file.
snowstead cache create team --protocols nix --private
# For an existing cache: snowstead cache set team --private
snowstead token create builders --caches team --perms pullNew caches require an explicit protocol selection: --protocols nix,
--protocols go, --protocols sccache, or a comma-separated combination (no default).
API creation payloads likewise require a nonempty protocols array of these names on
POST /_api/v1/admin/caches and POST /_api/v1/orgs/:org/caches.
Existing caches keep their protocols when omitted on updates. To add Go to a
Nix cache, use snowstead cache set team --protocols nix,go; updates must keep
all enabled protocols and cannot remove them. Cache lists, details, and creation
responses include the protocols array. Existing database caches keep Nix and
Go enabled; sccache requires explicit opt-in.
On the machines that substitute from it:
# /etc/nix/netrc
machine cache.example.com
password <the pull token>
nix.settings = {
substituters = [ "https://cache.example.com/team" ];
netrc-file = "/etc/nix/netrc";
};Only empty caches can change visibility. Private reads check current SQL authorization and publication state. Private NAR downloads are authorized and streamed by the Rust server, never redirected to a reusable presigned URL.
API tokens (snowstead_tok_…) scope access to specific caches. The token is
returned once on creation; the server keeps only a hash of it. Save it immediately.
# Create a token scoped to the "main" cache with push access
snowstead token create ci-runner --caches main --perms push --expires 90
# Create a token with access to all caches
snowstead token create admin-bot --caches "*" --perms push,pull
# List all tokens
snowstead token list
# Revoke a token
snowstead token revoke $JTI "employee offboarded"| Option | Default | Description |
|---|---|---|
--caches |
* |
Comma-separated cache names or * for all |
--perms |
push |
Comma-separated: push and/or pull |
--expires |
90 |
Token lifetime in days (1-365) |
Log in with a push token (not the admin secret):
snowstead login --token-stdin --url https://cache.example.com --name prod < push-token-fileThen push:
# A path and its runtime closure
snowstead push main /run/current-system
# Built flake outputs, read from stdin
nix build --no-link --print-out-paths .#nixosConfigurations.myhost.config.system.build.toplevel \
| snowstead push main
# Only the exact path, without its references
snowstead push main /nix/store/abc...-hello --no-depsReferences are published before the paths that need them. .drv paths are
rejected unless you pass --allow-drv.
Use --jobs <n> to control how many paths are in flight (default 32).
Paths that an upstream cache already serves are not pushed: whoever
substitutes from a cache has its upstreams too. The upstreams are the
UPSTREAM_CACHES runtime variable, space-separated; set it empty to push everything.
The server remembers paths found upstream, so each is looked up there once.
- Set
ADMIN_SECRETin the Rust runtime's secret environment, and keepAUTH_TOKENwithALLOW_SHARED_AUTH_TOKEN=truemeanwhile. - Log in with the admin secret:
snowstead login --token-stdin --url "$URL" --name prod < admin-token-file - Create scoped tokens:
snowstead token create ci-runner --caches main - Update publishers to log in with their token
- Remove
ALLOW_SHARED_AUTH_TOKENandAUTH_TOKENto disable legacy auth
Authentication turns a request's bearer credential into one principal;
workers/server/src/authz.rs then decides, in one function, whether that
principal may take the route's action on its org, cache, or the service.
Anything it does not allow is denied.
| Credential | Principal | May |
|---|---|---|
ADMIN_SECRET |
operator | /_api/v1/admin/... only, including org creation and limits |
in-process TrustedPrincipal extension |
verified person (owner, admin, or member) | its org's management API; only the owner changes the slug, API keys, or signing keys, or moves caches |
snowstead_key_<id>_<secret> |
org API key | its org's management API, except creating keys and changing the slug |
snowstead_tok_<id>_<secret> |
API token | Nix, Go, and sccache in its org, per its permissions and caches; pins with push |
| JWT-era token | API token | the same, only while its jti is listed and unrevoked in api_tokens |
AUTH_TOKEN |
shared token | Nix pushes to the default org, only with ALLOW_SHARED_AUTH_TOKEN=true |
API tokens store only a SHA-256 hash of their secret. Their permissions and
caches live in token_permissions and token_caches and are read on every
request; a JWT's own claims grant nothing. Migration 0015_central_auth.sql
backfills those tables from existing rows.
An embedding application can supply a verified person through a
TrustedPrincipal request extension. It must authenticate the person and
resolve current organization membership before dispatch. HTTP headers and
hosted web assertions cannot construct this identity. The route's org must
match, and audit logs name user:<email> from the verified identity.
The server refuses to start without an ADMIN_SECRET of at least 16
characters, with ALLOW_SHARED_AUTH_TOKEN=true but no AUTH_TOKEN, or
without JWT_SECRET while unrevoked JWT-era tokens exist. Standalone
installations need no hosted assertion keys or dashboard-disable switch.
Host aliases use the generic CACHE_HOST_SUFFIX and
CACHE_CANONICAL_HOSTS settings. Embedding applications can reserve
organization names with CACHE_RESERVED_NAMES and exact DNS hosts with
CACHE_RESERVED_HOSTS; both accept whitespace-separated lowercase values.
Reserved hosts take precedence over canonical hosts. Core protocol names
remain reserved independently of these settings. Hosted DNS policy belongs
to the embedding application, not the cache core. On an organization host,
/<cache>/go, /<cache>/sccache/, and /<cache>/bazel (Bazel over HTTP)
serve the same handlers as /_api/v1/<product>/<org-slug>/<cache>, which
keeps working; every other path on that host, apart from the Bazel gRPC
services, is a Nix read. Bazel over gRPC names the organization by host and
the cache by instance:
--remote_cache=grpcs://<org-slug>.<suffix> --remote_instance_name=<cache>.
The organization host rejects the <org-slug>/<cache> instance shape that the
shared host keeps. Organization hosts answer HTTP/2 requests that send no
Host header by their :authority.
Validly signed JWTs whose jti is not in api_tokens are refused. Only
listed tokens can be revoked. A JWT-era row whose
perms_json or caches_json was invalid, or empty, gets no permissions or
caches, so it is refused rather than widened. After migrating, list live
JWT-era tokens left without either:
SELECT t.jti, t.org_id, t.subject
FROM api_tokens t
WHERE t.secret_hash IS NULL
AND t.revoked_at IS NULL
AND (t.expires_at IS NULL OR t.expires_at > now())
AND (NOT EXISTS (SELECT 1 FROM token_permissions p WHERE p.token_jti = t.jti)
OR NOT EXISTS (SELECT 1 FROM token_caches c WHERE c.token_jti = t.jti));All write operations and admin actions are logged to Postgres with actor,
action, cache name, IP, and status code. Logs are retained for 30 days.
Each entry also records its org; an org reads its own with
GET /_api/v1/orgs/:org/audit.
A hosted service runs Snowstead for many tenants. Each tenant is an org, which has caches, tokens, API keys, an audit log, and operator-set limits in Postgres. Membership and commercial policy stay with the service that runs Snowstead, outside the cache service.
A self-hosted install doesn't need to know about any of this: everything
belongs to the default org, which has no storage limit, and tokens made
without an org are its tokens.
- Cache names are an org's own. Each org has a slug, and its caches
are served at
<host>/<slug>/<name>, so every org can have amain. The default org's caches stay at<host>/<name>.PUT /orgs/:orgchanges SQL routing immediately; Rust serves metadata at the new slug and refuses the old slug.POST /orgs/:org/movequeues the old slug'snix-cache-infokeys for cleanup and clearsmovingFrom. It does not move NARs or rewrite narinfo. Old raw CDN/client copies can remain stale. - Each org has its own API under
/_api/v1/orgs/:org: its storage, caches, tokens, API keys, and audit log. The dashboard reaches it for its signed-in users, and an org's API key (snowstead_key_…) reaches only its own;ADMIN_SECRETdoes not. Only the operator sets org limits and permissions, through/_api/v1/admin/orgs/:org. An org without private-cache permission keeps its private caches private and readable, but they take no pushes, and it can't make new ones. - Tokens reach only their org's caches, for pushing and for pulling
from a private cache. A token is made naming caches, which it keeps by
id, so it outlives a rename, or patterns over names where
*matches anything:ci-*reaches every cache starting withci-, including ones made later, and*reaches them all. A push names its cache within the token's org, sosnowstead push mainpushes to that org'smain. - Storage limits. Limits are eventually consistent. Publication does not check them: a background evictor removes eligible LRU entries across products afterwards, so usage can briefly exceed the limit. New uploads get a retryable 503 from 110% of the limit, and 413 at the limit when only pinned or job-protected paths remain; delete retained paths explicitly in that case. Staging has its own budget.
- Admission. Rust does not retain the Worker's per-minute request caps. Storage limits, staging safeguards, and bounded runtime resources still apply.
- Bandwidth.
GET /orgs/:orgreportsservedThisMonthfor the current UTC month's origin bytes. Rust records selected object bytes in SQL, not confirmed client receipts. Configured CDN imports report public bytes separately aspublicServedThisMonth. - Signing keys. Each new cache gets its own cryptographically generated
Ed25519 key, shown as its
publicKey. Managed key names use the immutable cache ID and a random UUID; managed public bytes are unique in Postgres. Creation and initial key registration are atomic.customSigningKeysis an operator-set org permission for BYOK; org API keys cannot change it. Removing permission does not disable an active imported key.
All responses are Cache-Control: no-store. storageLimit is a nonnegative
safe integer number of bytes, or null for unlimited; 0 refuses new storage.
GET /_api/v1/admin/orgsreturns{orgs: [{id, slug, storageLimit, privateCaches, customSigningKeys}]}.GET /_api/v1/admin/orgs/:orgreturns one org as the org GET does.PUT /_api/v1/admin/orgs/:orgtakes{slug?, storageLimit?, privateCaches?, customSigningKeys?}. Creation requires a slug and defaults to unlimited storage, private caches allowed, and custom signing keys disallowed. Omitted limits preserve existing values, including concurrent updates. Unknown fields are rejected with 400.- Org GET and PUT responses contain
id,slug,movingFrom,storageLimit,privateCaches,customSigningKeys,used,pending, andservedThisMonth. GET /_api/v1/admin/overviewretains the global operator overview.
Admin endpoints require operator ADMIN_SECRET authentication. An org's
owner can change only its slug, with PUT /_api/v1/orgs/:org and
{slug}; org API keys cannot change limits or list all orgs. org.put
records old/new slug and limits in the affected org's audit log, with the
operator (admin) or the signed-in user as actor.
Limits are read from Postgres on every enforcement lookup. Updates apply immediately on subsequent requests. Lowering a limit never deletes published data or makes a private cache public: it refuses disallowed new pushes and new private caches, and blocks BYOK import/activation while leaving already-active keys working.
Rust reads authorization from SQL; no KV synchronization is required.
The compatibility POST /_api/v1/admin/auth/sync route does not populate KV.
Apply the Rust migrations through infra before deploying. All endpoints below
use /_api/v1/orgs/:org (the org ID, not its mutable slug),
return JSON with Cache-Control: no-store, and require a dashboard user's
assertion for that org or that org's API key unless stated otherwise. Dates are ISO strings; sizes
are integer bytes. Errors use { "error": "message" }.
| Method and suffix | Body | Response |
|---|---|---|
GET /tokens |
none | { tokens: Token[] } |
GET /tokens/:jti |
none | { token: Token }; 404 for missing or another org's token |
Token is { jti, subject, caches: string[], perms: string[], org, createdAt, lastUsedAt, expiresAt, createdBy, revokedAt, revokedBy, revocationReason }.
lastUsedAt is an ISO string or null; it is the only addition to the existing
org token-list shape. The operator-wide admin token list is unchanged.
expiresAt, revokedAt, revokedBy, and revocationReason can be null.
Successful API token authentication for Nix, Go, and sccache attempts a conditional SQL update through the bounded connection pool. A token's timestamp changes at most once per five minutes, including concurrent calls. It records authentication, not successful artifact delivery. Anonymous public/CDN reads cannot identify a token. This is best-effort activity telemetry, not a security audit or exact request counter; failed writes can leave it stale.
GET /eviction?limit=50&nextCursor=...&recentCursor=... has no body. limit
defaults to 50, must be 1–200, and applies separately to each list. Cursors are
opaque strings; URL-encode them and pass each list's cursor independently.
Null means the end of that list. The JSON response is:
{
next: { items: Candidate[], cursor: string | null, bytesToFree: number },
recent: { items: Eviction[], cursor: string | null }
}
// Candidate:
{ product: "nix" | "go" | "sccache", cache: string, cacheId: number,
key: string, size: number, lastAccessedAt: string | null }
// Eviction: Candidate plus
{ id: number, evictedAt: string,
reason: "capacity" | "retention" | "clear" | "evict" }For Nix, key is the full store path; for Go/sccache it is the action key.
next uses capacity's shared selection query, ordered by the later of last
publication and access, then product and entry ID. It excludes pinned closures,
active Nix uploads/installations, and expired/obsolete action entries. It is
an advisory preview, not a reservation: row locks, accesses, and uploads can
change what the evictor actually removes. Each response uses a consistent DB
snapshot; subsequent pages are live views, not a frozen multi-request snapshot.
bytesToFree is max(used - storageLimit, 0), or zero for unlimited storage;
it excludes pending transfers and hypothetical future uploads. Candidates are
still available when this number is zero.
recent is newest-first by log ID. Set-based SQL triggers record logical
removal transactionally: capacity (capacity), expired-data GC (retention),
product resets, explicit path deletion and cache deletion (clear), or the
whole-cache endpoint below (evict). Later physical cleanup does not duplicate
events. Logs retain the cache's ID and name even after deletion/name reuse.
Only the last 30 days are returned; Rust's scheduled GC sweeps older rows.
| Method and suffix | Body | Response |
|---|---|---|
GET /caches/:name/pins |
none | { pins: Pin[] }, ordered by name |
POST /caches/:name/pins |
{ name, storePath, keep } |
{ pin: Pin }; 201 created, 200 updated |
DELETE /caches/:name/pins/:pin |
none | { deleted: true, name: string }, including an already absent pin |
type Keep = "forever" | { days: number } | { revisions: number };
type Pin = {
name: string; keep: Keep; currentRevision: number;
history: { revision: number; storePath: string; createdAt: string;
supersededAt: string | null; pinned: boolean }[];
};Pin names match [A-Za-z0-9][A-Za-z0-9._-]{0,127}; storePath is a full
/nix/store/<hash>-<name> path; days/revisions are integers 1–10000. Unknown
request fields and invalid policies return 400. POST and DELETE also accept an
API token with Nix push permission for this exact cache and org. GET remains
management-only. Writes audit pin.create, pin.update, or pin.delete.
Re-pinning a name to a different path adds a monotonically numbered revision;
retrying the current path does not add another. History is newest-first.
The current revision always remains pinned. forever keeps every revision;
revisions: N keeps the newest N including current; days: N keeps an old
revision for N days after it was superseded. Old history remains visible after
protection expires, until the pin is deleted. Extending keep returns 409 if it
would reactivate a revision with missing paths.
Closure handling is cache-local and transitive: a recursive SQL query deduplicates cycles/shared dependencies, then inserts indexed membership rows in one batch per revision. The root and every referenced dependency must already be published in this cache (409 otherwise); upstream-only dependencies are not fetched or silently assumed protected. More than 10000 distinct paths returns 413. Creation cost and stored membership scale with closure size; history storage scales with revisions and their closures. Candidate scans use indexed membership checks, not recursive traversal. A pin mutation holds its cache's row until it commits, which orders it against publication, deletion, retention, and eviction in that cache: a path is either evicted before the pin is created, and the pin is refused, or protected by it. These are short SQL decisions, not locks held across storage I/O. Pinned roots and dependencies cannot be removed by capacity, retention, or the existing individual-path deletion endpoint. Whole-cache eviction and cache deletion explicitly override pins.
POST /caches/:name/evict takes no body and returns:
{ evicted: true, name: string,
removed: { nix: number, go: number, sccache: number }, pinsDeleted: number }This management-only operation audits cache.evict, removes all Nix paths and
Go/sccache rows (including staged/obsolete rows still attached to the cache),
and deletes all its pins. It preserves the cache ID, slug, protocols/settings,
tokens, and signing keys. It invalidates old action generations and aborts Nix
sessions, so pre-eviction uploads cannot publish later. A fresh upload after
the operation can repopulate the cache.
SQL withdrawal, upload invalidation, eviction logs, and exact-key cleanup records commit together. No storage deletion runs under those locks. An already empty cache returns zero counts. Maintenance deletes retired objects asynchronously and retries failures; it does not roll back withdrawal. Shared NAR blobs remain for reference-aware GC. Detached Go/sccache objects from earlier capacity eviction remain owned by GC. Large SQL withdrawals can still contend with writers; fail-fast callers must retry.
Run database migrations before deploying. Signing management uses the org
API's bearer authentication (a dashboard assertion or that org's API key), not a
push token. BYOK staging and activation require customSigningKeys; managed
key rotation does not. In the dashboard, mutations require the org owner.
All responses are
Cache-Control: no-store; private or sealed key material is never returned.
Under /_api/v1/orgs/:org/caches/:cache/signing-keys:
GETreturnsactivePublicKey,shared(true while the cache signs with the service's key rather than one of its own), andkeys, each withid,publicKey, andstate(staged,active, orretired). New initial managed keys have history IDs. A legacy shared key hasid: nulluntil its first rotation records its history.POST {}generates a Snowstead-managed key, staged, not yet signing.POST {"secretKey":"name:base64"}stages a BYOK Nix secret key. Import requires canonical base64 of exactly 64 bytes (32-byte Ed25519 seed then its matching public half). Names are 1–200 ASCII characters: alphanumeric first, followed by alphanumeric or._/+-. Do not send a PEM, bare seed, or trailing newline. The secret is sealed with AES-GCM underSIGNING_MASTER_KEY, authenticated to the version, cache ID, key UUID, name, and public key. Both staging forms return201 {id, publicKey, state: "staged"}. A key name cannot be reused in the cache, including its historical keys or the service key name.POST /:id/activate {"trustedPublicKeys":["name:base64", ...]}activates a staged key and retires the previous active key atomically. First deploy all keys returned by GET to clients'trusted-public-keys, retaining old keys; then pass that full set as explicit confirmation. Missing trust confirmation or a non-staged key returns 409; a missing cache/key is 404. Activation does not prove clients were updated: the caller must ensure it.
Staging and activation require SIGNING_MASTER_KEY (409 without it).
Legacy shared signing is available only with explicit self-hosted opt-out
from the required-per-cache-key policy. The
cache's existing publicKey field continues to show its current active key;
use this lifecycle GET to discover the complete trust set after rotations.
Retired public keys are retained permanently, and cannot be reactivated or
deleted through this API. Their sealed secrets are removed from history.
Rotation is trust overlap, not indefinite dual signing. Rust renders narinfo for existing and new publications with the current SQL signing key, including on public reads. Activation changes freshly rendered bytes and their ETags without an R2 rewrite or backfill. Publication validates current signing policy at commit; rotation does not cancel prepared candidates.
Keep historical public keys trusted while old CDN or client copies exist.
Cached API narinfo can also retain the prior signature until revalidation.
Rotation does not purge those copies. This API is not an emergency
compromise-revocation mechanism; removing old trust needs a separate CDN/client
invalidation and recovery plan.
Rotation does not rotate SIGNING_MASTER_KEY; retain that deployment secret
to open active and staged keys.
Back up Postgres (including caches and key history) and keep a separate,
access-controlled backup of SIGNING_MASTER_KEY in a password manager or
secrets vault. Restrict production secret access to operators; never put
keys in logs, source control, public R2, or support tickets. A database
backup alone cannot restore signing. The master key can decrypt every
tenant key, so it remains a shared security boundary, not HSM isolation.
Before inviting testers, restore a database backup into an isolated environment using the matching master key, publish a test narinfo, and verify its signature against the unchanged advertised public key. A wrong or missing master key must fail rather than fall back to a shared key. Existing v1 sealed keys remain readable; newly sealed keys use v2 identity authentication. Do not replace the master secret to rotate cache keys: that makes existing ciphertext unreadable. Master-key rotation and emergency compromise recovery need a separate, tested migration plan.