A cache for Nix paths and Go, Rust, and C/C++ compiler outputs.
The Rust server works with unmodified sccache
clients over WebDAV. Enable --protocols sccache and use scoped
sccache:read,sccache:write tokens for Rust and C/C++ CI builds.
The native Go helper uses
https://cache.snowstead.com/_api/v1/go/<org-slug>/<cache-name> as its base URL.
Public caches allow anonymous reads; private reads require go:read, and every
upload requires go:write. Nix permissions do not grant Go access. Give write
access only to protected CI jobs building reviewed code: hashes detect
corruption, not malicious outputs from authorized writers.
New caches require explicit protocols: snowstead cache create main --protocols nix
for Nix, --protocols go for Go, --protocols sccache for Rust/C/C++, or a
comma-separated combination such as --protocols nix,go,sccache; there is no
default. API creation requests require the corresponding protocols array.
Existing-cache updates may omit it to preserve the selection, or add protocols
but not remove them (for example, snowstead cache set main --protocols nix,go).
Existing database caches keep Nix and Go enabled; sccache requires explicit opt-in.
Go objects use the private bucket, expire seven days after upload, and count toward organization storage limits. Resetting Go leaves Nix paths intact. The Rust server owns all protocols, management, and maintenance.
The control-plane API exposes token last-use times, cross-product eviction candidates and 30-day history, named Nix closure pins, and a whole-cache eviction that keeps settings and credentials. These features are API-only; they do not add dashboard or CLI commands.
workers/gateway/ Forwarding-only gateway to API_ORIGIN
workers/server/ Rust API, protocols, maintenance, migrations, and tests
apps/cli/ CLI for pushing store paths
pkg/protocol/ Wire types, limits, and version shared by the CLI and server
apps/go-cache/ Native GOCACHEPROG helper
infra/ Rust service deployment
scripts/ Development and verification harnesses
nix develop # drops you into a shell with node, pnpm, wrangler, postgres
pnpm installpnpm check # type-check + tests across the workspaceRun the Rust server locally on http://127.0.0.1:8787:
pnpm devIt starts disposable Postgres and S3 services, applies the Rust migrations,
and generates fresh local secrets. It does not read or write .dev.vars;
local state is discarded when the stack stops. Database migrations live in
workers/server/migrations.
Deploy the standalone Rust service with the self-host infrastructure modules.
See docs.md for gateway configuration and cutover cautions.
The gateway only forwards to API_ORIGIN; it has no storage or backend secrets.
Hosted public reads retain the existing R2/CDN origin.
# Deploy only the production gateway; no database migrations run here.
pnpm deployThe hosted configuration requires per-cache keys and fails closed without
the master key. Existing caches without their own key need a staged key
and explicit trust-overlap activation before they can publish. Self-hosters
who deliberately want the legacy shared key can set
REQUIRE_CACHE_SIGNING_KEYS to "false" and configure
SIGNING_PRIVATE_KEY and SIGNING_KEY_NAME. See the signing-key lifecycle
and recovery guidance in docs.md.
Set DATABASE_READ_URL to a replica PostgreSQL URI to serve only
GET /_api/v1/admin/overview totals from that replica. These reporting values
may lag. All other queries, including authorization and organization overviews,
continue to use the primary DATABASE_URL.
For PlanetScale direct replicas, use port 5432 and append %7Creplica to
the URI username. Use verified TLS and a read-only role.
DATABASE_READ_MAX_CONNECTIONS defaults to 8 and must be positive. This is a
separate per-process pool limit; budget for application instances, deployment
overlap, and other clients. Do not assume replica connections divide evenly.
The primary pool keeps at least 2 connections warm and the replica pool keeps
at least 1, capped by each pool's configured maximum. Excess idle connections
are eligible to close after 60 seconds.
When DATABASE_READ_URL is unset, reads share the existing primary pool with
no additional connections. An invalid or unreachable configured replica fails
startup. Runtime replica errors do not fall back to primary. Migrations ignore
replica settings, and pnpm dev clears them to keep the local stack disposable.
Build the CLI (or nix run . for one-off use):
nix build # produces ./result/bin/snowsteadBuild the server with nix build .#server. Both packages use Crane to keep
vendored crate sources and compiled dependencies in separate Nix derivations.
Source-only changes rebuild the application, not its external crates. Changes
to dependencies, the toolchain, or build features can rebuild the dependency
cache. The first build populates it; later builds reuse it while its store paths
remain available locally or through a configured binary cache. Local development
still uses Cargo's incremental builds inside nix develop.
The CLI is a native Rust binary using the Zig NAR core. On Linux, NAR
serialization, SHA-256, and zstd compression run in-process; Nix is still used
for path metadata and closure resolution. On macOS, Nix also serializes the
NAR. The flake includes Nix as a runtime dependency, not Node or the workspace's
Worker dependencies. Existing credentials in ~/.config/snowstead/config.json
continue to work. The full development environment is separate (nix develop).
Sign in once (use --no-browser over SSH and open the printed URL elsewhere):
snowstead loginApprove the organization and token lifetime in your browser: 30 days, 90 days (the default), 365 days, or never. The ten-minute device approval window is separate from the saved token's lifetime.
For CI, set both SNOWSTEAD_TOKEN and SNOWSTEAD_URL (the cache base URL).
For self-hosted manual tokens, use
snowstead login --token-stdin --url https://cache.example.com --name prod < token-file.
Manual tokens are configured without remote validation; positional login tokens
are no longer accepted. snowstead auth status [--server <profile>] shows local
configuration, and snowstead logout [--server <profile>] removes local
credentials without revoking the remote token. See the
CLI authentication guide for profile details.
snowstead push uploads each given path together with its
runtime closure (everything it references), and reads paths from stdin when
none are given as arguments:
snowstead push main ./result
snowstead push main /run/current-system
nix build --no-link --print-out-paths .#nixosConfigurations.myhost.config.system.build.toplevel \
| snowstead push mainPaths must be built first; the CLI never evaluates or builds an installable.
Each path is published only after the references pushed with it, so an
interrupted push never leaves a published path whose references are missing.
Use --no-deps to push only the exact paths given; then a path already in the
cache is skipped without needing a local copy. .drv paths are build recipes,
not outputs, and are rejected unless you pass --allow-drv.
Use --resumable to save a push manifest on servers with background publication
workers enabled. The CLI prints a job ID before starting work. Compression and
transfer proceed independently of reference publication; the server publishes
each path only when its manifest prerequisites are available. The CLI reports
success only after all manifest members are available in the target cache.
Unlike ordinary pushes, upstream availability does not skip these members.
snowstead push main ./result --resumable
snowstead push main --resume JOB_IDResume uses the saved manifest, without paths or stdin. It reuses prepared
uploads and transfers only missing work. Keep unprepared store paths locally
until the push completes. --no-deps --resumable saves only the selected paths;
resume cannot change that selection. Interruptions and publication timeouts
leave unfinished jobs resumable while their server leases remain valid.
Resuming a completed job checks availability but never restores deleted paths.
Normal output contains a summary and any errors. Add --verbose (or -v) for
tracing diagnostics on stderr, including compression sizes/times and upload
request status/retries. Credentials and signed upload URLs are not logged.
RUST_LOG alone does not enable logging.
snowstead push main /run/current-system --verboseFor scripts and agents, snowstead push --json prints the result as one JSON
object on stdout instead of the Done: line. Progress and errors stay on
stderr.
{"pushed":12,"skipped":340,"upstream":25,"failed":1,"errors":[{"path":"/nix/store/…-app","message":"publication failed (status 507)"}],"cancelled":false}A push that stops early (a fatal error, Ctrl-C, or SIGTERM) still prints this
summary, or the Done: line, for the paths it finished, and aborts its open
upload sessions. If the push fails before any upload starts, nothing is printed
on stdout; use the exit code. --json affects only push.
Exit codes are stable:
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | Any other failure |
| 2 | Usage error (invalid arguments) |
| 3 | The push finished, but some store paths failed to upload |
| 4 | The server answered 401 or 403; run snowstead login or check SNOWSTEAD_TOKEN |
| 130 | Cancelled by Ctrl-C or SIGTERM |
A push that stops on a fatal error exits with that error's code (1 or 4), even
when some paths had already failed. Code 4 means a whole request was refused;
a 403 for a single store path is reported in errors and exits with 3.
Compression defaults to zstd level 3; use --compression-level <1-19> to change
it. Independent paths use a bounded native worker pool. NARs of at least
64 MiB use up to four additional zstd workers; the producer and codec workers
share one CPU budget, rather than multiplying workers by --jobs.
Resumable pushes limit active preparation and multipart transfer to --jobs,
but allow up to max(jobs, 8) paths to await preparation receipts. This lets
small inline uploads fill an eight-path batch even with --jobs 1. Queued
frames retain their charge against the same upload-buffer budget.
The Rust CLI streams compressed NARs directly into multipart uploads: no NAR
temporary files, whole-NAR buffering, or compression pass just to learn the
size. Upload buffers share a global budget of 64 MiB, or one negotiated part
if a very large NAR needs larger parts. Codec memory is additional. Slow
uploads pause compression; queued and retrying parts keep their buffer
reservation, so retries reuse the same bytes without recompression.
The server reserves exact-size parts from a separate, bounded staging budget and grants 15-minute presigned S3 UploadPart URLs for a private staging upload. The CLI sends part bodies directly to storage, without API credentials or a server-side upload buffer. The signature binds the staging key, upload ID, part number, and Content-Length. Re-reserving parts renews their grants without replacing the provider upload. Small inline uploads still use the API.
Finalization freezes the ordered part receipts in PostgreSQL before completing the private staging object. The server verifies the compressed size and hash and the decompressed NAR size and hash before publication. Completed or aborted provider uploads no longer accept their old grants. Abort and expiry reclaim staging uploads through the durable cleanup queue; R2's incomplete-upload lifecycle is the backstop for a crash during provider initialization (seven days by default). Presigned URLs are bearer credentials: do not log or share them. Revoking an API token prevents finalization but does not revoke an already-issued URL; aborting its provider upload does.
Deployment replaces the old relayed-part upload path. Drain or cancel in-flight
push jobs before upgrading, then restart any cancelled pushes; their old
multipart sessions cannot resume against the direct-storage server. The deploy
script runs the candidate's read-only check-upload-cutover command before
activation. Startup repeats the check and refuses active relayed sessions with
uploaded parts rather than discarding their bytes. It also refuses upload
sessions and reservations that predate staged uploads. Published NARs are
unaffected.
See the rollout procedure for verification
and rollback limits.
Publication owns a unique immutable NAR key before copying bytes, without a retained-byte reservation or a borrowed SQL connection. One short transaction then adopts the pointer and records completion, without checking the storage limit. A background evictor afterwards removes eligible least-recently-used entries across the org's products until it is back under its limit, so usage can briefly exceed it. From 110% of the limit new uploads get 503 and retry while eviction catches up. Completed retries return their recorded result and cannot restore a deleted path. Writers in one organization run concurrently. A SQL decision that meets a conflicting writer waits briefly or fails fast with a retryable 503; it is not literally lock-free. Cleanup of retired exact keys is asynchronous. Direct CDN downloads do not update access times; pushes and service reads update use. Each session uploads private staging bytes, even when the hash is already stored. Session creation overlaps transfers instead of delaying the first upload until every path has a session.
Only the streaming upload protocol is supported; the old upload-session, blob, complete, publish, and commit routes have been removed. Deploy database migrations and the Rust server before using the streaming CLI. It does not fall back to disk spooling on older servers. Config and credential writes still use atomic temporary files.
The CLI sends User-Agent: snowstead-cli/<version> and its protocol version in
Snowstead-Protocol-Version. Upload, missing-path, and push-job responses carry
the server's supported range in Snowstead-Protocol-Version and
Snowstead-Min-Protocol-Version; an unsupported client gets HTTP 400 and the
CLI says whether to upgrade itself or the server. Requests without the header
are treated as version 1 and refused by version-2 servers. Deploy the server
and update the CLI together: version 2 makes multipart finalization asynchronous.
GET /_api/v1/protocol reports the range and capabilities without credentials.
Multipart finalization first seals the immutable receipts and records durable
work in PostgreSQL, then returns HTTP 202. The CLI polls
POST /_api/v1/uploads/status with {"sessions":["<session-id>"]} until each
upload is published, prepared for its push job, or rejected. Queued and verifying
results are not successful publications and do not spend the CLI's transport
retry budget. Bounded workers recover accepted work after a restart without
requiring the client to resubmit its receipt. Storage reads have a network-stall
deadline, not a whole-transfer deadline that counts downstream backpressure.
Integrity verification and publication fences still apply before bytes become
available; asynchronous finalization does not bypass them.
Uploads that wrote straight to a NAR's final key predate staged uploads and
are no longer protected or reclaimed. check-upload-cutover and startup refuse
to proceed while such an upload session or a legacy-unresolved reservation
remains: an operator must reconcile the ambiguous R2 operation and remove the
row first. Staged session bytes are retained for retries and removed by
garbage collection after expiry.
Old raw CDN metadata and client caches can outlive SQL changes. Slug changes, deletion, and signing rotation do not erase those copies; see the deployment caveats.
For local CLI development, run cargo build -p snowstead-cli and
cargo test -p snowstead-cli inside nix develop. pnpm check includes the
Rust checks and a real CLI upload against the disposable local stack.
For development-only function and thread profiling, use
nix run .#hotpath -- push main /nix/store/…. This runs the same CLI with
Hotpath annotations; Hotpath is a Cargo dev-dependency, not linked into the
normal CLI. Set HOTPATH_OUTPUT_FORMAT=json and HOTPATH_OUTPUT_PATH=report.json
to save a report instead of printing it. Reports include metadata queries,
NAR encoding, buffer backpressure, batch waits, transfers, and finalization.
Async durations include waiting, and concurrent/nested durations overlap;
do not add them together or treat them as CPU time. poop is available in the
Linux development shell for separate CPU-counter comparisons. Profiling adds
overhead, so use the normal binary for throughput benchmarks.
To isolate local work from HTTP, profile existing store paths with
nix run .#hotpath -- local --closure /nix/store/… --rounds 1.
This development-only command resolves the closure, queries metadata, and
prepares NARs sequentially, without loading credentials or uploading anything.
It validates each NAR against the Nix database and discards encoded output
directly into a sink. Hotpath separates Nix queries and encoding; the Zig
call includes filesystem reads, hashing, and compression.
It is elapsed-time instrumentation, not a CPU sample inside Zig or libzstd.
For throughput without profiler overhead, build the same development target:
RUSTFLAGS="--cfg snowstead_local" cargo build --release -p snowstead-cli --example snowstead
target/release/examples/snowstead local /nix/store/… --rounds 5
target/release/examples/snowstead local /nix/store/… --rounds 5 --upstreamRun inside nix develop. --upstream replaces the native walker with
nix-store --dump, keeping the same Zig hashing, zstd settings, and sink
output. Use --compression-level 0 for raw NAR encoding plus hashing,
or the default level 3 for upload preparation; --threads 4 enables four
additional zstd workers. Use the same Nix on PATH for both runs. JSON lines
report metadata timings and input MB/s (decimal), not compressed-output MB/s.
One warm-up pass is omitted from throughput rows but included in Hotpath.
Results describe warm-cache, sequential local preparation, not parallel upload
speed, durable disk throughput, or a pure in-process libnix comparison.
See docs.md for token management and parallelism options.
# configuration.nix or flake
nix.settings.substituters = [ "https://cache.example.com/main" ];Source-available. See LICENSE.