Skip to content
Polymath-ASPublic

About

No description, website, or topics provided.

Resources

Stars

7 stars

Watchers

0 watching

Forks

Repository files navigation

Snowstead

A cache for Nix paths and Go, Rust, and C/C++ compiler outputs.

The Rust server works with unmodified sccache clients over WebDAV. Enable --protocols sccache and use scoped sccache:read,sccache:write tokens for Rust and C/C++ CI builds.

The native Go helper uses https://cache.snowstead.com/_api/v1/go/<org-slug>/<cache-name> as its base URL. Public caches allow anonymous reads; private reads require go:read, and every upload requires go:write. Nix permissions do not grant Go access. Give write access only to protected CI jobs building reviewed code: hashes detect corruption, not malicious outputs from authorized writers.

New caches require explicit protocols: snowstead cache create main --protocols nix for Nix, --protocols go for Go, --protocols sccache for Rust/C/C++, or a comma-separated combination such as --protocols nix,go,sccache; there is no default. API creation requests require the corresponding protocols array. Existing-cache updates may omit it to preserve the selection, or add protocols but not remove them (for example, snowstead cache set main --protocols nix,go). Existing database caches keep Nix and Go enabled; sccache requires explicit opt-in.

Go objects use the private bucket, expire seven days after upload, and count toward organization storage limits. Resetting Go leaves Nix paths intact. The Rust server owns all protocols, management, and maintenance.

The control-plane API exposes token last-use times, cross-product eviction candidates and 30-day history, named Nix closure pins, and a whole-cache eviction that keeps settings and credentials. These features are API-only; they do not add dashboard or CLI commands.

Workspace

workers/gateway/      Forwarding-only gateway to API_ORIGIN
workers/server/       Rust API, protocols, maintenance, migrations, and tests
apps/cli/             CLI for pushing store paths
pkg/protocol/         Wire types, limits, and version shared by the CLI and server
apps/go-cache/        Native GOCACHEPROG helper
infra/                Rust service deployment
scripts/              Development and verification harnesses

Setup

nix develop # drops you into a shell with node, pnpm, wrangler, postgres
pnpm install

Develop

pnpm check # type-check + tests across the workspace

Run the Rust server locally on http://127.0.0.1:8787:

pnpm dev

It starts disposable Postgres and S3 services, applies the Rust migrations, and generates fresh local secrets. It does not read or write .dev.vars; local state is discarded when the stack stops. Database migrations live in workers/server/migrations.

Deploy

Deploy the standalone Rust service with the self-host infrastructure modules. See docs.md for gateway configuration and cutover cautions. The gateway only forwards to API_ORIGIN; it has no storage or backend secrets. Hosted public reads retain the existing R2/CDN origin.

# Deploy only the production gateway; no database migrations run here.
pnpm deploy

The hosted configuration requires per-cache keys and fails closed without the master key. Existing caches without their own key need a staged key and explicit trust-overlap activation before they can publish. Self-hosters who deliberately want the legacy shared key can set REQUIRE_CACHE_SIGNING_KEYS to "false" and configure SIGNING_PRIVATE_KEY and SIGNING_KEY_NAME. See the signing-key lifecycle and recovery guidance in docs.md.

Optional read replica

Set DATABASE_READ_URL to a replica PostgreSQL URI to serve only GET /_api/v1/admin/overview totals from that replica. These reporting values may lag. All other queries, including authorization and organization overviews, continue to use the primary DATABASE_URL.

For PlanetScale direct replicas, use port 5432 and append %7Creplica to the URI username. Use verified TLS and a read-only role. DATABASE_READ_MAX_CONNECTIONS defaults to 8 and must be positive. This is a separate per-process pool limit; budget for application instances, deployment overlap, and other clients. Do not assume replica connections divide evenly. The primary pool keeps at least 2 connections warm and the replica pool keeps at least 1, capped by each pool's configured maximum. Excess idle connections are eligible to close after 60 seconds.

When DATABASE_READ_URL is unset, reads share the existing primary pool with no additional connections. An invalid or unreachable configured replica fails startup. Runtime replica errors do not fall back to primary. Migrations ignore replica settings, and pnpm dev clears them to keep the local stack disposable.

Push store paths

Build the CLI (or nix run . for one-off use):

nix build       # produces ./result/bin/snowstead

Build the server with nix build .#server. Both packages use Crane to keep vendored crate sources and compiled dependencies in separate Nix derivations. Source-only changes rebuild the application, not its external crates. Changes to dependencies, the toolchain, or build features can rebuild the dependency cache. The first build populates it; later builds reuse it while its store paths remain available locally or through a configured binary cache. Local development still uses Cargo's incremental builds inside nix develop.

The CLI is a native Rust binary using the Zig NAR core. On Linux, NAR serialization, SHA-256, and zstd compression run in-process; Nix is still used for path metadata and closure resolution. On macOS, Nix also serializes the NAR. The flake includes Nix as a runtime dependency, not Node or the workspace's Worker dependencies. Existing credentials in ~/.config/snowstead/config.json continue to work. The full development environment is separate (nix develop).

Sign in once (use --no-browser over SSH and open the printed URL elsewhere):

snowstead login

Approve the organization and token lifetime in your browser: 30 days, 90 days (the default), 365 days, or never. The ten-minute device approval window is separate from the saved token's lifetime.

For CI, set both SNOWSTEAD_TOKEN and SNOWSTEAD_URL (the cache base URL). For self-hosted manual tokens, use snowstead login --token-stdin --url https://cache.example.com --name prod < token-file. Manual tokens are configured without remote validation; positional login tokens are no longer accepted. snowstead auth status [--server <profile>] shows local configuration, and snowstead logout [--server <profile>] removes local credentials without revoking the remote token. See the CLI authentication guide for profile details.

snowstead push uploads each given path together with its runtime closure (everything it references), and reads paths from stdin when none are given as arguments:

snowstead push main ./result
snowstead push main /run/current-system
nix build --no-link --print-out-paths .#nixosConfigurations.myhost.config.system.build.toplevel \
  | snowstead push main

Paths must be built first; the CLI never evaluates or builds an installable. Each path is published only after the references pushed with it, so an interrupted push never leaves a published path whose references are missing. Use --no-deps to push only the exact paths given; then a path already in the cache is skipped without needing a local copy. .drv paths are build recipes, not outputs, and are rejected unless you pass --allow-drv.

Use --resumable to save a push manifest on servers with background publication workers enabled. The CLI prints a job ID before starting work. Compression and transfer proceed independently of reference publication; the server publishes each path only when its manifest prerequisites are available. The CLI reports success only after all manifest members are available in the target cache. Unlike ordinary pushes, upstream availability does not skip these members.

snowstead push main ./result --resumable
snowstead push main --resume JOB_ID

Resume uses the saved manifest, without paths or stdin. It reuses prepared uploads and transfers only missing work. Keep unprepared store paths locally until the push completes. --no-deps --resumable saves only the selected paths; resume cannot change that selection. Interruptions and publication timeouts leave unfinished jobs resumable while their server leases remain valid. Resuming a completed job checks availability but never restores deleted paths.

Normal output contains a summary and any errors. Add --verbose (or -v) for tracing diagnostics on stderr, including compression sizes/times and upload request status/retries. Credentials and signed upload URLs are not logged. RUST_LOG alone does not enable logging.

snowstead push main /run/current-system --verbose

For scripts and agents, snowstead push --json prints the result as one JSON object on stdout instead of the Done: line. Progress and errors stay on stderr.

{"pushed":12,"skipped":340,"upstream":25,"failed":1,"errors":[{"path":"/nix/store/…-app","message":"publication failed (status 507)"}],"cancelled":false}

A push that stops early (a fatal error, Ctrl-C, or SIGTERM) still prints this summary, or the Done: line, for the paths it finished, and aborts its open upload sessions. If the push fails before any upload starts, nothing is printed on stdout; use the exit code. --json affects only push.

Exit codes are stable:

Code Meaning
0 Success
1 Any other failure
2 Usage error (invalid arguments)
3 The push finished, but some store paths failed to upload
4 The server answered 401 or 403; run snowstead login or check SNOWSTEAD_TOKEN
130 Cancelled by Ctrl-C or SIGTERM

A push that stops on a fatal error exits with that error's code (1 or 4), even when some paths had already failed. Code 4 means a whole request was refused; a 403 for a single store path is reported in errors and exits with 3.

Compression defaults to zstd level 3; use --compression-level <1-19> to change it. Independent paths use a bounded native worker pool. NARs of at least 64 MiB use up to four additional zstd workers; the producer and codec workers share one CPU budget, rather than multiplying workers by --jobs. Resumable pushes limit active preparation and multipart transfer to --jobs, but allow up to max(jobs, 8) paths to await preparation receipts. This lets small inline uploads fill an eight-path batch even with --jobs 1. Queued frames retain their charge against the same upload-buffer budget. The Rust CLI streams compressed NARs directly into multipart uploads: no NAR temporary files, whole-NAR buffering, or compression pass just to learn the size. Upload buffers share a global budget of 64 MiB, or one negotiated part if a very large NAR needs larger parts. Codec memory is additional. Slow uploads pause compression; queued and retrying parts keep their buffer reservation, so retries reuse the same bytes without recompression.

The server reserves exact-size parts from a separate, bounded staging budget and grants 15-minute presigned S3 UploadPart URLs for a private staging upload. The CLI sends part bodies directly to storage, without API credentials or a server-side upload buffer. The signature binds the staging key, upload ID, part number, and Content-Length. Re-reserving parts renews their grants without replacing the provider upload. Small inline uploads still use the API.

Finalization freezes the ordered part receipts in PostgreSQL before completing the private staging object. The server verifies the compressed size and hash and the decompressed NAR size and hash before publication. Completed or aborted provider uploads no longer accept their old grants. Abort and expiry reclaim staging uploads through the durable cleanup queue; R2's incomplete-upload lifecycle is the backstop for a crash during provider initialization (seven days by default). Presigned URLs are bearer credentials: do not log or share them. Revoking an API token prevents finalization but does not revoke an already-issued URL; aborting its provider upload does.

Deployment replaces the old relayed-part upload path. Drain or cancel in-flight push jobs before upgrading, then restart any cancelled pushes; their old multipart sessions cannot resume against the direct-storage server. The deploy script runs the candidate's read-only check-upload-cutover command before activation. Startup repeats the check and refuses active relayed sessions with uploaded parts rather than discarding their bytes. It also refuses upload sessions and reservations that predate staged uploads. Published NARs are unaffected. See the rollout procedure for verification and rollback limits.

Publication owns a unique immutable NAR key before copying bytes, without a retained-byte reservation or a borrowed SQL connection. One short transaction then adopts the pointer and records completion, without checking the storage limit. A background evictor afterwards removes eligible least-recently-used entries across the org's products until it is back under its limit, so usage can briefly exceed it. From 110% of the limit new uploads get 503 and retry while eviction catches up. Completed retries return their recorded result and cannot restore a deleted path. Writers in one organization run concurrently. A SQL decision that meets a conflicting writer waits briefly or fails fast with a retryable 503; it is not literally lock-free. Cleanup of retired exact keys is asynchronous. Direct CDN downloads do not update access times; pushes and service reads update use. Each session uploads private staging bytes, even when the hash is already stored. Session creation overlaps transfers instead of delaying the first upload until every path has a session.

Only the streaming upload protocol is supported; the old upload-session, blob, complete, publish, and commit routes have been removed. Deploy database migrations and the Rust server before using the streaming CLI. It does not fall back to disk spooling on older servers. Config and credential writes still use atomic temporary files.

The CLI sends User-Agent: snowstead-cli/<version> and its protocol version in Snowstead-Protocol-Version. Upload, missing-path, and push-job responses carry the server's supported range in Snowstead-Protocol-Version and Snowstead-Min-Protocol-Version; an unsupported client gets HTTP 400 and the CLI says whether to upgrade itself or the server. Requests without the header are treated as version 1 and refused by version-2 servers. Deploy the server and update the CLI together: version 2 makes multipart finalization asynchronous. GET /_api/v1/protocol reports the range and capabilities without credentials.

Multipart finalization first seals the immutable receipts and records durable work in PostgreSQL, then returns HTTP 202. The CLI polls POST /_api/v1/uploads/status with {"sessions":["<session-id>"]} until each upload is published, prepared for its push job, or rejected. Queued and verifying results are not successful publications and do not spend the CLI's transport retry budget. Bounded workers recover accepted work after a restart without requiring the client to resubmit its receipt. Storage reads have a network-stall deadline, not a whole-transfer deadline that counts downstream backpressure. Integrity verification and publication fences still apply before bytes become available; asynchronous finalization does not bypass them.

Uploads that wrote straight to a NAR's final key predate staged uploads and are no longer protected or reclaimed. check-upload-cutover and startup refuse to proceed while such an upload session or a legacy-unresolved reservation remains: an operator must reconcile the ambiguous R2 operation and remove the row first. Staged session bytes are retained for retries and removed by garbage collection after expiry.

Old raw CDN metadata and client caches can outlive SQL changes. Slug changes, deletion, and signing rotation do not erase those copies; see the deployment caveats.

For local CLI development, run cargo build -p snowstead-cli and cargo test -p snowstead-cli inside nix develop. pnpm check includes the Rust checks and a real CLI upload against the disposable local stack.

For development-only function and thread profiling, use nix run .#hotpath -- push main /nix/store/…. This runs the same CLI with Hotpath annotations; Hotpath is a Cargo dev-dependency, not linked into the normal CLI. Set HOTPATH_OUTPUT_FORMAT=json and HOTPATH_OUTPUT_PATH=report.json to save a report instead of printing it. Reports include metadata queries, NAR encoding, buffer backpressure, batch waits, transfers, and finalization. Async durations include waiting, and concurrent/nested durations overlap; do not add them together or treat them as CPU time. poop is available in the Linux development shell for separate CPU-counter comparisons. Profiling adds overhead, so use the normal binary for throughput benchmarks.

To isolate local work from HTTP, profile existing store paths with nix run .#hotpath -- local --closure /nix/store/… --rounds 1. This development-only command resolves the closure, queries metadata, and prepares NARs sequentially, without loading credentials or uploading anything. It validates each NAR against the Nix database and discards encoded output directly into a sink. Hotpath separates Nix queries and encoding; the Zig call includes filesystem reads, hashing, and compression. It is elapsed-time instrumentation, not a CPU sample inside Zig or libzstd.

For throughput without profiler overhead, build the same development target:

RUSTFLAGS="--cfg snowstead_local" cargo build --release -p snowstead-cli --example snowstead
target/release/examples/snowstead local /nix/store/… --rounds 5
target/release/examples/snowstead local /nix/store/… --rounds 5 --upstream

Run inside nix develop. --upstream replaces the native walker with nix-store --dump, keeping the same Zig hashing, zstd settings, and sink output. Use --compression-level 0 for raw NAR encoding plus hashing, or the default level 3 for upload preparation; --threads 4 enables four additional zstd workers. Use the same Nix on PATH for both runs. JSON lines report metadata timings and input MB/s (decimal), not compressed-output MB/s. One warm-up pass is omitted from throughput rows but included in Hotpath. Results describe warm-cache, sequential local preparation, not parallel upload speed, durable disk throughput, or a pure in-process libnix comparison.

See docs.md for token management and parallelism options.

Use as a substituter

# configuration.nix or flake
nix.settings.substituters = [ "https://cache.example.com/main" ];

License

Source-available. See LICENSE.

About

No description, website, or topics provided.

Resources

Stars

7 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages