caddy-rqlite-storage

A CertMagic / Caddy storage backend on top of rqlite — distributed, Raft-replicated SQLite.

For a fleet of Caddy instances that need a shared certificate store: certificates and keys are shared, and ACME challenge tokens live in rqlite, so an ACME challenge landing on any instance can be solved, and a cert issued by one instance is immediately usable by the others.

Why rqlite

The hard requirement of a multi-instance Caddy cluster is correct distributed locking — two instances must never solve or issue the same certificate concurrently. rqlite's Raft consensus provides linearizable locking for that. If you already run an rqlite cluster, this backend reuses it — no separate datastore to operate — and gets stronger locking than a single Redis would.

Layout

Package	Imports	Purpose
. (rqlitestorage)	stdlib only	Transport-agnostic core: storage + locking logic, rqlite HTTP transport. Fully unit-tested.
./caddymodule	caddy/v2, certmagic	Registers the caddy.storage.rqlite Caddy module and adapts the core to certmagic.Storage.

The split keeps the core's tests fast and dependency-light (only an in-memory SQLite for the test fake), while the Caddy glue stays isolated.

Build into Caddy

xcaddy build \
  --with github.com/caddy-dns/rfc2136 \
  --with github.com/mholt/caddy-ratelimit \
  --with github.com/tissue-systems/caddy-rqlite-storage/caddymodule

Configure (Caddyfile global options)

{
    storage rqlite {
        url      http://127.0.0.1:4001   # this instance's local rqlite node
        username {$RQLITE_USER}           # optional HTTP basic-auth
        password {$RQLITE_PASSWORD}
        lock_ttl 60s                      # how long a held ACME lock survives before it may be stolen
        read_level weak                   # rqlite read consistency: weak (default) | none | linearizable | strong
        leader_retry 10s                  # how long a request retries rqlite's 503 "leader not found" (0 = off)
    }
}

Each instance points at its local rqlite node; Raft replication makes the store identical everywhere. Lock acquisition is always a leader write (consistent). Cert/key lookups default to weak reads (routed through the Raft leader, so the store always sees its own writes, at the cost of one intra-cluster hop). read_level none (local FSM reads) is faster but is safe only on single-node deployments: on a cluster, a node that is not the leader can read its own just-forwarded write back as missing (follower FSM lag), which fails CertMagic's write-then-read storage preflight and aborts certificate obtains with failed storage check: file does not exist.

Rebooting one node costs the cluster a Raft election, during which its peers answer 503 leader not found. A request that lands in that window is retried for leader_retry (default 10s, exponential backoff, cancelled with the context) so a renewal or a lock acquisition rides the election out instead of failing. Only 503 is retried: it means the node had no leader to apply or forward to, so the statement never entered the Raft log and a replay cannot double-apply it. A transport error is never retried, because a write may have been applied with the response lost. The budget is bounded on purpose — Caddy loads every managed certificate through this path at startup, and an unbounded wait would trade a visible failure for a hung process.

Schema

Created automatically on provision (idempotent):

CREATE TABLE caddy_storage (key TEXT PRIMARY KEY, value TEXT /*base64*/, size INTEGER, modified INTEGER);
CREATE TABLE caddy_locks   (name TEXT PRIMARY KEY, token TEXT, expires INTEGER);

Test

go test ./... -race

The core tests run the exact SQL the store emits against an in-memory SQLite, covering storage round-trips (incl. binary), not-found semantics, listing (recursive / non-recursive / LIKE-escaping), the lock primitives (acquire / steal-after-expiry / refresh / release / token-ownership), an N-goroutine mutual-exclusion contention test, and the rqlite HTTP wire format end-to-end.

License

MIT