Skip to content

deploying.md

Latest commit

 

History

History
292 lines (234 loc) · 13.3 KB

deploying.md

File metadata and controls

292 lines (234 loc) · 13.3 KB
Raw

Deploying Seedling

Seedling ships as three binaries — the seedling daemon, the seedling-web interface, and the seedling-ctl operator CLI. On Debian/Ubuntu the supported path is the seedling package from the BES APT repository; the same binaries are also published as standalone tarballs for other Linux hosts, and seedling-ctl and seedling-web are built for macOS and Windows too.

The daemon is Linux-only: it loads the jool kernel module for NAT64, programs nftables, drives systemd, and controls podman. seedling-ctl and seedling-web have no such coupling and run anywhere.

Install from the APT repository

Configure the repository once:

curl -fsSL https://tools.ops.tamanu.io/apt/bes-tools.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/bes-tools.gpg
echo "deb [signed-by=/etc/apt/keyrings/bes-tools.gpg] https://tools.ops.tamanu.io/apt stable main" | sudo tee /etc/apt/sources.list.d/bes-tools.list
sudo tee /etc/apt/preferences.d/bes-tools <<EOF
Package: *
Pin: origin tools.ops.tamanu.io
Pin-Priority: 999
EOF
sudo apt-get update
sudo apt-get install seedling

The package depends on podman (5.x — podman 4 is too old and 6 is not yet supported), nftables, libsystemd0, and bestool (the BES deployment CLI, which drives Tamanu through the daemon on a Seedling host — see Host tooling), and recommends btrfs-progs (for named-volume snapshots) and jool-dkms + jool-tools (for NAT64). Recommends are installed by default; if you run without NAT64 or on a non-btrfs data directory you can skip them with --no-install-recommends. It also suggests tailscale, used to front the web interface (see below); install it separately if you want that.

What the package installs

  • Binaries: /usr/bin/seedling, /usr/bin/seedling-web, /usr/bin/seedling-ctl.
  • Units: seedling.service and seedling-web.service (both enabled and started on install), plus seedling-web-tailscale-serve.service (shipped disabled; enable it to expose the web interface over Tailscale).
  • State: /var/lib/seedling (data directory: database, keys, authorized keys) and /var/log/seedling (audit log, rotated by /etc/logrotate.d/seedling).
  • Config: /etc/seedling/web.toml (a conffile — your edits survive upgrades).
  • Host tooling credentials: /etc/bestool/seedling.key, pre-generated and authorised with the daemon (see Host tooling).

On first install the daemon starts immediately with sensible defaults (--data-dir /var/lib/seedling). Workloads keep running while the daemon is restarted, so upgrades (apt-get upgrade) are non-disruptive.

Known host issue: AppArmor denies container signals

On hosts where AppArmor mediates signals between the container stack's profiles without matching rules (seen on Ubuntu 25.04, apparmor 4.1.0~beta5 with crun 1.20 and podman 5.4; tracked upstream as Launchpad #2040483), workloads break in two ways:

  • Signals between processes inside a container are denied, because no-new-privileges (which Seedling sets on every workload container) prevents the exec-time profile transition and sibling processes end up under mismatched confinement. PostgreSQL 18 is a prominent casualty: its latch wakeups are SIGURG between backends and io workers, so with the default io_method = worker every connection hangs in authentication until too many clients already. Tools that drop privileges in-process (gosu) spin forever for the same reason.
  • Stop signals from the runtime are denied, so every container stop escalates to SIGKILL after the timeout, and stateful workloads go through crash recovery on each restart.

The diagnostic signature is in dmesg:

apparmor="DENIED" operation="signal" ... signal=urg peer="containers-default-..."

Remedies, in preference order: a distribution whose AppArmor profiles carry the signal rules (the Launchpad fix); otherwise disable the podman profile (ln -s /etc/apparmor.d/podman /etc/apparmor.d/disable/ && apparmor_parser -R /etc/apparmor.d/podman) and set runtime = "runc" in a /etc/containers/containers.conf.d/ drop-in, which restores working signal delivery at the cost of confining podman no further than root already is.

Operator access (seedling-ctl)

seedling-ctl authenticates to the daemon with its own key. Authorise your operator key once:

# Print your client key fingerprint (generates the key on first run).
seedling-ctl client fingerprint

# Authorise it with the daemon (needs write access to the data directory).
echo "<your-fingerprint> your-name" | sudo tee -a /var/lib/seedling/authorized_keys
sudo systemctl restart seedling.service   # the daemon imports new entries on start

On first connection seedling-ctl captures and asks you to confirm the daemon's fingerprint (trust-on-first-use). You can read it ahead of time from /var/lib/seedling/oi.fingerprint.

The daemon's OI listens on loopback and the tailnet (the unit binds --interface lo,tailscale0), so you can run seedling-ctl remotely against the host's MagicDNS name — --endpoint accepts a hostname, not just an IP:

seedling-ctl --endpoint <node>.<tailnet>.ts.net:7891 status

The OI is mTLS with authorised-key gating, so exposing it on the tailnet is safe. The unit orders after tailscaled and waits (briefly, best-effort) for the tailnet address at startup, so a normal boot binds tailscale0 automatically; if you enable Tailscale on an already-running host, sudo systemctl restart seedling.service to pick it up.

Host tooling (bestool)

bestool is a hard dependency of the package. On a Seedling host it is the tool that operates Tamanu — bestool tamanu start, stop, restart, status, logs, psql, doctor — and it detects the host by the presence of seedling.service, then acts through the OI instead of through the host service manager.

So that it works on a freshly provisioned host with no operator step, the package bootstraps a host-wide interface identity for it, the same way it does for the web interface:

  • /etc/bestool/seedling.key, generated on install (and backfilled on upgrade if absent), mode 0600 and owned by root.
  • Its fingerprint authorised in /var/lib/seedling/authorized_keys under the label bestool.

Because the key is root-only, the host identity is reachable by sudo bestool tamanu ... and by root-run automation and by nothing else, and grants nothing that root does not already have: root can authorise any key it likes by writing to the data directory.

Picking the key up is bestool's side of the contract, and is not in every release yet. A bestool that reads it still prefers an operator's own key where they have one, so the daemon's record of who acted names the person; it falls back to the host identity otherwise, re-running itself under sudo to reach it rather than asking the operator to. One that does not read it uses the operator's own key only, which has to be authorised as under Operator access.

To see the entry, or to revoke it:

seedling-ctl user list
seedling-ctl user remove <fingerprint>

Removing the key file alone does not revoke it — the daemon imports authorized_keys into its database on start, so revoke through seedling-ctl user remove (or the web interface). A dpkg --purge seedling deletes the key file along with the rest of the daemon's state.

Reaching Canopy through bestool

Seedling has no Canopy identity of its own. Giving it one would mean a second enrolment and a second key to rotate on every host, so instead the connected bestool offers to carry Seedling's Canopy requests: Seedling hands over a whole HTTP request on the interface connection, bestool issues it under the host's device identity, and hands the response back. Seedling uses that channel to report its own health to Canopy as the seedling source, alongside whatever bestool reports as its own.

Nothing happens on a host where no client offers one. Seedling never dials out — bestool initiates both the connection and the offer — so a host without it, or without a Canopy auth path, simply has the facility sitting idle: no retry loop, no connection timeout, and no fault to explain.

To see the current state, or to turn the facility off on a host that has a bestool but should not appear in Canopy:

seedling-ctl canopy status
seedling-ctl canopy disable
seedling-ctl canopy enable

Disabling refuses new offers and revokes any live one immediately, rather than waiting for the carrying client to reconnect. The same state and the on/off control are on the web interface's Canopy page.

There is deliberately no way to relay an arbitrary request through the OI. The relay carries what the runtime itself needs, and an interface for relaying anything else would hand every authorised operator the full authority of the carrying client's Canopy identity. The path is exercised end to end by the status reports the runtime already sends, whose outcome canopy status reports.

The web interface (seedling-web)

seedling-web.service is enabled and started on install. Its daemon credentials are already bootstrapped — the package pre-generated /var/lib/seedling/web.key, authorised that key in /var/lib/seedling/authorized_keys, and the unit pins the daemon from /var/lib/seedling/oi.fingerprint. Its HTTP listener binds loopback only and authenticates operators via Tailscale identity headers (--trust-tailscale-headers), so out of the box it runs but the HTTP interface is not reachable until you put a front-end in front of it. Its WebTransport listener is bound separately on the tailnet (--wt-interface lo,tailscale0) — see below.

Tailscale (default)

Expose the loopback interface over your tailnet with tailscale serve, which terminates HTTPS (the secure context WebTransport needs) and injects the Tailscale-User-* identity headers seedling-web trusts. A unit for this ships disabled; enable it once Tailscale is up on the host:

sudo systemctl enable --now seedling-web-tailscale-serve.service

That runs tailscale serve --https=7895 --bg localhost:7894, so the interface is reachable at https://<node>.<tailnet>.ts.net:7895/. It uses 7895 (in Seedling's 789x range), not 443 — 443 is reserved for app workloads (Caddy). If Tailscale is not installed the unit is a no-op (a ConditionPathExists=/usr/bin/tailscale gate skips it rather than failing), so it is safe to leave enabled. To stop exposing the interface, systemctl disable --now seedling-web-tailscale-serve.service (which runs tailscale serve --https=7895 off).

Only loopback and the local tailscale serve reach the HTTP port, so the trusted identity headers cannot be spoofed by tailnet peers.

WebTransport (used for the live session) is HTTP/3 over QUIC and cannot be carried by tailscale serve (a TCP HTTP proxy), so the browser connects to it directly at <node>.<tailnet>.ts.net:7893. The unit binds it there with --wt-interface lo,tailscale0; it is gated by a per-session token and pins the server certificate, so exposing it on the tailnet is safe without header trust. The unit orders after tailscaled and waits (briefly, best-effort) for the tailnet address at startup, so a normal boot binds tailscale0 automatically. If you enable Tailscale on an already-running host, sudo systemctl restart seedling-web.service to pick it up — otherwise the browser reaches the login page but the session can't connect and falls back to the password prompt.

Password login (alternative)

To use a password instead of Tailscale, set an Argon2id hash in /etc/seedling/web.toml (generate one with the argon2 CLI from the argon2 package):

printf '%s' 'your-password' | argon2 "$(head -c16 /dev/urandom | base64)" -id -e

Paste the resulting $argon2id$... string as password_hash under [auth], drop --trust-tailscale-headers with a drop-in, bind a reachable interface, and front the HTTP port with a TLS-terminating reverse proxy (WebTransport requires a secure context). Note the WebTransport port cannot go through that proxy either, so bind it on the reachable interface too (--wt-interface eth0, or --wt-listen); the browser connects to it directly on :7893:

sudo systemctl edit seedling-web.service
# [Service]
# ExecStart=
# ExecStart=/usr/bin/seedling-web --config /etc/seedling/web.toml \
#     --daemon-fingerprint-file /var/lib/seedling/oi.fingerprint \
#     --key-file /var/lib/seedling/web.key \
#     --interface eth0 --wt-interface eth0

Standalone binaries (no APT)

Every release also publishes each binary as a .tar.zst under https://tools.ops.tamanu.io/:

  • seedling, seedling-ctl, seedling-web for Linux (x64 and arm64).
  • seedling-ctl and seedling-web for macOS (Intel and Apple Silicon) and Windows (x64).
https://tools.ops.tamanu.io/<name>/<version>/<name>-<target>-<version>.tar.zst
https://tools.ops.tamanu.io/<name>/latest/<name>-<target>-<version>.tar.zst

where <target> is a Rust target triple (e.g. x86_64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-pc-windows-msvc). Extract with zstd -d < file.tar.zst | tar -x. These carry build-provenance attestations, verifiable with gh attestation verify <file> -R beyondessential/seedling.

Running the daemon from a tarball means creating the state directories, the systemd unit, and the log rotation yourself; the package is the supported way to run the daemon.