diff --git a/.gitignore b/.gitignore index 40c4a78..815e95c 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,11 @@ dist/ # Fetched kubeconfigs embed cluster client certificates and keys. *kubeconfig*.yaml sandbox-kubeconfig.yaml + +# Credentials for live testing. The DigitalOcean token in .env.production is a +# working key against a real account, and this repository is public — one +# `git add -A` is all it takes. +.env +.env.* +!.env.example +test/do/.state/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 9db0e5c..01bb2b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,110 @@ Notable changes per release. The release workflow publishes the section matching the tag it is building, so this file is the source of the release notes on GitHub. +## Unreleased + +### Added + +- **`k3helper serve` — a web interface.** The same binary, the same data layer + and the same diagnosis as the CLI and the TUI, shown in a browser: an + overview with a card per node, tables for pods, workloads, nodes and events, + logs and `describe` in a drawer, and Doctor's ranked findings with their + remediation. `--host` and `--port` work the way they do in any dev server; a + port left at its default steps past one already in use, an explicit one does + not. + + It is read-only — everything lists, describes or tails, and any non-GET is + refused — and binds to loopback unless given `--host` *and* `--allow-remote`, + because a per-launch session token is the only authentication there is. + + No Electron, no WebView2, no npm: the assets are embedded, so it runs on + Windows, macOS, Linux and in a container with nothing else installed. It + costs about 3MB of binary. +- **A Dockerfile**, `FROM scratch`, around 17MB. Credentials are mounted at run + time, never built in. + +Live testing against real DigitalOcean VMs, and the four bugs it found. + +### Added + +- **`k3helper bundle k3s`** builds an offline install bundle — the k3s binary, + the airgap image archive and the installer — verified against the release + sha256 manifest. `k3helper vm setup --bundle ` installs from it with + `INSTALL_K3S_SKIP_DOWNLOAD`, so the nodes need no internet at all. Proven on + two DigitalOcean droplets with egress blocked at the provider firewall: + `github=000, get.k3s.io=000`, cluster Ready. +- **`--k3s-version`** pins an exact release and skips the update.k3s.io channel + lookup. During testing that service served a Traefik default certificate from + every one of its addresses, which breaks `curl -sfL https://get.k3s.io | sh -` + everywhere; a pinned version fetches from the GitHub release instead. +- **`--join-address`** overrides the address other nodes dial to reach the + first server, for when k3helper reaches the nodes over one network and the + cluster talks over another. +- **`ssh.Client.WriteFileFrom`** streams a file to a node with progress, + instead of holding it in memory. The airgap image archive is 184MB. + +- **`--apt-mirror` and `--k8s-apt-repo`** point a node's package manager at + mirrors you run instead of the distribution's own archive. Both source + formats are rewritten — 24.04's deb822 `.sources` and older `.list` entries — + with a backup of every file first, and third-party repositories left alone. +- **`--via-proxy`** lends the nodes the operator's internet connection for the + length of an install, over the SSH connection already open to them, through a + proxy with a host allowlist. It is what makes an air-gapped kubeadm install + possible at all: unlike k3s, kubeadm needs apt packages and registry images + that do not fit in a bundle. Off unless asked for, announced when used, and + removed from the node afterwards. Proven on two DigitalOcean droplets with + egress blocked at the provider firewall: `✓ cluster ready`, both nodes Ready, + ten pods Running, `github=000`. + +### Fixed + +- **kubeadm advertised the wrong API server address**, for the same reason the + k3s path did: `kubeadm init` defaults it to the default route's interface, + which on a cloud VM is the public one, and the join command handed to every + agent is built from it. Both paths now share one resolver. +- **The TUI dashboard was blank for a kubeconfig cluster.** It drew one card + per node in the targets file, and a kubeconfig cluster has no nodes — so the + skipped host checks and the reason for them never reached the screen. An + empty dashboard reads as "all clear", which is what those results exist to + prevent. Skipped checks are also now counted in the summary line instead of + vanishing from `0 ok, 0 warn, 0 fail`. +- **`--bundle` was silently ignored with `--distro kubeadm`**, so an operator + asking for an offline install got an online one and found out on an + air-gapped node. It is refused now, as is `--bundle` with `--k3s-version`, + which contradict each other. +- **A bundle's architecture was recorded and never checked.** Installing an + arm64 build on an amd64 node would have failed as "cannot execute binary + file" after a 260MB upload. Every node is checked before any node is uploaded + to. +- **Uploaded bundles were neither verified nor cleaned up.** The manifest + already had the hashes; they are now checked on the node after the transfer, + and the staged copy is removed once the installer has run rather than left on + every node's disk. +- **cloud-init raced the installer.** A fresh cloud image is still replacing + ca-certificates when sshd starts answering, and https downloads fail TLS + verification until it finishes — which looks like a firewall problem and is + not one. Both install paths wait first; kubeadm needed it more, since its + prerequisites run apt straight into cloud-init's dpkg lock. +- **Agents joined on the wrong address.** The join address was discovered with + `hostname -I`, which returns a cloud VM's public address first — the one + address an air-gapped network cannot reach. Agents retried "failed to get CA + certs" indefinitely while the server ran fine beside them. It now comes from + the targets file: the address the operator chose, and the one k3helper has + just proved works by connecting over it. +- **`%!w()` in install failures.** A command that ran and exited non-zero + has no error to wrap, and the `%w` verb printed its own failure as the last + thing an operator saw when an install failed. +- **A healthy managed cluster scored 0% in the TUI.** Every host check is + skipped on a kubeconfig cluster, and skips were counted as "not OK". A skip + is now left out of both halves of the fraction, so the score reads `n/a` + rather than putting the most alarming number on screen for a healthy cluster. + +### Testing + +- `test/do/do.sh` provisions, air-gaps and destroys DigitalOcean droplets + through the v2 API for live scenario testing. Everything it creates is tagged + `k3helper-test`, so teardown can never touch anything else. + ## v0.5.0 Clusters you cannot SSH into. k3helper now reaches a cluster either by its diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..d54263c --- /dev/null +++ b/Dockerfile @@ -0,0 +1,42 @@ +# k3helper as a container, for `serve`. +# +# FROM scratch, because the binary needs nothing: the web assets are embedded +# in it, and the cluster is reached over SSH or by a kubectl that lives on the +# nodes. The image is the binary plus certificates, which is what makes it +# around 12MB rather than the few hundred a base image would add. +# +# docker build -t k3helper . +# docker run --rm -p 8787:8787 \ +# -v "$PWD/targets.yaml:/targets.yaml:ro" \ +# -v "$HOME/.ssh/id_ed25519:/key:ro" \ +# k3helper serve -t /targets.yaml --host 0.0.0.0 --allow-remote +# +# Mount credentials at run time; never build them in. An image with a private +# key in a layer keeps that key even after a later layer deletes it, and +# anywhere the image goes, the key goes. + +FROM golang:1.26-alpine AS build +WORKDIR /src + +# Dependencies first, so a change to the source does not refetch them. +COPY go.mod go.sum ./ +RUN go mod download + +COPY . . +ARG VERSION=dev +# CGO off: the result has to run on scratch, which has no libc to link against. +RUN CGO_ENABLED=0 go build \ + -ldflags "-s -w -X github.com/solutionforest/k3helper/internal/cli.version=${VERSION}" \ + -o /k3helper ./cmd/k3helper + +FROM scratch +# Certificates, for the API server's TLS and for `bundle k3s` reaching GitHub. +COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ +COPY --from=build /k3helper /k3helper + +# Nobody. There is nothing in this image to be root for, and a mounted key +# should be readable by this user rather than by everyone. +USER 65534:65534 +EXPOSE 8787 +ENTRYPOINT ["/k3helper"] +CMD ["serve", "--host", "0.0.0.0", "--allow-remote", "--no-browser"] diff --git a/Makefile b/Makefile index 008a10f..48c1b02 100644 --- a/Makefile +++ b/Makefile @@ -5,6 +5,12 @@ LDFLAGS := -ldflags "-X $(VERPKG)=$(VERSION)" # -s -w strips the symbol table and DWARF: ~25% smaller downloads, and Go # panics keep their function names because the runtime carries its own tables. RELFLAGS := -ldflags "-s -w -X $(VERPKG)=$(VERSION)" +# The k3s release the sandbox and the E2E install. Pinned on purpose: a channel +# is a lookup against update.k3s.io, and when that served a Traefik default +# certificate from every one of its addresses it took every `curl -sfL +# https://get.k3s.io | sh -` on the internet down with it — including CI's. +# A pinned version fetches straight from the GitHub release. +K3S_VERSION := v1.36.4+k3s1 PLATFORMS := linux/amd64 linux/arm64 darwin/arm64 darwin/amd64 windows/amd64 windows/arm64 # Windows will not execute a downloaded file without the extension, so those # two assets carry .exe. Everything else stays extensionless; install.sh builds @@ -157,6 +163,7 @@ sandbox-ssh: bootstrap: go run ./cmd/k3helper vm setup -t $(TARGETS) \ + --k3s-version $(K3S_VERSION) \ --server-extra-args "--snapshotter=native --disable=traefik" \ --agent-extra-args "--snapshotter=native" \ --kubeconfig sandbox-kubeconfig.yaml diff --git a/README.md b/README.md index fc84f7f..c6de6cf 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ k3helper is a portable k3s/kubernetes helper with a TUI. registry configure the private registries the cluster pulls from ctx list clusters defined in the targets file tui interactive dashboard + resource browser + serve the same, in a web browser (read-only) ``` ## Why @@ -86,6 +87,106 @@ Windows reaches clusters through a kubeconfig rather than over SSH — see reachable, but the host-layer commands (`vm setup`, `registry apply`) target Linux nodes. +### Air-gapped nodes + +Nodes with no route to the internet cannot run the usual installer: it reaches +update.k3s.io to resolve a channel, GitHub for the k3s binary, and a registry +for every image a pod pulls. Build a bundle where there *is* a connection, then +install from it: + +```bash +# on a machine with internet (a laptop, or a jump host inside the network) +k3helper bundle k3s --version v1.31.2+k3s1 --arch amd64 -o ./k3s-bundle +# k3s 75MB +# k3s-airgap-images.tar.zst 184MB ← imported into containerd on first start +# install.sh 37KB +# all verified against the release sha256 manifest + +# from anywhere that can reach the nodes over SSH +k3helper vm setup -t targets.yaml --bundle ./k3s-bundle +``` + +The nodes need nothing but SSH from wherever k3helper runs, and a route to each +other. `--bundle` uploads the binary and the image archive to every node, puts +them where the installer looks, and runs it with `INSTALL_K3S_SKIP_DOWNLOAD`. + +Two things worth knowing: + +- **Agents join on the address in your targets file**, not one discovered on + the server. `hostname -I` reports a cloud VM's public address first, which is + usually the one address an air-gapped network cannot use. Override it with + `--join-address` when k3helper reaches the nodes over one network and the + cluster talks over another. +- **Workload images still have to come from somewhere.** The bundle covers the + cluster's own images; for yours, point the nodes at an internal registry with + a `registries:` block and `k3helper registry apply`. + +### Air-gapped kubeadm: a mirror, or your own connection + +k3s installs offline from a bundle because it is one binary and one image +archive. kubeadm cannot: it needs apt packages and images from +`registry.k8s.io`, and neither fits in a file you carry in. Two answers. + +**Point at your own mirror.** Most sites that run air-gapped Kubernetes already +have one — Artifactory, Nexus, Satellite. This does not replace it, it points +at it: + +```bash +k3helper vm setup -t targets.yaml --distro kubeadm --apt-mirror https://nexus.corp/repository/ubuntu --k8s-apt-repo https://nexus.corp/repository/kubernetes +``` + +The distribution archive is substituted in both source formats — 24.04's +deb822 `.sources` files and older `.list` entries — and every file is backed up +as `*.k3helper.bak` first. Third-party repositories are left alone. + +**Or lend the nodes your connection.** When there is no mirror either, +`--via-proxy` opens a proxy on the machine running k3helper and reaches the +nodes through the SSH connection already open to them: + +```bash +k3helper vm setup -t targets.yaml --distro kubeadm --via-proxy +``` + +``` +--via-proxy: these nodes will reach the internet through this machine for the +length of the install, and only through it. + allowed: 30 default hosts (distribution mirrors, pkgs.k8s.io, registry.k8s.io, docker.io) + the tunnel and its configuration are removed when the install finishes. +``` + +The proxy resolves names on your side, so the nodes need no working DNS either +— which a genuinely cut-off machine does not have. Traffic is restricted to an +allowlist; anything else is refused with the flag that would permit it +(`--proxy-allow HOST`, or `--proxy-allow "*"`). Cluster-internal addresses +never go through the tunnel. + +Three things to be clear about: + +- **This gives an isolated machine a route out.** It is temporary, proxied and + allowlisted, but in some environments opening one at all is a policy breach + regardless. It is off unless asked for, and it says what it is doing every + time. That call is the operator's, and sometimes not theirs to make. +- **The tunnel is install-time only.** When `vm setup` finishes it is removed, + and the cluster goes back to having no internet — so it cannot pull a + workload image afterwards. For anything beyond the install, point the nodes + at an internal registry with a `registries:` block. +- **For a genuinely disconnected site, prefer k3s.** `--bundle` installs it with + no network at all, which is a better fit than a cluster that needed a + temporary hole to be built. + +### If the k3s channel service is down + +`--k3s-version` pins an exact release and skips the channel lookup entirely: + +```bash +k3helper vm setup -t targets.yaml --k3s-version v1.31.2+k3s1 +``` + +This is not hypothetical. During live testing `update.k3s.io` served a Traefik +default certificate from all three of its addresses, so every +`curl -sfL https://get.k3s.io | sh -` on the internet failed TLS verification. +Pinning a version fetches straight from the GitHub release and is unaffected. + ### From source ```bash @@ -440,6 +541,56 @@ A node it cannot reach is reported, never skipped — partial inspection must no 1. [85%] Node unreachable over SSH (no evidence could be gathered) ``` +### 6b. The web interface + +Same data, same checks, same diagnosis — in a browser instead of a terminal: + +```bash +k3helper serve # http://127.0.0.1:8787 +k3helper serve --port 9000 # somewhere else +k3helper serve --host 0.0.0.0 --allow-remote # reachable from the network +``` + +![overview](docs/screenshots/web-overview.png) + +The Overview is the dashboard: one card per node, every host check with its +verdict, and a banner that either says the cluster is healthy or how many +issues are waiting in Doctor. + +![pods](docs/screenshots/web-pods.png) + +Pods, workloads, nodes and events are tables — filter by namespace in the +header, and open logs or `describe` for any row. Doctor is the same ranked +findings the CLI prints, each one expanding to its remediation. + +![doctor](docs/screenshots/web-doctor.png) + +Three things worth knowing: + +- **It is read-only.** Everything lists, describes or tails. Nothing applies, + deletes or restarts. A browser tab that can change a production cluster + deserves more thought about who is holding it than a localhost token + provides, and none of that is needed to make a cluster legible. +- **It binds to loopback** unless you pass `--host` *and* `--allow-remote`. A + session token is the only authentication, and it is printed once, at + startup, fresh for every run. +- **It is the same binary.** No Electron, no WebView2, no npm: the assets are + embedded, so `serve` works on Windows, macOS, Linux and in a container with + nothing else installed. It costs about 3MB. + +Running it in Docker, where `--host 0.0.0.0` is the usual answer: + +```bash +docker run --rm -p 8787:8787 \ + -v "$PWD/targets.yaml:/targets.yaml:ro" \ + -v "$HOME/.ssh/id_ed25519:/key:ro" \ + k3helper serve -t /targets.yaml --host 0.0.0.0 --allow-remote +``` + +The image is `FROM scratch` and around 17MB. Mount credentials at run time and +never build them in: a key in a layer stays in the image even after a later +layer deletes it. + ### 7. TUI ```bash diff --git a/docs/screenshots/web-doctor.png b/docs/screenshots/web-doctor.png new file mode 100644 index 0000000..d89151d Binary files /dev/null and b/docs/screenshots/web-doctor.png differ diff --git a/docs/screenshots/web-overview.png b/docs/screenshots/web-overview.png new file mode 100644 index 0000000..49b33f9 Binary files /dev/null and b/docs/screenshots/web-overview.png differ diff --git a/docs/screenshots/web-pods.png b/docs/screenshots/web-pods.png new file mode 100644 index 0000000..9ad8ea6 Binary files /dev/null and b/docs/screenshots/web-pods.png differ diff --git a/internal/bundle/bundle.go b/internal/bundle/bundle.go new file mode 100644 index 0000000..b3b4230 --- /dev/null +++ b/internal/bundle/bundle.go @@ -0,0 +1,282 @@ +// Package bundle assembles everything a node needs to install k3s without +// reaching the internet. +// +// The normal install is `curl -sfL https://get.k3s.io | sh -`, which needs +// three things the node cannot have in an air-gapped network: the channel +// service that resolves "stable" to a version, the GitHub release that holds +// the k3s binary, and the container registry every pod pulls its image from. +// A bundle is those first two fetched somewhere with a connection; the third +// is covered by the airgap image archive, which k3s imports into containerd on +// first start. +package bundle + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "path/filepath" + "strings" + "time" +) + +// Files inside a bundle directory. The names are fixed so that `vm setup +// --bundle` can find them without being told, and so a bundle can be checked +// by eye. +const ( + BinaryName = "k3s" + ImagesName = "k3s-airgap-images.tar.zst" + InstallName = "install.sh" + ManifestName = "bundle.json" +) + +// Manifest records what a bundle holds. It is written into the directory so a +// bundle carried on a USB stick still knows its own version and architecture — +// installing an arm64 k3s on an amd64 node otherwise fails as "cannot execute +// binary file", which is a long way from the cause. +type Manifest struct { + Version string `json:"version"` + Arch string `json:"arch"` + Created time.Time `json:"created"` + BinarySHA string `json:"binary_sha256"` + ImagesSHA string `json:"images_sha256"` +} + +// Options control a fetch. +type Options struct { + Version string // e.g. v1.31.2+k3s1; required, because a channel lookup is the thing being avoided + Arch string // amd64 or arm64 + Dir string // where to write + // Progress receives one line per step. + Progress io.Writer + // HTTPClient is overridden by tests. + HTTPClient *http.Client +} + +func (o Options) client() *http.Client { + if o.HTTPClient != nil { + return o.HTTPClient + } + // Generous: these are large files over a link that may be slow, and a + // timeout that kills a 90%-complete 250MB download helps nobody. + return &http.Client{Timeout: 30 * time.Minute} +} + +// ReleaseBase is where the k3s release assets live. Overridden by tests. +var ReleaseBase = "https://github.com/k3s-io/k3s/releases/download" + +// InstallScriptURL is the installer itself. +var InstallScriptURL = "https://get.k3s.io" + +// binaryAsset is the release asset name for the k3s binary on an architecture. +// amd64 is the unsuffixed one, which is easy to get wrong in the other +// direction and produces a bundle that cannot run anywhere. +func binaryAsset(arch string) string { + if arch == "amd64" { + return "k3s" + } + return "k3s-" + arch +} + +func imagesAsset(arch string) string { + return fmt.Sprintf("k3s-airgap-images-%s.tar.zst", arch) +} + +func checksumAsset(arch string) string { + return fmt.Sprintf("sha256sum-%s.txt", arch) +} + +// Fetch downloads a bundle into o.Dir. +func Fetch(o Options) (*Manifest, error) { + if o.Version == "" { + return nil, fmt.Errorf("a version is required (e.g. v1.31.2+k3s1) — " + + "an air-gapped install cannot resolve a channel, which is the point of a bundle") + } + if o.Arch == "" { + o.Arch = "amd64" + } + switch o.Arch { + case "amd64", "arm64": + default: + return nil, fmt.Errorf("unsupported architecture %q: k3s publishes amd64 and arm64", o.Arch) + } + if o.Dir == "" { + return nil, fmt.Errorf("an output directory is required") + } + if err := os.MkdirAll(o.Dir, 0o755); err != nil { + return nil, fmt.Errorf("create %s: %w", o.Dir, err) + } + + base := fmt.Sprintf("%s/%s", ReleaseBase, urlVersion(o.Version)) + + // The checksum manifest first: downloading 250MB and only then finding out + // there is nothing to check it against wastes the slow part. + sums, err := fetchChecksums(o, base+"/"+checksumAsset(o.Arch)) + if err != nil { + return nil, err + } + + m := &Manifest{Version: o.Version, Arch: o.Arch, Created: time.Now().UTC()} + + binSum, err := download(o, base+"/"+binaryAsset(o.Arch), + filepath.Join(o.Dir, BinaryName), 0o755, sums[binaryAsset(o.Arch)]) + if err != nil { + return nil, err + } + m.BinarySHA = binSum + + imgSum, err := download(o, base+"/"+imagesAsset(o.Arch), + filepath.Join(o.Dir, ImagesName), 0o644, sums[imagesAsset(o.Arch)]) + if err != nil { + return nil, err + } + m.ImagesSHA = imgSum + + // The install script has no published checksum, so it is fetched without + // one rather than pretending otherwise. + if _, err := download(o, InstallScriptURL, + filepath.Join(o.Dir, InstallName), 0o755, ""); err != nil { + return nil, err + } + + data, err := json.MarshalIndent(m, "", " ") + if err != nil { + return nil, err + } + if err := os.WriteFile(filepath.Join(o.Dir, ManifestName), append(data, '\n'), 0o644); err != nil { + return nil, fmt.Errorf("write manifest: %w", err) + } + progressf(o, "✓ bundle ready in %s (%s, %s)", o.Dir, o.Version, o.Arch) + return m, nil +} + +// urlVersion escapes the "+" in a k3s version, which is a real character in +// the tag and means "space" in a URL path if left alone. +func urlVersion(v string) string { + return strings.ReplaceAll(v, "+", "%2B") +} + +// Load reads the manifest of an existing bundle and checks the files are +// actually there, so `vm setup --bundle` fails at the front door rather than +// half way through an install on the first node. +func Load(dir string) (*Manifest, error) { + data, err := os.ReadFile(filepath.Join(dir, ManifestName)) + if err != nil { + if os.IsNotExist(err) { + return nil, fmt.Errorf("%s is not a k3s bundle (no %s) — make one with `k3helper bundle k3s --version -o %s`", + dir, ManifestName, dir) + } + return nil, err + } + m := &Manifest{} + if err := json.Unmarshal(data, m); err != nil { + return nil, fmt.Errorf("parse %s: %w", ManifestName, err) + } + for _, f := range []string{BinaryName, ImagesName, InstallName} { + if _, err := os.Stat(filepath.Join(dir, f)); err != nil { + return nil, fmt.Errorf("bundle %s is incomplete: %s is missing", dir, f) + } + } + return m, nil +} + +func fetchChecksums(o Options, url string) (map[string]string, error) { + progressf(o, "fetching checksums...") + resp, err := o.client().Get(url) + if err != nil { + return nil, fmt.Errorf("fetch checksums: %w", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("fetch checksums: %s returned %s — is %q a real k3s release?", + url, resp.Status, o.Version) + } + body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return nil, err + } + sums := map[string]string{} + for _, line := range strings.Split(string(body), "\n") { + f := strings.Fields(line) + if len(f) == 2 { + sums[strings.TrimPrefix(f[1], "./")] = f[0] + } + } + return sums, nil +} + +// download fetches one asset, verifying it against want when there is one. +// +// The hash is computed while the bytes are written rather than by reading the +// file back: these are large, and a bundle built on a laptop should not need +// to read 250MB twice. +func download(o Options, url, dest string, mode os.FileMode, want string) (string, error) { + progressf(o, "downloading %s...", filepath.Base(dest)) + resp, err := o.client().Get(url) + if err != nil { + return "", fmt.Errorf("fetch %s: %w", url, err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("fetch %s: %s", url, resp.Status) + } + + tmp := dest + ".part" + f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, mode) + if err != nil { + return "", fmt.Errorf("create %s: %w", tmp, err) + } + h := sha256.New() + n, err := io.Copy(io.MultiWriter(f, h), resp.Body) + closeErr := f.Close() + if err != nil { + os.Remove(tmp) + return "", fmt.Errorf("download %s: %w", url, err) + } + if closeErr != nil { + os.Remove(tmp) + return "", closeErr + } + + got := hex.EncodeToString(h.Sum(nil)) + if want != "" && !strings.EqualFold(got, want) { + os.Remove(tmp) + return "", fmt.Errorf("checksum mismatch for %s: expected %s, got %s", filepath.Base(dest), want, got) + } + if err := os.Rename(tmp, dest); err != nil { + return "", err + } + if err := os.Chmod(dest, mode); err != nil { + return "", err + } + progressf(o, " %s %s %s", filepath.Base(dest), humanSize(n), shortSum(got)) + return got, nil +} + +func progressf(o Options, format string, args ...any) { + if o.Progress == nil { + return + } + fmt.Fprintf(o.Progress, format+"\n", args...) +} + +func humanSize(n int64) string { + switch { + case n >= 1<<30: + return fmt.Sprintf("%.1fGB", float64(n)/float64(1<<30)) + case n >= 1<<20: + return fmt.Sprintf("%.0fMB", float64(n)/float64(1<<20)) + default: + return fmt.Sprintf("%dKB", n/1024) + } +} + +func shortSum(s string) string { + if len(s) > 12 { + return s[:12] + } + return s +} diff --git a/internal/cli/bundle_cmd.go b/internal/cli/bundle_cmd.go new file mode 100644 index 0000000..bef8fc6 --- /dev/null +++ b/internal/cli/bundle_cmd.go @@ -0,0 +1,68 @@ +package cli + +import ( + "fmt" + "runtime" + + "github.com/solutionforest/k3helper/internal/bundle" + "github.com/spf13/cobra" +) + +func newBundleCmd() *cobra.Command { + cmd := &cobra.Command{ + Use: "bundle", + Short: "Assemble what an air-gapped node needs to install Kubernetes", + Long: `Build an offline install bundle on a machine that has internet, so nodes +that do not have any can still be built. + +The usual install reaches out three times: to update.k3s.io to turn "stable" +into a version, to the GitHub release for the k3s binary, and to a registry for +every image a pod pulls. A bundle covers all three — the binary, the airgap +image archive k3s imports into containerd on first start, and the installer +itself. + + k3helper bundle k3s --version v1.31.2+k3s1 -o ./k3s-bundle + k3helper vm setup -t targets.yaml --bundle ./k3s-bundle`, + } + cmd.AddCommand(newBundleK3sCmd()) + return cmd +} + +func newBundleK3sCmd() *cobra.Command { + var ( + version string + arch string + out string + ) + cmd := &cobra.Command{ + Use: "k3s", + Short: "Download a k3s release and its airgap images into a directory", + RunE: func(cmd *cobra.Command, args []string) error { + if version == "" { + return fmt.Errorf("--version is required (e.g. v1.31.2+k3s1)\n\n" + + "A bundle cannot resolve a channel — not needing update.k3s.io is the point of it.\n" + + "Releases are listed at https://github.com/k3s-io/k3s/releases") + } + m, err := bundle.Fetch(bundle.Options{ + Version: version, + Arch: arch, + Dir: out, + Progress: cmd.OutOrStdout(), + }) + if err != nil { + return err + } + fmt.Fprintf(cmd.OutOrStdout(), + "\nNext, from a machine that can reach the nodes:\n"+ + " k3helper vm setup -t targets.yaml --bundle %s\n", out) + _ = m + return nil + }, + } + cmd.Flags().StringVar(&version, "version", "", "k3s release to fetch, e.g. v1.31.2+k3s1") + // The nodes' architecture, which is usually but not always this machine's: + // bundles get built on a laptop for servers that are not one. + cmd.Flags().StringVar(&arch, "arch", runtime.GOARCH, "architecture of the target nodes (amd64 or arm64)") + cmd.Flags().StringVarP(&out, "out", "o", "k3s-bundle", "directory to write the bundle into") + return cmd +} diff --git a/internal/cli/root.go b/internal/cli/root.go index 3f766de..09b8620 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -45,9 +45,11 @@ func NewRootCmd() *cobra.Command { check run cluster/node/k3s health checks doctor troubleshoot: find issues + remediation (--watch to keep looking) registry configure the private registries the cluster pulls from + bundle build an offline install bundle for air-gapped nodes ctx list clusters defined in the targets file init create a targets.yaml describing your nodes - tui interactive dashboard + resource browser`, + tui interactive dashboard + resource browser + serve the same, in a web browser (read-only)`, SilenceUsage: true, } // --context selects a cluster from a multi-cluster targets file. It is @@ -78,8 +80,10 @@ func NewRootCmd() *cobra.Command { root.AddCommand(newVMCmd()) root.AddCommand(newDoctorCmd()) root.AddCommand(newRegistryCmd()) + root.AddCommand(newBundleCmd()) root.AddCommand(newDeployCmd()) root.AddCommand(newTUICmd()) + root.AddCommand(newServeCmd()) return root } diff --git a/internal/cli/serve_cmd.go b/internal/cli/serve_cmd.go new file mode 100644 index 0000000..f9408c0 --- /dev/null +++ b/internal/cli/serve_cmd.go @@ -0,0 +1,196 @@ +package cli + +import ( + "crypto/rand" + "encoding/hex" + "fmt" + "net" + "net/http" + "os/exec" + "runtime" + "strings" + "time" + + "github.com/solutionforest/k3helper/internal/web" + "github.com/spf13/cobra" +) + +func newServeCmd() *cobra.Command { + var ( + targetsPath string + host string + port int + token string + noBrowser bool + allowRemote bool + ) + cmd := &cobra.Command{ + Use: "serve", + Short: "Browse the cluster from a web browser", + Long: `Serve a read-only web interface for the cluster. + +The same binary, the same checks and the same diagnosis as the CLI and the TUI +— shown in a browser instead of a terminal. Nothing here changes the cluster: +it lists, describes and tails, and that is all. + + k3helper serve # http://127.0.0.1:8787 + k3helper serve --port 9000 + k3helper serve --host 0.0.0.0 --allow-remote + +The address defaults to loopback on purpose. A session token is the only +authentication there is, so the server is reachable from this machine and no +other until you say otherwise.`, + RunE: func(cmd *cobra.Command, args []string) error { + targets, err := loadTargets(targetsPath) + if err != nil { + return err + } + + bind, err := resolveBind(host, allowRemote) + if err != nil { + return err + } + if token == "" { + if token, err = newToken(); err != nil { + return err + } + } + + // The port is claimed before anything is printed, so the URL on + // screen is one that works. + ln, chosen, err := listen(bind, port, cmd.Flags().Changed("port")) + if err != nil { + return err + } + defer ln.Close() + + srv := &web.Server{Targets: targets, Token: token, Version: version} + url := fmt.Sprintf("http://%s/?token=%s", hostPort(bind, chosen), token) + + out := cmd.OutOrStdout() + fmt.Fprintf(out, "k3helper %s — serving %q (%s)\n\n", version, targets.Cluster, targets.Mode()) + fmt.Fprintf(out, " %s\n\n", url) + if bind != "127.0.0.1" && bind != "::1" { + fmt.Fprintf(out, " Reachable from the network on %s. The token above is the only thing\n"+ + " stopping anyone who can reach this port from reading your cluster.\n\n", bind) + } + fmt.Fprintln(out, " Read-only. Ctrl-C to stop.") + + if !noBrowser { + openBrowser(url) + } + + httpSrv := &http.Server{ + Handler: srv.Handler(), + // A diagnosis walks every node over SSH and can take a while + // on a slow link; the live tests saw ten seconds against + // Singapore. The read side is short because requests are tiny. + ReadHeaderTimeout: 10 * time.Second, + WriteTimeout: 2 * time.Minute, + } + if err := httpSrv.Serve(ln); err != nil && err != http.ErrServerClosed { + return err + } + return nil + }, + } + cmd.Flags().StringVarP(&targetsPath, "targets", "t", "targets.yaml", "path to targets YAML") + cmd.Flags().StringVar(&host, "host", "127.0.0.1", "address to bind (use 0.0.0.0 with --allow-remote to serve the network)") + cmd.Flags().IntVarP(&port, "port", "p", 8787, "port to listen on") + cmd.Flags().StringVar(&token, "token", "", "session token (default: a new random one each launch)") + cmd.Flags().BoolVar(&noBrowser, "no-browser", false, "do not open a browser") + cmd.Flags().BoolVar(&allowRemote, "allow-remote", false, + "permit binding somewhere other than loopback, making the cluster readable by anyone who can reach the port") + return cmd +} + +// resolveBind refuses a non-loopback address unless it was asked for twice. +// +// Everything this serves is a read of a production cluster, and a session +// token in a URL is not much of a fence. Binding to 0.0.0.0 on a shared +// network by accident — because a container needs it, or because a flag was +// copied from somewhere — should not be one flag away. +func resolveBind(host string, allowRemote bool) (string, error) { + h := strings.TrimSpace(host) + if h == "" { + return "127.0.0.1", nil + } + if h == "localhost" { + return "127.0.0.1", nil + } + if ip := net.ParseIP(h); ip == nil { + return "", fmt.Errorf("--host %q is not an IP address; use 127.0.0.1, 0.0.0.0, or an address of this machine", host) + } else if ip.IsLoopback() { + return h, nil + } + if !allowRemote { + return "", fmt.Errorf("--host %s would let anyone who can reach that address read this cluster, "+ + "and a session token is the only thing in the way.\n\n"+ + "Add --allow-remote if that is what you want. In a container, that is the usual answer:\n"+ + " k3helper serve --host 0.0.0.0 --allow-remote", host) + } + return h, nil +} + +// listen claims the port, stepping to the next free one when the default was +// left alone. +// +// An explicit --port is honoured exactly: someone who names a port has a +// reason, and quietly using a different one would break whatever that reason +// was. Without one, walking up from the default beats refusing to start +// because a previous run is still shutting down. +func listen(bind string, port int, explicit bool) (net.Listener, int, error) { + tries := 1 + if !explicit { + tries = 10 + } + var lastErr error + for i := 0; i < tries; i++ { + p := port + i + ln, err := net.Listen("tcp", net.JoinHostPort(bind, fmt.Sprint(p))) + if err == nil { + return ln, p, nil + } + lastErr = err + } + if explicit { + return nil, 0, fmt.Errorf("port %d on %s is not available: %w\n\n"+ + "Choose another with --port, or leave it off and one will be found", port, bind, lastErr) + } + return nil, 0, fmt.Errorf("no free port between %d and %d on %s: %w", port, port+tries-1, bind, lastErr) +} + +// hostPort renders the address for a URL: 0.0.0.0 is a bind, not somewhere a +// browser can go. +func hostPort(bind string, port int) string { + h := bind + if h == "0.0.0.0" || h == "::" { + h = "127.0.0.1" + } + return net.JoinHostPort(h, fmt.Sprint(port)) +} + +// newToken makes the session key. +func newToken() (string, error) { + b := make([]byte, 24) + if _, err := rand.Read(b); err != nil { + return "", fmt.Errorf("generate a session token: %w", err) + } + return hex.EncodeToString(b), nil +} + +// openBrowser is best-effort: failing to open one is not a reason to fail to +// serve, and the URL is on screen either way. +func openBrowser(url string) { + var cmd string + var args []string + switch runtime.GOOS { + case "darwin": + cmd = "open" + case "windows": + cmd, args = "rundll32", []string{"url.dll,FileProtocolHandler"} + default: + cmd = "xdg-open" + } + exec.Command(cmd, append(args, url)...).Start() +} diff --git a/internal/cli/serve_cmd_test.go b/internal/cli/serve_cmd_test.go new file mode 100644 index 0000000..06c1f1b --- /dev/null +++ b/internal/cli/serve_cmd_test.go @@ -0,0 +1,103 @@ +package cli + +import ( + "net" + "strings" + "testing" +) + +// Everything this serves is a read of a production cluster, and a session +// token in a URL is not much of a fence. Binding to the network by accident — +// a flag copied from somewhere, a container that needs it — should not be one +// flag away. +func TestResolveBindRefusesTheNetworkWithoutSayingSoTwice(t *testing.T) { + if _, err := resolveBind("0.0.0.0", false); err == nil { + t.Fatal("0.0.0.0 was accepted without --allow-remote") + } else if !strings.Contains(err.Error(), "--allow-remote") { + t.Errorf("the refusal does not say how to proceed: %v", err) + } + if got, err := resolveBind("0.0.0.0", true); err != nil || got != "0.0.0.0" { + t.Errorf("--allow-remote did not permit it: %q %v", got, err) + } +} + +func TestResolveBindAllowsLoopbackFreely(t *testing.T) { + for _, in := range []string{"", "127.0.0.1", "localhost", "::1"} { + if _, err := resolveBind(in, false); err != nil { + t.Errorf("resolveBind(%q) refused loopback: %v", in, err) + } + } +} + +func TestResolveBindRejectsSomethingThatIsNotAnAddress(t *testing.T) { + if _, err := resolveBind("example.com", true); err == nil { + t.Error("a hostname was accepted as a bind address") + } +} + +// An explicit --port is honoured exactly: someone who names one has a reason, +// and quietly using a different port would break whatever that reason was. +func TestExplicitPortIsNotMovedWhenBusy(t *testing.T) { + busy, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer busy.Close() + port := busy.Addr().(*net.TCPAddr).Port + + if _, _, err := listen("127.0.0.1", port, true); err == nil { + t.Fatal("an explicit port that was busy was silently moved") + } else if !strings.Contains(err.Error(), "--port") { + t.Errorf("the error does not suggest what to do: %v", err) + } +} + +// Without one, walking up from the default beats refusing to start because a +// previous run is still shutting down. +func TestDefaultPortStepsPastOneInUse(t *testing.T) { + busy, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer busy.Close() + port := busy.Addr().(*net.TCPAddr).Port + + ln, chosen, err := listen("127.0.0.1", port, false) + if err != nil { + t.Fatalf("no port was found: %v", err) + } + defer ln.Close() + if chosen == port { + t.Error("the busy port was reported as claimed") + } + if chosen < port || chosen > port+9 { + t.Errorf("chose %d, want something just above %d", chosen, port) + } +} + +// 0.0.0.0 is a bind, not somewhere a browser can go. +func TestHostPortRendersSomewhereReachable(t *testing.T) { + for _, bind := range []string{"0.0.0.0", "::"} { + if got := hostPort(bind, 8787); !strings.Contains(got, "127.0.0.1") { + t.Errorf("hostPort(%q) = %q, want a loopback address a browser can open", bind, got) + } + } + if got := hostPort("192.168.1.5", 8787); got != "192.168.1.5:8787" { + t.Errorf("a real address was rewritten: %q", got) + } +} + +// A new one per launch, and long enough to be worth having. +func TestTokensAreFreshAndLong(t *testing.T) { + a, err := newToken() + if err != nil { + t.Fatal(err) + } + b, _ := newToken() + if a == b { + t.Error("two launches produced the same token") + } + if len(a) < 32 { + t.Errorf("token is %d characters, which is not much of a secret", len(a)) + } +} diff --git a/internal/cli/vm_cmd.go b/internal/cli/vm_cmd.go index b765658..9596691 100644 --- a/internal/cli/vm_cmd.go +++ b/internal/cli/vm_cmd.go @@ -5,6 +5,7 @@ import ( "strings" "github.com/solutionforest/k3helper/internal/config" + "github.com/solutionforest/k3helper/internal/proxy" "github.com/solutionforest/k3helper/internal/registry" "github.com/solutionforest/k3helper/internal/ssh" "github.com/solutionforest/k3helper/internal/transport" @@ -75,6 +76,13 @@ func newVMSetupCmd() *cobra.Command { extraArgs string distro string k8sVersion string + k3sVersion string + bundleDir string + joinAddress string + aptMirror string + k8sAptRepo string + viaProxy bool + proxyAllow []string cni string noConntrack bool regFlags registryFlags @@ -83,6 +91,25 @@ func newVMSetupCmd() *cobra.Command { Use: "setup", Short: "Install k3s server + agents on all nodes in targets file", RunE: func(cmd *cobra.Command, args []string) error { + // Flag combinations are settled before anything is dialled. A + // contradiction is not worth an SSH round trip to every node to + // discover, and one of these decides whether the nodes are given + // a route to the internet at all. + if err := checkSetupFlags(distro, bundleDir, k3sVersion, viaProxy, proxyAllow); err != nil { + return err + } + if viaProxy { + // Said out loud, every time. Lending an isolated machine a + // route out is the operator's decision to make knowingly, and + // in some environments it is not theirs to make at all. + fmt.Fprintf(cmd.OutOrStdout(), + "--via-proxy: these nodes will reach the internet through this machine for the "+ + "length of the install, and only through it.\n"+ + " allowed: %s\n"+ + " the tunnel and its configuration are removed when the install finishes.\n\n", + strings.Join(proxyAllowSummary(proxyAllow), ", ")) + } + targets, err := loadTargets(targetsPath) if err != nil { return err @@ -157,6 +184,10 @@ func newVMSetupCmd() *cobra.Command { CNI: cni, InitExtraArgs: extraArgs, SkipConntrackTuning: noConntrack, + JoinAddress: joinAddress, + Mirror: vm.AptMirror{URL: aptMirror, K8sRepo: k8sAptRepo}, + ViaProxy: viaProxy, + ProxyAllow: proxyAllow, Progress: cmd.OutOrStdout(), }); err != nil { return err @@ -176,6 +207,12 @@ func newVMSetupCmd() *cobra.Command { err = vm.Setup(servers, agents, vm.Options{ Channel: channel, + Version: k3sVersion, + BundleDir: bundleDir, + JoinAddress: joinAddress, + Mirror: vm.AptMirror{URL: aptMirror, K8sRepo: k8sAptRepo}, + ViaProxy: viaProxy, + ProxyAllow: proxyAllow, Token: token, ServerExtraArgs: extraArgs, AgentExtraArgs: agentExtraArgs, @@ -201,6 +238,22 @@ func newVMSetupCmd() *cobra.Command { cmd.Flags().StringVar(&extraArgs, "server-extra-args", "", "extra args for the server install (k3s install script, or `kubeadm init`)") cmd.Flags().StringVar(&distro, "distro", "k3s", "distribution to install: k3s or kubeadm") cmd.Flags().StringVar(&k8sVersion, "k8s-version", "", "Kubernetes minor series for kubeadm, e.g. v1.31") + cmd.Flags().StringVar(&k3sVersion, "k3s-version", "", + "pin an exact k3s release, e.g. v1.31.2+k3s1 (skips the update.k3s.io channel lookup)") + cmd.Flags().StringVar(&bundleDir, "bundle", "", + "install from an offline bundle (see `k3helper bundle k3s`); the nodes need no internet") + cmd.Flags().StringVar(&joinAddress, "join-address", "", + "address the other nodes dial to reach the first server (default: its host from the targets file)") + cmd.Flags().StringVar(&aptMirror, "apt-mirror", "", + "point the nodes' package manager at this archive instead of the distribution's, "+ + "e.g. https://nexus.corp/repository/ubuntu") + cmd.Flags().StringVar(&k8sAptRepo, "k8s-apt-repo", "", + "mirror of the Kubernetes package repository (default: pkgs.k8s.io); kubeadm only") + cmd.Flags().BoolVar(&viaProxy, "via-proxy", false, + "lend the nodes this machine's internet connection for the length of the install, "+ + "over the SSH connection already open to them") + cmd.Flags().StringArrayVar(&proxyAllow, "proxy-allow", nil, + "extra host allowed through --via-proxy; repeat for more, or \"*\" for anything") cmd.Flags().StringVar(&cni, "cni", "flannel", "CNI for kubeadm: flannel or calico") cmd.Flags().BoolVar(&noConntrack, "no-conntrack-tuning", false, "stop kube-proxy managing nf_conntrack_max; needed where that sysctl is read-only or capped (nested VMs, containers)") @@ -214,3 +267,50 @@ var agentExtraArgs string func toSSHNode(n config.Node) ssh.Node { return n.SSH() } + +// proxyAllowSummary describes what --via-proxy will permit, for the notice +// printed before it is opened. +func proxyAllowSummary(extra []string) []string { + for _, a := range extra { + if a == "*" { + return []string{"anything (--proxy-allow \"*\")"} + } + } + out := []string{fmt.Sprintf("%d default hosts (distribution mirrors, pkgs.k8s.io, registry.k8s.io, docker.io)", + len(proxy.DefaultAllow))} + if len(extra) > 0 { + out = append(out, strings.Join(extra, ", ")) + } + return out +} + +// checkSetupFlags rejects combinations that cannot mean what they say. +// +// Silently ignoring one of these is how an operator asks for an offline +// install, gets an online one, and finds out on an air-gapped node from a TLS +// error at the worst possible moment. +func checkSetupFlags(distro, bundleDir, k3sVersion string, viaProxy bool, proxyAllow []string) error { + if distro == "kubeadm" { + if bundleDir != "" { + return fmt.Errorf("--bundle builds a k3s bundle and only the k3s installer can use it; " + + "an offline kubeadm install needs distribution packages and registry.k8s.io images. " + + "Use --apt-mirror and --k8s-apt-repo to point at your own mirrors, --via-proxy to " + + "lend the nodes this machine's connection, or --distro k3s, which installs fully offline") + } + if k3sVersion != "" { + return fmt.Errorf("--k3s-version applies to --distro k3s; for kubeadm use --k8s-version") + } + } + if bundleDir != "" && k3sVersion != "" { + return fmt.Errorf("--bundle and --k3s-version contradict each other: " + + "a bundle already contains one exact k3s release, recorded in its bundle.json") + } + if viaProxy && bundleDir != "" { + return fmt.Errorf("--via-proxy and --bundle are two answers to the same problem: " + + "a bundle installs with no network at all, so lending the nodes one does nothing") + } + if len(proxyAllow) > 0 && !viaProxy { + return fmt.Errorf("--proxy-allow only means something with --via-proxy") + } + return nil +} diff --git a/internal/proxy/deadline_test.go b/internal/proxy/deadline_test.go new file mode 100644 index 0000000..666da9a --- /dev/null +++ b/internal/proxy/deadline_test.go @@ -0,0 +1,83 @@ +package proxy + +import ( + "bufio" + "errors" + "io" + "net" + "net/http" + "testing" + "time" +) + +// nodeConn behaves like the connections x/crypto/ssh hands back from a +// reverse tunnel: a net.Conn that refuses deadlines. +type nodeConn struct{ net.Conn } + +func (nodeConn) SetDeadline(time.Time) error { + return errors.New("ssh: tcpChan: deadline not supported") +} +func (nodeConn) SetReadDeadline(time.Time) error { + return errors.New("ssh: tcpChan: deadline not supported") +} +func (nodeConn) SetWriteDeadline(time.Time) error { + return errors.New("ssh: tcpChan: deadline not supported") +} + +type nodeListener struct{ net.Listener } + +func (l nodeListener) Accept() (net.Conn, error) { + c, err := l.Listener.Accept() + if err != nil { + return nil, err + } + return nodeConn{c}, nil +} + +// The connections this proxy serves arrive over an SSH reverse tunnel, and +// x/crypto/ssh channels refuse to set deadlines. Served through an +// http.Server, CONNECT hangs on that: Hijack calls abortPendingRead, which +// interrupts its background read by setting a deadline in the past, and on a +// connection that cannot do that it waits forever and never replies. +// +// Found against real air-gapped nodes, where every https fetch through the +// tunnel failed with "Proxy CONNECT aborted due to timeout" while plain http +// went through — because only https uses CONNECT. +func TestConnectOverADeadlinelessConn(t *testing.T) { + echo, _ := net.Listen("tcp", "127.0.0.1:0") + defer echo.Close() + go func() { + c, err := echo.Accept() + if err != nil { + return + } + defer c.Close() + io.Copy(c, c) + }() + + raw, _ := net.Listen("tcp", "127.0.0.1:0") + s := &Server{Allow: []string{"127.0.0.1"}} + go s.Serve(nodeListener{raw}) + defer func() { s.Close(); raw.Close() }() + + c, err := net.Dial("tcp", raw.Addr().String()) + if err != nil { + t.Fatal(err) + } + defer c.Close() + c.SetDeadline(time.Now().Add(10 * time.Second)) + io.WriteString(c, "CONNECT "+echo.Addr().String()+" HTTP/1.1\r\nHost: "+echo.Addr().String()+"\r\n\r\nping") + br := bufio.NewReader(c) + r, err := http.ReadResponse(br, nil) + if err != nil { + t.Fatalf("no response to CONNECT over a deadlineless conn: %v", err) + } + if r.StatusCode != 200 { + t.Fatalf("status %d", r.StatusCode) + } + buf := make([]byte, 4) + if _, err := io.ReadFull(br, buf); err != nil { + t.Fatalf("tunnel stalled: %v", err) + } + t.Logf("got %q", buf) +} diff --git a/internal/proxy/proxy.go b/internal/proxy/proxy.go new file mode 100644 index 0000000..5fb4f36 --- /dev/null +++ b/internal/proxy/proxy.go @@ -0,0 +1,303 @@ +// Package proxy is a small HTTP forward proxy, used to lend a node the +// operator's own internet connection for the length of an install. +// +// An air-gapped node cannot reach a distribution mirror or a container +// registry, and for kubeadm that is fatal: unlike k3s it needs apt packages +// and images from registry.k8s.io, neither of which fits in a bundle. The +// machine running k3helper usually *can* reach those, and it already holds an +// SSH connection to every node — so the connection can carry the traffic +// backwards. +// +// This deliberately does less than a general proxy: +// +// - Only CONNECT and absolute-form requests, which is all apt and containerd +// produce. +// - A host allowlist, checked before anything is dialled. Lending a machine +// a route to the internet is not the same as lending it a route to +// everything, and an air-gapped network is air-gapped for a reason. +// - Nothing is cached, logged to disk, or modified in flight. +package proxy + +import ( + "bufio" + "fmt" + "io" + "net" + "net/http" + "strings" + "sync" + "time" +) + +// DefaultAllow is what a Kubernetes install actually needs to reach. +// +// Matching is on the host, by exact name or by dot-suffix, so "pkg.dev" admits +// "us-central1-docker.pkg.dev" and nothing that merely ends in those letters. +var DefaultAllow = []string{ + // distribution packages, including the mirrors cloud images actually ship + // pointing at — a stock Ubuntu image on DigitalOcean uses the provider's + // own, which the first live run of this proxy refused. + "archive.ubuntu.com", "security.ubuntu.com", "ports.ubuntu.com", + "deb.debian.org", "security.debian.org", + "mirrors.digitalocean.com", "repos-droplet.digitalocean.com", + // Ubuntu Pro's ESM endpoint is configured on stock images whether or not + // the machine is subscribed; refusing it makes apt retry and fail noisily + // for something the install does not even need. + "esm.ubuntu.com", "motd.ubuntu.com", "changelogs.ubuntu.com", + "mirrors.linode.com", "mirror.hetzner.com", "azure.archive.ubuntu.com", + "clouds.archive.ubuntu.com", "ec2.archive.ubuntu.com", + "europe-west1.gce.archive.ubuntu.com", "gce.archive.ubuntu.com", + // kubernetes packages and images + // pkgs.k8s.io redirects the actual packages to a CDN on another name, and + // registry.k8s.io redirects images to a cloud provider's registry. An + // allowlist with only the names an operator would think to write refuses + // the download that follows the redirect. + // dl.k8s.io serves the version markers kubeadm reads during init. + "pkgs.k8s.io", "packages.k8s.io", "registry.k8s.io", "dl.k8s.io", + "pkg.dev", "storage.googleapis.com", "amazonaws.com", "cloudfront.net", + // k3s + "get.k3s.io", "update.k3s.io", "github.com", "githubusercontent.com", + // docker hub, for CNI and workload images + "docker.io", "docker.com", "cloudflare.docker.com", + // quay and ghcr, where the CNIs live. Flannel's images are on ghcr.io and + // its blobs on pkg-containers.githubusercontent.com — an allowlist built + // from "what Kubernetes needs" misses both, and the cluster then installs + // perfectly and sits NotReady because its network plugin cannot start. + "quay.io", "ghcr.io", "pkg-containers.githubusercontent.com", +} + +// Server forwards requests from a node to the internet. +type Server struct { + // Allow lists permitted hosts. Empty means DefaultAllow; the single entry + // "*" means anything, which the caller has to ask for explicitly. + Allow []string + // Dial is overridden by tests. + Dial func(network, addr string) (net.Conn, error) + // OnRefuse is called with the host of a rejected request, so the caller + // can tell an operator why an install could not fetch something. + OnRefuse func(host string) + + mu sync.Mutex + closed bool + conns map[net.Conn]struct{} +} + +func (s *Server) allowed(host string) bool { + h := bareHost(host) + list := s.Allow + if len(list) == 0 { + list = DefaultAllow + } + for _, a := range list { + if a == "*" { + return true + } + // The entry is stripped too. A CONNECT names host:port, so an operator + // copying one out of a refusal would otherwise add "host:443" to the + // allowlist and find it still refused. + a = bareHost(a) + if h == a || strings.HasSuffix(h, "."+a) { + return true + } + } + return false +} + +// bareHost lowercases a host and drops any port. +func bareHost(host string) string { + h := strings.ToLower(strings.TrimSpace(host)) + if i := strings.LastIndex(h, ":"); i > 0 && !strings.Contains(h[i:], "]") { + h = h[:i] + } + return h +} + +func (s *Server) dial(addr string) (net.Conn, error) { + if s.Dial != nil { + return s.Dial("tcp", addr) + } + return net.DialTimeout("tcp", addr, 30*time.Second) +} + +// Serve reads the proxy protocol off each connection itself rather than +// handing the listener to an http.Server. +// +// That is not a preference, it is a requirement. These connections arrive over +// an SSH reverse tunnel, and x/crypto/ssh's channels do not support deadlines. +// http.Server's CONNECT path goes through Hijack, which calls +// abortPendingRead, which interrupts its background read by setting a deadline +// in the past — on a connection that cannot do that, it waits for a read that +// will never be interrupted and the handler never replies. The client sees +// "Proxy CONNECT aborted due to timeout" and the install stops, which is +// exactly what happened the first time this ran against real nodes. +// +// Reading the request directly is also simply less machinery: a forward proxy +// needs a request line, a host, and two io.Copys. +func (s *Server) Serve(l net.Listener) error { + for { + c, err := l.Accept() + if err != nil { + return err + } + if !s.track(c) { + c.Close() + return net.ErrClosed + } + go func() { + defer s.done(c) + s.handle(c) + }() + } +} + +func (s *Server) track(c net.Conn) bool { + s.mu.Lock() + defer s.mu.Unlock() + if s.closed { + return false + } + if s.conns == nil { + s.conns = map[net.Conn]struct{}{} + } + s.conns[c] = struct{}{} + return true +} + +func (s *Server) done(c net.Conn) { + s.mu.Lock() + delete(s.conns, c) + s.mu.Unlock() + c.Close() +} + +// Close stops the proxy and drops every connection it is carrying. Safe to +// call before Serve, after it, or twice. +func (s *Server) Close() error { + s.mu.Lock() + s.closed = true + conns := make([]net.Conn, 0, len(s.conns)) + for c := range s.conns { + conns = append(conns, c) + } + s.conns = nil + s.mu.Unlock() + for _, c := range conns { + c.Close() + } + return nil +} + +// handle serves one connection, which may carry several requests: apt reuses +// a connection for every index and package it fetches. +func (s *Server) handle(c net.Conn) { + br := bufio.NewReader(c) + for { + req, err := http.ReadRequest(br) + if err != nil { + return + } + if req.Method == http.MethodConnect { + // CONNECT takes the connection over for good; nothing follows it. + s.connect(c, br, req) + return + } + if !s.forward(c, req) { + return + } + } +} + +// connect splices the connection to the destination. +// +// The reader carries anything already buffered — a client that pipelines, as +// curl does when it sends its TLS ClientHello straight after CONNECT without +// waiting for the 200, has those bytes read before the reply is even written. +// Splicing the bare connection instead of the reader drops them, and the +// handshake then waits for a hello that was received and discarded. +func (s *Server) connect(c net.Conn, br *bufio.Reader, req *http.Request) { + host := req.Host + if host == "" { + host = req.URL.Host + } + if !s.allowed(host) { + s.refuseConn(c, host) + return + } + if _, _, err := net.SplitHostPort(host); err != nil { + host = net.JoinHostPort(host, "443") + } + upstream, err := s.dial(host) + if err != nil { + fmt.Fprintf(c, "HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\n\r\n") + return + } + defer upstream.Close() + + if _, err := io.WriteString(c, "HTTP/1.1 200 Connection established\r\n\r\n"); err != nil { + return + } + + var wg sync.WaitGroup + wg.Add(2) + go func() { defer wg.Done(); io.Copy(upstream, br); closeWrite(upstream) }() + go func() { defer wg.Done(); io.Copy(c, upstream); closeWrite(c) }() + wg.Wait() +} + +// forward handles the plain-http absolute-form request apt uses for a mirror +// that is not behind TLS. It reports whether the connection can carry another. +func (s *Server) forward(c net.Conn, req *http.Request) bool { + if !req.URL.IsAbs() { + writeError(c, http.StatusBadRequest, + "proxy: this is a forward proxy; requests must name a full URL") + return false + } + if !s.allowed(req.URL.Host) { + s.refuseConn(c, req.URL.Host) + return false + } + out := req.Clone(req.Context()) + out.RequestURI = "" + // Hop-by-hop headers do not belong on the outbound request. + for _, h := range []string{"Proxy-Connection", "Proxy-Authenticate", "Proxy-Authorization"} { + out.Header.Del(h) + } + resp, err := http.DefaultTransport.RoundTrip(out) + if err != nil { + writeError(c, http.StatusBadGateway, "proxy: "+err.Error()) + return false + } + defer resp.Body.Close() + if err := resp.Write(c); err != nil { + return false + } + return !resp.Close && !req.Close +} + +func writeError(c net.Conn, status int, body string) { + fmt.Fprintf(c, "HTTP/1.1 %d %s\r\nContent-Type: text/plain\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s", + status, http.StatusText(status), len(body), body) +} + +func (s *Server) refuseConn(c net.Conn, host string) { + // Reported without the port, because that is what goes in the allowlist. + host = bareHost(host) + if s.OnRefuse != nil { + s.OnRefuse(host) + } + writeError(c, http.StatusForbidden, fmt.Sprintf( + "proxy: %s is not on the allowlist. k3helper lends a node its own connection for "+ + "the length of an install, not general internet access; pass --proxy-allow %s to permit it.", + host, host)) +} + +// closeWrite half-closes where the connection supports it, so the far end sees +// a clean end of stream rather than a reset. +func closeWrite(c net.Conn) { + type cw interface{ CloseWrite() error } + if v, ok := c.(cw); ok { + v.CloseWrite() + return + } + c.Close() +} diff --git a/internal/proxy/proxy_test.go b/internal/proxy/proxy_test.go new file mode 100644 index 0000000..0821e38 --- /dev/null +++ b/internal/proxy/proxy_test.go @@ -0,0 +1,325 @@ +package proxy + +import ( + "bufio" + "io" + "net" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +// serve starts a proxy on a local listener and returns its address. +func serve(t *testing.T, s *Server) string { + t.Helper() + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + go s.Serve(l) + t.Cleanup(func() { s.Close(); l.Close() }) + return l.Addr().String() +} + +func TestAllowlistMatching(t *testing.T) { + s := &Server{Allow: []string{"pkgs.k8s.io", "pkg.dev"}} + tests := []struct { + host string + want bool + }{ + {"pkgs.k8s.io", true}, + {"pkgs.k8s.io:443", true}, + {"us-central1-docker.pkg.dev", true}, + {"PKGS.K8S.IO", true}, + {"evil.com", false}, + // A suffix match must be on a dot boundary, or "notpkg.dev" would pass + // for "pkg.dev" and an allowlist would mean very little. + {"notpkg.dev", false}, + {"pkgs.k8s.io.evil.com", false}, + } + for _, tc := range tests { + if got := s.allowed(tc.host); got != tc.want { + t.Errorf("allowed(%q) = %v, want %v", tc.host, got, tc.want) + } + } +} + +func TestAllowStarPermitsAnything(t *testing.T) { + s := &Server{Allow: []string{"*"}} + if !s.allowed("anything.example") { + t.Error(`Allow "*" did not permit an arbitrary host`) + } +} + +func TestDefaultAllowCoversWhatAnInstallNeeds(t *testing.T) { + s := &Server{} + for _, host := range []string{ + "archive.ubuntu.com", "security.ubuntu.com", + "pkgs.k8s.io", "registry.k8s.io", + "us-central1-docker.pkg.dev", "storage.googleapis.com", + "get.k3s.io", "github.com", "objects.githubusercontent.com", + "registry-1.docker.io", "production.cloudflare.docker.com", + } { + if !s.allowed(host) { + t.Errorf("an install needs %s and the default allowlist refuses it", host) + } + } + if s.allowed("example.com") { + t.Error("the default allowlist is not a list at all") + } +} + +// The refusal has to say what to do about it, or an install fails against a +// mirror nobody thought to name and the operator has nothing to go on. +func TestRefusalExplainsItself(t *testing.T) { + var refused string + s := &Server{Allow: []string{"pkgs.k8s.io"}, OnRefuse: func(h string) { refused = h }} + addr := serve(t, s) + + // Driven directly rather than through a Transport, which would try to + // resolve nexus.corp itself before the proxy ever saw it. + c, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + defer c.Close() + io.WriteString(c, "GET http://nexus.corp/ubuntu/dists/noble/Release HTTP/1.1\r\nHost: nexus.corp\r\n\r\n") + br := bufio.NewReader(c) + r, err := http.ReadResponse(br, nil) + if err != nil { + t.Fatal(err) + } + defer r.Body.Close() + if r.StatusCode != http.StatusForbidden { + t.Errorf("status = %d, want 403", r.StatusCode) + } + body, _ := io.ReadAll(r.Body) + if !strings.Contains(string(body), "--proxy-allow") { + t.Errorf("the refusal does not say how to permit it: %s", body) + } + if refused != "nexus.corp" { + t.Errorf("OnRefuse got %q, want nexus.corp", refused) + } +} + +// A plain http fetch is what apt does against a mirror without TLS. +func TestForwardsAbsoluteFormRequests(t *testing.T) { + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Write([]byte("Origin: Ubuntu\n")) + })) + defer upstream.Close() + host := strings.TrimPrefix(upstream.URL, "http://") + + s := &Server{Allow: []string{"127.0.0.1"}} + addr := serve(t, s) + + c, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + defer c.Close() + io.WriteString(c, "GET "+upstream.URL+"/dists/noble/Release HTTP/1.1\r\nHost: "+host+"\r\n\r\n") + r, err := http.ReadResponse(bufio.NewReader(c), nil) + if err != nil { + t.Fatal(err) + } + defer r.Body.Close() + body, _ := io.ReadAll(r.Body) + if !strings.Contains(string(body), "Origin: Ubuntu") { + t.Errorf("body = %q", body) + } +} + +// CONNECT is what apt and containerd use for https, and the tunnel is what +// makes an air-gapped node able to fetch at all — the proxy resolves the name, +// so the node does not need working DNS. +func TestConnectTunnelsBytesBothWays(t *testing.T) { + // A trivial echo server standing in for the far end of the tunnel. + echo, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer echo.Close() + go func() { + c, err := echo.Accept() + if err != nil { + return + } + defer c.Close() + io.Copy(c, c) + }() + + s := &Server{Allow: []string{"127.0.0.1"}} + addr := serve(t, s) + + c, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + defer c.Close() + io.WriteString(c, "CONNECT "+echo.Addr().String()+" HTTP/1.1\r\nHost: "+echo.Addr().String()+"\r\n\r\n") + br := bufio.NewReader(c) + r, err := http.ReadResponse(br, nil) + if err != nil { + t.Fatal(err) + } + if r.StatusCode != http.StatusOK { + t.Fatalf("CONNECT returned %d", r.StatusCode) + } + io.WriteString(c, "ping") + buf := make([]byte, 4) + if _, err := io.ReadFull(br, buf); err != nil { + t.Fatal(err) + } + if string(buf) != "ping" { + t.Errorf("tunnel returned %q, want ping", buf) + } +} + +// A CONNECT to a host that is not allowed must be refused before anything is +// dialled — the point of the allowlist is that the connection is never made. +func TestConnectRefusedBeforeDialling(t *testing.T) { + dialled := false + s := &Server{ + Allow: []string{"pkgs.k8s.io"}, + Dial: func(n, a string) (net.Conn, error) { + dialled = true + return nil, nil + }, + } + addr := serve(t, s) + c, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + defer c.Close() + io.WriteString(c, "CONNECT evil.example:443 HTTP/1.1\r\nHost: evil.example:443\r\n\r\n") + r, err := http.ReadResponse(bufio.NewReader(c), nil) + if err != nil { + t.Fatal(err) + } + defer r.Body.Close() + if r.StatusCode != http.StatusForbidden { + t.Errorf("status = %d, want 403", r.StatusCode) + } + if dialled { + t.Error("a refused host was dialled anyway") + } +} + +// curl does not wait for the 200 before sending its TLS ClientHello, so those +// bytes are already in the http server's buffer when the connection is +// hijacked. Reading from the bare connection instead of that buffer drops +// them, and the far end then waits forever for a hello it was sent. +// +// Found live: every https fetch through the tunnel failed with "Proxy CONNECT +// aborted due to timeout" while plain http went through fine, because only the +// https path uses CONNECT. +func TestConnectKeepsBytesSentBeforeTheResponse(t *testing.T) { + echo, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer echo.Close() + go func() { + c, err := echo.Accept() + if err != nil { + return + } + defer c.Close() + io.Copy(c, c) + }() + + s := &Server{Allow: []string{"127.0.0.1"}} + addr := serve(t, s) + + c, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + defer c.Close() + + // The request and the payload go out together, in one write, exactly as a + // pipelining client sends them. + io.WriteString(c, "CONNECT "+echo.Addr().String()+" HTTP/1.1\r\nHost: "+ + echo.Addr().String()+"\r\n\r\nhello-before-200") + + br := bufio.NewReader(c) + r, err := http.ReadResponse(br, nil) + if err != nil { + t.Fatal(err) + } + if r.StatusCode != http.StatusOK { + t.Fatalf("CONNECT returned %d", r.StatusCode) + } + c.SetReadDeadline(time.Now().Add(5 * time.Second)) + buf := make([]byte, len("hello-before-200")) + if _, err := io.ReadFull(br, buf); err != nil { + t.Fatalf("the bytes sent before the 200 were dropped: %v", err) + } + if string(buf) != "hello-before-200" { + t.Errorf("tunnel returned %q", buf) + } +} + +// A stock cloud image points apt at its provider's mirror, not at +// archive.ubuntu.com. The first live run refused DigitalOcean's and stopped +// the install. +func TestDefaultAllowCoversCloudProviderMirrors(t *testing.T) { + s := &Server{} + for _, host := range []string{ + "repos-droplet.digitalocean.com", + "mirrors.digitalocean.com", + "us-east-1.ec2.archive.ubuntu.com", + "azure.archive.ubuntu.com", + } { + if !s.allowed(host) { + t.Errorf("a stock cloud image installs from %s and the allowlist refuses it", host) + } + } +} + +// A refusal names a host:port, and an operator copies that straight into +// --proxy-allow. If the entry is not stripped the same way the request is, the +// host they just permitted is refused again. +func TestAllowlistEntriesMayCarryAPort(t *testing.T) { + s := &Server{Allow: []string{"prod-cdn.packages.k8s.io:443"}} + if !s.allowed("prod-cdn.packages.k8s.io:443") { + t.Error("an entry with a port does not match the request it was copied from") + } + if !s.allowed("prod-cdn.packages.k8s.io") { + t.Error("an entry with a port does not match the bare host") + } +} + +// The package repositories redirect the actual download elsewhere, and an +// allowlist that only knows the name an operator would write refuses whatever +// the redirect points at. Both were hit live. +func TestDefaultAllowFollowsRedirectTargets(t *testing.T) { + s := &Server{} + for _, host := range []string{ + "prod-cdn.packages.k8s.io", + "us-central1-docker.pkg.dev", + "d1.cloudfront.net", + } { + if !s.allowed(host) { + t.Errorf("%s is where a redirect lands and the allowlist refuses it", host) + } + } +} + +// The CNI's own images are the ones an allowlist built from "what Kubernetes +// needs" forgets. Live, every control-plane image pulled and the cluster still +// sat NotReady, because flannel is on ghcr.io. +func TestDefaultAllowCoversCNIImages(t *testing.T) { + s := &Server{} + for _, host := range []string{ + "ghcr.io", "pkg-containers.githubusercontent.com", "quay.io", + } { + if !s.allowed(host) { + t.Errorf("a cluster cannot start its network plugin without %s", host) + } + } +} diff --git a/internal/ssh/local.go b/internal/ssh/local.go index 686b397..1c1a22e 100644 --- a/internal/ssh/local.go +++ b/internal/ssh/local.go @@ -76,6 +76,19 @@ func writeFileLocal(path string, data []byte, mode os.FileMode) error { return nil } +// writeFileStreamLocal backs WriteFileFrom for a local node. +func writeFileStreamLocal(path string, r io.Reader, mode os.FileMode) error { + f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, mode.Perm()) + if err != nil { + return fmt.Errorf("create %s: %w", path, err) + } + defer f.Close() + if _, err := io.Copy(f, r); err != nil { + return fmt.Errorf("write %s: %w", path, err) + } + return os.Chmod(path, mode.Perm()) +} + func removeFileLocal(path string) error { if err := os.Remove(path); err != nil && !os.IsNotExist(err) { return fmt.Errorf("rm %s: %w", path, err) diff --git a/internal/ssh/ssh.go b/internal/ssh/ssh.go index a1ae7bb..95833df 100644 --- a/internal/ssh/ssh.go +++ b/internal/ssh/ssh.go @@ -4,6 +4,7 @@ import ( "bytes" "fmt" "io" + "net" "os" "path/filepath" "strings" @@ -160,6 +161,77 @@ func (c *Client) WriteFile(remotePath string, data []byte, mode os.FileMode) err return nil } +// WriteFileFrom streams a file to the node instead of holding it in memory. +// +// WriteFile takes a []byte, which is fine for a manifest and wrong for a k3s +// airgap image archive: those run to a few hundred megabytes, and buffering +// one per node to send it a chunk at a time serves no purpose. Progress, when +// given, is called with the running byte count so a long upload can say what +// it is doing rather than appearing hung. +func (c *Client) WriteFileFrom(remotePath string, r io.Reader, mode os.FileMode, progress func(sent int64)) error { + if err := validRemotePath(remotePath); err != nil { + return err + } + if c.local { + return writeFileStreamLocal(remotePath, r, mode) + } + sess, err := c.conn.NewSession() + if err != nil { + return fmt.Errorf("new session: %w", err) + } + defer sess.Close() + sess.Stdin = &countingReader{r: r, cb: progress} + var errBuf bytes.Buffer + sess.Stderr = &errBuf + cmd := fmt.Sprintf("umask 077 && cat > '%s' && chmod %o '%s'", remotePath, mode.Perm(), remotePath) + if err := sess.Run(cmd); err != nil { + return fmt.Errorf("write %s: %w: %s", remotePath, err, strings.TrimSpace(errBuf.String())) + } + return nil +} + +// countingReader reports progress as the bytes go past. +type countingReader struct { + r io.Reader + n int64 + cb func(int64) + next int64 +} + +func (c *countingReader) Read(p []byte) (int, error) { + n, err := c.r.Read(p) + c.n += int64(n) + // Report every 8MB rather than every read: a callback per 32KB chunk turns + // a progress line into a flood. + if c.cb != nil && c.n >= c.next { + c.next = c.n + 8<<20 + c.cb(c.n) + } + return n, err +} + +// ListenRemote opens a listener on the *node*, carried back over this +// connection. +// +// It is how a node with no route out borrows the operator's: k3helper runs a +// proxy here, the node connects to a port on its own loopback, and the SSH +// channel carries the traffic. Nothing new is exposed to the network — the +// listener is on the node's loopback and the traffic rides a connection that +// already exists. +// +// Needs AllowTcpForwarding on the node's sshd, which is the default; a server +// with it disabled fails here rather than half way through an install. +func (c *Client) ListenRemote(addr string) (net.Listener, error) { + if c.local { + return nil, fmt.Errorf("a local node does not need a tunnel: it is this machine") + } + l, err := c.conn.Listen("tcp", addr) + if err != nil { + return nil, fmt.Errorf("open a listener on the node (is AllowTcpForwarding enabled in its sshd?): %w", err) + } + return l, nil +} + // RemoveFile deletes remotePath, ignoring "already gone". func (c *Client) RemoveFile(remotePath string) error { if err := validRemotePath(remotePath); err != nil { diff --git a/internal/tui/tui.go b/internal/tui/tui.go index 03beac6..5db202c 100644 --- a/internal/tui/tui.go +++ b/internal/tui/tui.go @@ -4,6 +4,7 @@ package tui import ( "fmt" + "sort" "strconv" "strings" "time" @@ -1060,10 +1061,17 @@ func (m Model) header() string { // healthScore is the share of check results that are OK, which is what the // dashboard ring and the status bar both report. +// A skipped check is not a failed one, so it is left out of both halves of the +// fraction. On a kubeconfig cluster every host check is skipped, and counting +// them as "not OK" scored a perfectly healthy managed cluster at 0% — which is +// the most alarming thing on the screen and means nothing at all. func (m Model) healthScore() (int, bool) { ok, total := 0, 0 for _, rs := range m.results { for _, r := range rs { + if r.Status == check.Skip { + continue + } total++ if r.Status == check.OK { ok++ @@ -1125,7 +1133,7 @@ func (m Model) footer() string { // dashboardBody renders the per-node check cards with load sparklines. func (m Model) dashboardBody() string { var b strings.Builder - ok, warn, fail := 0, 0, 0 + ok, warn, fail, skipped := 0, 0, 0, 0 for _, rs := range m.results { for _, r := range rs { switch r.Status { @@ -1135,6 +1143,8 @@ func (m Model) dashboardBody() string { warn++ case check.Fail: fail++ + case check.Skip: + skipped++ } } } @@ -1144,13 +1154,26 @@ func (m Model) dashboardBody() string { } else { b.WriteString(" ") } - b.WriteString(fmt.Sprintf("%s %d ok %s %d warn %s %d fail\n\n", + counts := fmt.Sprintf("%s %d ok %s %d warn %s %d fail", statusOKStyle.Render("●"), ok, statusWarnStyle.Render("●"), warn, - statusFailStyle.Render("●"), fail)) - - for _, node := range m.targets.Nodes { - rs := m.results[node.Name] + statusFailStyle.Render("●"), fail) + // Skipped checks are counted out loud. Without this a kubeconfig cluster + // reads "0 ok, 0 warn, 0 fail", which looks like nothing ran rather than + // like six host checks were deliberately not applicable. + if skipped > 0 { + counts += fmt.Sprintf(" %s %d skipped", helpStyle.Render("●"), skipped) + } + b.WriteString(counts + "\n\n") + + // Nodes first, in the order the targets file lists them, then anything + // else that produced results. A kubeconfig cluster has no nodes at all, + // and iterating only over them drew an empty dashboard — which is exactly + // the "nothing shown reads as all clear" that the skipped results exist to + // avoid. + for _, name := range m.resultOrder() { + rs := m.results[name] + node := m.nodeNamed(name) var lines []string for _, r := range rs { style := statusOKStyle @@ -1165,15 +1188,54 @@ func (m Model) dashboardBody() string { lines = append(lines, " "+statusFailStyle.Render("↳ "+r.Remediation)) } } - if graph := m.nodeGraphs(node.Name); graph != "" { + if graph := m.nodeGraphs(name); graph != "" { lines = append(lines, graph) } - card := fmt.Sprintf("%s (%s)\n%s", node.Name, node.Role, strings.Join(lines, "\n")) + title := name + if node.Role != "" { + title = fmt.Sprintf("%s (%s)", name, node.Role) + } + card := fmt.Sprintf("%s\n%s", title, strings.Join(lines, "\n")) b.WriteString(paneBorder.Render(card) + "\n") } return b.String() } +// resultOrder lists the groups the dashboard draws: the targets file's nodes +// first, in its order, then any group that produced results without being a +// node in the file — which is how a kubeconfig cluster's skipped host checks +// reach the screen at all. +func (m Model) resultOrder() []string { + var out []string + seen := map[string]bool{} + for _, n := range m.targets.Nodes { + if _, ok := m.results[n.Name]; ok { + out = append(out, n.Name) + seen[n.Name] = true + } + } + var rest []string + for name := range m.results { + if !seen[name] { + rest = append(rest, name) + } + } + sort.Strings(rest) + return append(out, rest...) +} + +// nodeNamed returns the targets-file node with this name, or a zero node when +// the group is not one — a kubeconfig cluster's results are filed under the +// context, which is not a machine. +func (m Model) nodeNamed(name string) config.Node { + for _, n := range m.targets.Nodes { + if n.Name == name { + return n + } + } + return config.Node{} +} + // nodeGraphs is the CPU/memory sparkline pair for one node's card. func (m Model) nodeGraphs(node string) string { cpu, mem := m.hist.cpuFor(node), m.hist.memFor(node) diff --git a/internal/tui/tui_test.go b/internal/tui/tui_test.go index 6b5633e..91f2a9c 100644 --- a/internal/tui/tui_test.go +++ b/internal/tui/tui_test.go @@ -9,6 +9,7 @@ import ( "github.com/solutionforest/k3helper/internal/check" "github.com/solutionforest/k3helper/internal/config" + "strings" ) func testTargets() *config.Targets { @@ -68,3 +69,78 @@ func stringsIndexOf(s, sub string) int { } return -1 } + +// A skipped check must not drag the health score down. +// +// Every host check is skipped on a kubeconfig cluster, and counting those as +// "not OK" put "score 0%" at the top of a screen showing a perfectly healthy +// cluster — spotted in a screenshot of a live DigitalOcean cluster. +func TestHealthScoreIgnoresSkippedChecks(t *testing.T) { + m := Model{results: map[string][]check.Result{ + "kubeconfig": check.SkippedHostResults("server"), + }} + if score, have := m.healthScore(); have { + t.Errorf("a cluster with nothing but skipped checks reported a score of %d%%, want none", score) + } + + m = Model{results: map[string][]check.Result{ + "node1": { + {Status: check.OK}, + {Status: check.OK}, + {Status: check.Skip}, + {Status: check.Fail}, + }, + }} + score, have := m.healthScore() + if !have { + t.Fatal("no score for a node with real results") + } + // 2 OK out of 3 that were actually run; the skip is not a third failure. + if score != 66 { + t.Errorf("score = %d%%, want 66%% (2 of 3 run, skip excluded)", score) + } +} + +// A kubeconfig cluster has no nodes in its targets file, and the dashboard +// used to iterate only over those — so the skipped host checks, and the reason +// they were skipped, never reached the screen. An empty dashboard reads as +// "all clear", which is the one thing those results exist to prevent. +func TestDashboardShowsResultsForClustersWithNoNodes(t *testing.T) { + m := Model{ + targets: &config.Targets{Cluster: "prod", Kubeconfig: "/k/c.yaml"}, + results: map[string][]check.Result{ + "prod-admin (kubeconfig)": check.SkippedHostResults("server"), + }, + hist: newHistory(), + } + body := m.dashboardBody() + if !strings.Contains(body, "prod-admin") { + t.Errorf("the result group is not on the dashboard:\n%s", body) + } + if !strings.Contains(body, "kubeconfig cluster") { + t.Errorf("the reason the checks were skipped is not shown:\n%s", body) + } + if !strings.Contains(body, "skipped") { + t.Errorf("the counter does not mention skipped checks:\n%s", body) + } +} + +// Nodes keep the targets file's order, which is the order an operator wrote +// them in and expects to read them in. +func TestDashboardKeepsNodeOrder(t *testing.T) { + m := Model{ + targets: &config.Targets{Nodes: []config.Node{ + {Name: "server", Role: "server"}, + {Name: "agent1", Role: "agent"}, + }}, + results: map[string][]check.Result{ + "agent1": {{Name: "Disk", Status: check.OK, Summary: "fine"}}, + "server": {{Name: "Disk", Status: check.OK, Summary: "fine"}}, + }, + hist: newHistory(), + } + body := m.dashboardBody() + if strings.Index(body, "server") > strings.Index(body, "agent1") { + t.Errorf("agent1 was drawn before server:\n%s", body) + } +} diff --git a/internal/vm/kubeadm.go b/internal/vm/kubeadm.go index 7c76e24..230f93e 100644 --- a/internal/vm/kubeadm.go +++ b/internal/vm/kubeadm.go @@ -4,6 +4,7 @@ import ( "encoding/base64" "fmt" "io" + "net" "strings" "time" ) @@ -37,15 +38,39 @@ func SetupKubeadm(servers []Target, agents []Target, opts KubeadmOptions) error all := append(append([]Target{}, servers...), agents...) + // A node that cannot reach a distribution mirror cannot install kubeadm at + // all: unlike k3s it is apt packages and registry images, not one binary. + // Both answers are set up before any of it runs. + var proxies []*nodeProxy + if opts.ViaProxy { + var err error + proxies, err = startProxies(all, opts.ProxyAllow, opts.Progress) + if err != nil { + return err + } + defer stopProxies(proxies) + } + if !opts.Mirror.empty() { + for _, t := range all { + if err := applyAptMirror(t, opts.Mirror, opts.Progress); err != nil { + return err + } + } + } + // 1. prerequisites, every node for _, t := range all { + // Before apt runs, or it collides with cloud-init's dpkg lock on a + // machine that has only just booted. + waitForCloudInit(t, opts.Progress) progressf("[%s] preparing host (containerd, kernel modules, sysctls, kubeadm)...", t.Node.Host) // The script is multi-line and contains quotes of its own, so it is // shipped base64-encoded and fed to a root shell rather than being // interpolated into one. - run := fmt.Sprintf(`printf %%s %s | base64 -d | %sbash -s`, - shellQuote(base64.StdEncoding.EncodeToString([]byte(kubeadmPrereqScript(opts.Version)))), - t.Client.SudoPrefix()) + run := fmt.Sprintf(`printf %%s %s | base64 -d | %s%sbash -s`, + shellQuote(base64.StdEncoding.EncodeToString([]byte( + kubeadmPrereqScript(opts.Version, opts.Mirror)))), + t.Client.SudoPrefix(), proxyEnv(proxies, t.Node.Host)) if code, err := streamSudo(t.Client, run, opts.Progress); err != nil || code != 0 { return fmt.Errorf("prepare %s: %s", t.Node.Host, exitReason(code, err)) } @@ -53,10 +78,30 @@ func SetupKubeadm(servers []Target, agents []Target, opts KubeadmOptions) error // 2. control plane first := servers[0] + + // Pin the address the API server advertises, because the join command the + // agents are handed later is built from it. + // + // kubeadm's default is the address of the default route's interface, which + // on a cloud VM is the public one — the same trap the k3s path fell into, + // where agents on a private network were handed a public address they + // could not reach and retried against it indefinitely. The address from + // the targets file is the one the operator chose and the one k3helper has + // just proved works by connecting over it. + advertise := "" + if ip, err := resolveJoinAddress(opts.JoinAddress, first, first.Client); err == nil && net.ParseIP(ip) != nil { + advertise = " --apiserver-advertise-address=" + shellQuote(ip) + progressf("[%s] API server will advertise %s", first.Node.Host, ip) + } + progressf("[%s] kubeadm init (pod network %s)...", first.Node.Host, opts.podCIDR()) + // The environment goes after sudo, not before it: sudo resets it, so a + // prefix on the outside reaches the shell and not the command that + // actually fetches anything. + firstEnv := proxyEnv(proxies, first.Node.Host) initCmd := fmt.Sprintf( - `%skubeadm init --pod-network-cidr=%s%s`, - first.Client.SudoPrefix(), shellQuote(opts.podCIDR()), withSpace(opts.InitExtraArgs)) + `%s%skubeadm init --pod-network-cidr=%s%s%s`, + first.Client.SudoPrefix(), firstEnv, shellQuote(opts.podCIDR()), advertise, withSpace(opts.InitExtraArgs)) if opts.SkipConntrackTuning { // A config file rather than flags: kube-proxy's conntrack settings // have no command-line equivalent on kubeadm init. @@ -67,8 +112,8 @@ func SetupKubeadm(servers []Target, agents []Target, opts KubeadmOptions) error )); err != nil || code != 0 { return fmt.Errorf("write kubeadm config: %s", exitReason(code, err)) } - initCmd = fmt.Sprintf(`%skubeadm init --config /etc/kubernetes/k3helper-init.yaml%s`, - first.Client.SudoPrefix(), withSpace(opts.InitExtraArgs)) + initCmd = fmt.Sprintf(`%s%skubeadm init --config /etc/kubernetes/k3helper-init.yaml%s%s`, + first.Client.SudoPrefix(), firstEnv, advertise, withSpace(opts.InitExtraArgs)) } if code, err := streamSudo(first.Client, initCmd, opts.Progress); err != nil || code != 0 { return fmt.Errorf("kubeadm init failed: %s", exitReason(code, err)) @@ -85,8 +130,12 @@ func SetupKubeadm(servers []Target, agents []Target, opts KubeadmOptions) error // 3. CNI — without one every node stays NotReady, which looks like a // broken install rather than a missing component. progressf("[%s] installing CNI (%s)...", first.Node.Host, opts.cni()) - cniCmd := fmt.Sprintf(`%skubectl --kubeconfig /etc/kubernetes/admin.conf apply -f %s`, - first.Client.SudoPrefix(), shellQuote(opts.cniManifest())) + // The CNI manifest is a URL, and kubectl fetches it itself — so this needs + // the proxy as much as apt did. Without it the step failed with a bare + // "dial tcp 20.205.243.166:443: i/o timeout" on a node that had just + // installed Kubernetes perfectly well through the tunnel. + cniCmd := fmt.Sprintf(`%s%skubectl --kubeconfig /etc/kubernetes/admin.conf apply -f %s`, + first.Client.SudoPrefix(), firstEnv, shellQuote(opts.cniManifest())) if code, err := streamSudo(first.Client, cniCmd, opts.Progress); err != nil || code != 0 { return fmt.Errorf("install CNI: %s", exitReason(code, err)) } @@ -104,7 +153,8 @@ func SetupKubeadm(servers []Target, agents []Target, opts KubeadmOptions) error } for _, a := range agents { progressf("[%s] joining cluster...", a.Node.Host) - if code, err := streamSudo(a.Client, a.Client.SudoPrefix()+join, opts.Progress); err != nil || code != 0 { + joinCmd := a.Client.SudoPrefix() + proxyEnv(proxies, a.Node.Host) + join + if code, err := streamSudo(a.Client, joinCmd, opts.Progress); err != nil || code != 0 { return fmt.Errorf("join %s failed: %s", a.Node.Host, exitReason(code, err)) } } @@ -125,6 +175,19 @@ type KubeadmOptions struct { CNI string // InitExtraArgs is appended to `kubeadm init`. InitExtraArgs string + // JoinAddress overrides the address the API server advertises, which is + // the address the join command hands to every agent. Empty takes it from + // the targets file. + JoinAddress string + // Mirror points the node's package manager at an internal mirror instead + // of the distribution's own archive. + Mirror AptMirror + // ViaProxy lends each node this machine's internet connection for the + // length of the install, over the SSH connection already open to it. + // Off unless asked for: see viaproxy.go. + ViaProxy bool + // ProxyAllow extends the proxy's host allowlist. + ProxyAllow []string // SkipConntrackTuning stops kube-proxy managing nf_conntrack_max. // // kube-proxy raises that sysctl at startup and dies if it cannot. On hosts @@ -182,13 +245,29 @@ func (o KubeadmOptions) readyTimeout() int { // kubelet refuse to start, the br_netfilter module and its sysctls are what // let pod traffic be seen by iptables, and without a CRI there is nothing to // run containers with. -func kubeadmPrereqScript(version string) string { +func kubeadmPrereqScript(version string, m AptMirror) string { if version == "" { version = "v1.31" } + // The Kubernetes packages live on their own service, mirrored separately + // from the distribution's archive — a site may well have one and not the + // other. + k8sRepo := "https://pkgs.k8s.io/core:/stable:/" + version + "/deb/" + if m.K8sRepo != "" { + k8sRepo = strings.TrimRight(m.K8sRepo, "/") + "/" + } return `set -e export DEBIAN_FRONTEND=noninteractive +# Wait for the apt lock rather than failing on it. +# +# Waiting for cloud-init is not enough: apt-daily and unattended-upgrades are +# on timers and can take the lock minutes after boot, long after cloud-init has +# finished. The failure is "Could not get lock /var/lib/apt/lists/lock", which +# says nothing about the timer that is holding it. apt has taken this option +# since 1.9, and Ubuntu 24.04 is well past that. +APT="apt-get -o DPkg::Lock::Timeout=300" + # kubelet refuses to start with swap enabled. swapoff -a || true sed -i.bak '/[[:space:]]swap[[:space:]]/s/^/#/' /etc/fstab 2>/dev/null || true @@ -208,11 +287,11 @@ printf 'overlay\nbr_netfilter\nnf_conntrack\n' > /etc/modules-load.d/k8s.conf printf 'net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward = 1\nnet.netfilter.nf_conntrack_max = 1048576\n' > /etc/sysctl.d/k8s.conf sysctl --system >/dev/null 2>&1 || true -apt-get update -qq +$APT update -qq # conntrack is a hard kubeadm preflight requirement; socat is what # "kubectl port-forward" uses on the node; ethtool and iptables are needed by # most CNIs. Missing any of them fails late and unhelpfully. -apt-get install -y -qq apt-transport-https ca-certificates curl gpg containerd \ +$APT install -y -qq apt-transport-https ca-certificates curl gpg containerd \ conntrack socat ethtool iptables # containerd's shipped default disables CRI; kubeadm needs it, and the cgroup @@ -224,12 +303,12 @@ systemctl restart containerd systemctl enable containerd install -m 755 -d /etc/apt/keyrings -curl -fsSL https://pkgs.k8s.io/core:/stable:/` + version + `/deb/Release.key | +curl -fsSL ` + k8sRepo + `Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg --yes -echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/` + version + `/deb/ /" \ +echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] ` + k8sRepo + ` /" \ > /etc/apt/sources.list.d/kubernetes.list -apt-get update -qq -apt-get install -y -qq kubelet kubeadm kubectl +$APT update -qq +$APT install -y -qq kubelet kubeadm kubectl apt-mark hold kubelet kubeadm kubectl >/dev/null systemctl enable kubelet ` diff --git a/internal/vm/kubeadm_test.go b/internal/vm/kubeadm_test.go new file mode 100644 index 0000000..8dfbe45 --- /dev/null +++ b/internal/vm/kubeadm_test.go @@ -0,0 +1,122 @@ +package vm + +import ( + "strings" + "testing" + + "github.com/solutionforest/k3helper/internal/ssh" +) + +// kubeadm defaults the API server's advertise address to the default route's +// interface, which on a cloud VM is the public one — and the join command +// handed to every agent is built from it. That is the same fault the k3s path +// had against real air-gapped VMs, where agents dialled a public address their +// network could not reach. +func TestKubeadmAdvertisesTheConfiguredAddress(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "10.104.0.13") + if err := SetupKubeadm(servers, nil, KubeadmOptions{}); err != nil { + t.Fatalf("SetupKubeadm: %v", err) + } + var init string + for _, c := range rec { + if strings.Contains(c, "kubeadm init") { + init = c + } + } + if init == "" { + t.Fatal("no kubeadm init was issued") + } + if !strings.Contains(init, "--apiserver-advertise-address='10.104.0.13'") { + t.Errorf("kubeadm init did not pin the advertise address: %s", init) + } +} + +func TestKubeadmAdvertiseOverride(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "203.0.113.10") + if err := SetupKubeadm(servers, nil, KubeadmOptions{JoinAddress: "10.104.0.13"}); err != nil { + t.Fatalf("SetupKubeadm: %v", err) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "--apiserver-advertise-address='10.104.0.13'") { + t.Errorf("--join-address was ignored:\n%s", all) + } +} + +// A hostname cannot be an advertise address: the flag takes an IP. Rather than +// pass something kubeadm will reject, the flag is left off and kubeadm's own +// default applies. +func TestKubeadmSkipsAdvertiseForHostname(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "server.internal.example") + if err := SetupKubeadm(servers, nil, KubeadmOptions{}); err != nil { + t.Fatalf("SetupKubeadm: %v", err) + } + all := strings.Join(rec, "\n") + if strings.Contains(all, "--apiserver-advertise-address") { + t.Errorf("a hostname was passed where kubeadm wants an IP:\n%s", all) + } +} + +// Every step that reaches the network needs the proxy, and it has to sit after +// sudo — sudo resets the environment, so a prefix on the outside reaches the +// shell and not the command doing the fetching. +// +// The CNI step is a URL kubectl fetches itself. Live, it was the one step that +// had been missed: the node installed Kubernetes fine through the tunnel and +// then failed with "dial tcp 20.205.243.166:443: i/o timeout". +func TestKubeadmProxyReachesEveryNetworkStep(t *testing.T) { + var rec []string + servers := []Target{{Node: ssh.Node{Host: "10.104.0.5"}, + Client: &tunnelHost{fakeHost: fakeHost{name: "s1", rec: &rec}}}} + agents := []Target{{Node: ssh.Node{Host: "10.104.0.6"}, + Client: &tunnelHost{fakeHost: fakeHost{name: "a1", rec: &rec}}}} + if err := SetupKubeadm(servers, agents, KubeadmOptions{ViaProxy: true}); err != nil { + t.Fatalf("SetupKubeadm: %v", err) + } + steps := map[string]string{ + "prerequisites": "base64 -d", + "kubeadm init": "kubeadm init", + "CNI": "apply -f", + "join": "kubeadm join", + } + for name, needle := range steps { + var cmd string + for _, c := range rec { + if strings.Contains(c, needle) { + cmd = c + } + } + if cmd == "" { + t.Errorf("no %s step was run", name) + continue + } + if !strings.Contains(cmd, "http_proxy=http://127.0.0.1:") { + t.Errorf("the %s step runs without the proxy: %s", name, cmd) + } + // After sudo, not before: sudo resets the environment. + if i, j := strings.Index(cmd, "sudo"), strings.Index(cmd, "http_proxy="); i >= 0 && j >= 0 && j < i { + t.Errorf("the %s step puts the proxy before sudo, which resets it: %s", name, cmd) + } + } +} + +// Waiting for cloud-init is not enough on its own: apt-daily and +// unattended-upgrades run on timers and can take the lock minutes after boot. +// Live, that failed the second node with "Could not get lock +// /var/lib/apt/lists/lock. It is held by process 10754 (apt-get)". +func TestKubeadmPrereqWaitsForTheAptLock(t *testing.T) { + script := kubeadmPrereqScript("v1.31", AptMirror{}) + if !strings.Contains(script, "DPkg::Lock::Timeout") { + t.Errorf("apt is run without waiting for the lock:\n%s", script) + } + // Every apt invocation, not just the first: the lock can be taken between + // them. + for _, line := range strings.Split(script, "\n") { + l := strings.TrimSpace(line) + if strings.HasPrefix(l, "apt-get ") { + t.Errorf("this apt-get does not wait for the lock: %s", l) + } + } +} diff --git a/internal/vm/mirror.go b/internal/vm/mirror.go new file mode 100644 index 0000000..fdf5b59 --- /dev/null +++ b/internal/vm/mirror.go @@ -0,0 +1,88 @@ +package vm + +import ( + "fmt" + "io" + "strings" +) + +// This file points a node's package manager at a mirror the operator runs. +// +// It is the other half of the air-gap answer, and for most organisations it is +// the half that matters. A site with no internet almost always already has an +// internal mirror — Artifactory, Nexus, Satellite, a plain reverse proxy — and +// the useful thing k3helper can do is point at it, not replace it. Lending a +// node the operator's own connection (see viaproxy.go) is for sites that have +// no mirror either. + +// AptMirror is a distribution mirror, e.g. https://nexus.corp/repository/ubuntu. +// +// The substitution is on the archive URL, keeping the suite and component +// after it, because that is how a mirror is meant to be a drop-in: the paths +// below the root are identical to the upstream's. +type AptMirror struct { + // URL replaces the distribution archive. + URL string + // K8sRepo replaces the Kubernetes package repository, which is a separate + // service from the distribution's and is mirrored separately. Empty leaves + // pkgs.k8s.io alone — a site may mirror one and not the other. + K8sRepo string +} + +func (m AptMirror) empty() bool { return m.URL == "" && m.K8sRepo == "" } + +// aptMirrorScript rewrites the node's sources to point at the mirror. +// +// Both source formats are handled: Ubuntu 24.04 ships deb822 files under +// /etc/apt/sources.list.d/*.sources, while older images and most third-party +// repositories use one-line entries. Missing one of them leaves half the +// sources pointing at an archive the node cannot reach, and `apt-get update` +// then fails on exactly the half that was missed. +// +// Every file is backed up before it is touched, so a mistyped mirror is one +// command away from being undone rather than a reinstall. +func aptMirrorScript(m AptMirror) string { + if m.URL == "" { + return "" + } + mirror := strings.TrimRight(m.URL, "/") + return ` +set -e +for f in /etc/apt/sources.list /etc/apt/sources.list.d/*.sources /etc/apt/sources.list.d/*.list; do + [ -f "$f" ] || continue + [ -f "$f.k3helper.bak" ] || cp -a "$f" "$f.k3helper.bak" + # Matches the archive root of a Debian or Ubuntu mirror — including the + # cloud images' own mirrors, which all end in /ubuntu or /ubuntu-ports — and + # leaves everything after it alone, which is the suite and components. + sed -i -E 's#https?://[A-Za-z0-9._~-]+(/[A-Za-z0-9._~-]+)*?/ubuntu-ports(/)?#` + mirror + `/#g; ' "$f" + sed -i -E 's#https?://[A-Za-z0-9._~-]+(/[A-Za-z0-9._~-]+)*?/ubuntu(/)?#` + mirror + `/#g; ' "$f" + sed -i -E 's#https?://[A-Za-z0-9._~-]+(/[A-Za-z0-9._~-]+)*?/debian(/)?#` + mirror + `/#g; ' "$f" +done +` +} + +// applyAptMirror rewrites the sources on one node. +func applyAptMirror(t Target, m AptMirror, progress io.Writer) error { + script := aptMirrorScript(m) + if script == "" { + return nil + } + if progress != nil { + fmt.Fprintf(progress, "[%s] pointing apt at %s\n", t.Node.Host, m.URL) + } + out, code, err := t.Client.SudoRun(fmt.Sprintf("sh -c %s", shellQuote(script))) + if err != nil || code != 0 { + return fmt.Errorf("node %s: could not point apt at %s (exit %d): %s", + t.Node.Host, m.URL, code, strings.TrimSpace(out)) + } + // Prove it: a rewrite that silently matched nothing leaves the node + // pointing at an archive it cannot reach, and the next apt-get update + // fails for a reason that looks nothing like a mirror problem. + check, _, _ := t.Client.Run( + `grep -rhoE 'https?://[^ ]+' /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | head -20`) + if check != "" && !strings.Contains(check, strings.TrimRight(m.URL, "/")) { + return fmt.Errorf("node %s: the sources still do not mention %s after rewriting them:\n%s\n"+ + "Restore them with: cp .k3helper.bak ", t.Node.Host, m.URL, strings.TrimSpace(check)) + } + return nil +} diff --git a/internal/vm/offline.go b/internal/vm/offline.go new file mode 100644 index 0000000..eb195c4 --- /dev/null +++ b/internal/vm/offline.go @@ -0,0 +1,219 @@ +package vm + +import ( + "fmt" + "io" + "os" + "path/filepath" + "strings" + + "github.com/solutionforest/k3helper/internal/bundle" +) + +// This file installs k3s on a node that cannot reach the internet. +// +// The online install is one pipe: `curl -sfL https://get.k3s.io | sh -`. That +// needs three things an air-gapped node does not have — update.k3s.io to +// resolve a channel, the GitHub release to fetch the binary, and a registry +// for every image a pod pulls. The offline install supplies all three from a +// bundle: the binary is placed where the installer expects to find it, the +// airgap image archive is dropped where k3s imports it into containerd on +// first start, and INSTALL_K3S_SKIP_DOWNLOAD tells the installer not to reach +// for any of it. + +// Uploader is a Host that can also receive files. Split from Host because the +// online path does not need it and its tests would otherwise have to grow +// methods they never call. +type Uploader interface { + Host + WriteFileFrom(remotePath string, r io.Reader, mode os.FileMode, progress func(int64)) error +} + +// Paths the k3s installer reads when INSTALL_K3S_SKIP_DOWNLOAD is set. +const ( + remoteBinary = "/usr/local/bin/k3s" + remoteImageDir = "/var/lib/rancher/k3s/agent/images" + remoteStage = "/tmp/k3helper-k3s-bundle" +) + +// stageBundle puts a bundle's contents on one node. +// +// Files land in /tmp first and are moved into place with sudo, because the +// login user can rarely write to /usr/local/bin or /var/lib/rancher directly — +// and a partially written k3s binary at the real path is worse than none. +func stageBundle(t Target, dir string, m *bundle.Manifest, progress io.Writer) error { + up, ok := t.Client.(Uploader) + if !ok { + return fmt.Errorf("node %s: this connection cannot upload files, so an offline install is not possible", t.Node.Host) + } + say := func(format string, a ...any) { + if progress != nil { + fmt.Fprintf(progress, format+"\n", a...) + } + } + + // Check the architecture before sending 260MB to a machine that cannot run + // it. The manifest records what the bundle is for; without comparing it to + // the node, the first sign of a mismatch is the installer reporting + // "cannot execute binary file" after the whole upload has finished. + if err := checkArch(t, m.Arch); err != nil { + return err + } + + if _, code, err := t.Client.Run(fmt.Sprintf("mkdir -p '%s'", remoteStage)); err != nil || code != 0 { + return fmt.Errorf("node %s: could not create a staging directory", t.Node.Host) + } + + for _, f := range []struct { + name string + mode os.FileMode + }{ + {bundle.BinaryName, 0o755}, + {bundle.ImagesName, 0o644}, + {bundle.InstallName, 0o755}, + } { + local := filepath.Join(dir, f.name) + info, err := os.Stat(local) + if err != nil { + return fmt.Errorf("bundle file %s: %w", local, err) + } + src, err := os.Open(local) + if err != nil { + return err + } + size := info.Size() + say("[%s] uploading %s (%s)...", t.Node.Host, f.name, humanSize(size)) + err = up.WriteFileFrom(remoteStage+"/"+f.name, src, f.mode, func(sent int64) { + if size > 0 { + say("[%s] %s %3d%%", t.Node.Host, f.name, sent*100/size) + } + }) + src.Close() + if err != nil { + return fmt.Errorf("node %s: upload %s: %w", t.Node.Host, f.name, err) + } + } + + // Verify what landed. 260MB over a link that may be slow and may be a + // tunnel is worth one hash: a truncated k3s binary fails as "cannot + // execute binary file" and a truncated image archive fails much later, as + // pods that will not start on a cluster that installed cleanly. + if err := verifyStaged(t, m); err != nil { + return err + } + + // Put everything where the installer looks. The image archive keeps its + // architecture-qualified name: k3s does not care what the file is called, + // but an operator looking at the directory later does. + place := fmt.Sprintf( + `install -m 0755 '%s/%s' '%s' && mkdir -p '%s' && install -m 0644 '%s/%s' '%s/k3s-airgap-images-%s.tar.zst'`, + remoteStage, bundle.BinaryName, remoteBinary, + remoteImageDir, + remoteStage, bundle.ImagesName, remoteImageDir, m.Arch, + ) + if out, code, err := t.Client.SudoRun(place); err != nil || code != 0 { + return fmt.Errorf("node %s: could not place the bundle (exit %d): %s", t.Node.Host, code, out) + } + return nil +} + +// verifyStaged hashes the uploaded files on the node and compares them with +// the manifest. +// +// A node without sha256sum is not failed over it — the check is a safeguard, +// not a requirement — but one that answers with a different hash is, because +// continuing means installing something that is not what was fetched. +func verifyStaged(t Target, m *bundle.Manifest) error { + for _, f := range []struct{ name, want string }{ + {bundle.BinaryName, m.BinarySHA}, + {bundle.ImagesName, m.ImagesSHA}, + } { + if f.want == "" { + continue + } + out, code, err := t.Client.Run(fmt.Sprintf( + `command -v sha256sum >/dev/null 2>&1 && sha256sum '%s/%s' | cut -d' ' -f1 || true`, + remoteStage, f.name)) + if err != nil || code != 0 { + continue + } + got := strings.TrimSpace(out) + if got == "" { + continue // no sha256sum on this node + } + if !strings.EqualFold(got, f.want) { + return fmt.Errorf("node %s: %s arrived corrupted (expected %s, got %s) — "+ + "the upload did not survive the link; try again", + t.Node.Host, f.name, f.want[:12], got[:min(12, len(got))]) + } + } + return nil +} + +// cleanStage removes the uploaded copies once they are installed. +// +// The bundle is a few hundred megabytes and the node keeps the parts it needs +// elsewhere: the binary at /usr/local/bin/k3s and the archive under +// /var/lib/rancher. Leaving the staging copy behind wastes that much disk on +// every node, on machines whose disks k3helper itself warns about when they +// fill up. +func cleanStage(t Target) { + t.Client.SudoRun(fmt.Sprintf("rm -rf '%s'", remoteStage)) +} + +// offlineInstallCmd renders the install command for a staged bundle. +// +// INSTALL_K3S_SKIP_DOWNLOAD makes the installer use the binary already at +// /usr/local/bin/k3s and skip both the channel lookup and the release +// download. Without it the script fetches regardless of what is on disk, which +// on an air-gapped node is a TLS error and a puzzled operator. +func offlineInstallCmd(sudoPrefix, env, role, args string) string { + return fmt.Sprintf( + `%sINSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_BIN_DIR=/usr/local/bin %ssh '%s/%s' %s%s`, + sudoPrefix, env, remoteStage, bundle.InstallName, role, args, + ) +} + +func humanSize(n int64) string { + switch { + case n >= 1<<30: + return fmt.Sprintf("%.1fGB", float64(n)/float64(1<<30)) + case n >= 1<<20: + return fmt.Sprintf("%.0fMB", float64(n)/float64(1<<20)) + default: + return fmt.Sprintf("%dKB", n/1024) + } +} + +// checkArch compares the bundle's architecture with the node's. +// +// A node that will not say what it is passes: `uname -m` is not worth failing +// an install over, and the installer will report a mismatch soon enough. A +// node that says something different is refused, because that is a certainty +// rather than a guess. +func checkArch(t Target, want string) error { + out, code, err := t.Client.Run("uname -m") + if err != nil || code != 0 { + return nil + } + got := archFromUname(strings.TrimSpace(out)) + if got == "" || got == want { + return nil + } + return fmt.Errorf("node %s is %s but the bundle is for %s — "+ + "rebuild it with `k3helper bundle k3s --version --arch %s`", + t.Node.Host, got, want, got) +} + +// archFromUname maps the kernel's name for an architecture onto Go's, which is +// what k3s uses in its release assets. An unrecognised value returns "", which +// checkArch treats as "do not know, do not block". +func archFromUname(m string) string { + switch m { + case "x86_64", "amd64": + return "amd64" + case "aarch64", "arm64": + return "arm64" + } + return "" +} diff --git a/internal/vm/setup.go b/internal/vm/setup.go index a29200d..d6722b8 100644 --- a/internal/vm/setup.go +++ b/internal/vm/setup.go @@ -3,6 +3,7 @@ package vm import ( "fmt" + "github.com/solutionforest/k3helper/internal/bundle" "io" "regexp" "strings" @@ -15,8 +16,30 @@ import ( type Options struct { // InstallURL overrides the k3s install script (tests use a local stub). InstallURL string - // Channel: stable, latest, or a fixed version like v1.31.2+k3s1 + // Channel: stable, latest, or testing. Resolved by update.k3s.io at + // install time, so it needs that service to be up. Channel string + // Version pins an exact k3s release, e.g. v1.31.2+k3s1. When set it wins + // over Channel and the install skips update.k3s.io entirely, fetching the + // build straight from its GitHub release. + Version string + // BundleDir installs from a k3s bundle instead of downloading anything. + // The nodes need no internet at all: see internal/bundle. + BundleDir string + // JoinAddress overrides the address the other nodes dial to reach the + // first server. Needed when k3helper connects over one network and the + // cluster talks over another — a public IP for SSH, a private one between + // nodes. + JoinAddress string + // Mirror points the node's package manager at an internal mirror. k3s + // itself needs no packages, but a node may still want a reachable archive + // for everything else on it. + Mirror AptMirror + // ViaProxy lends each node this machine's internet connection for the + // length of the install. Off unless asked for: see viaproxy.go. + ViaProxy bool + // ProxyAllow extends the proxy's host allowlist. + ProxyAllow []string // ExtraArgs appended to the install command (e.g. "--disable traefik") ServerExtraArgs string AgentExtraArgs string @@ -33,6 +56,22 @@ func (o *Options) installURL() string { return "https://get.k3s.io" } +// installFailure explains why an install step did not work. +// +// A step fails in two different ways and they need different words: the +// connection itself broke (err is non-nil), or the command ran and exited +// non-zero (err is nil, and the reason is in the output already streamed to +// the operator). Passing a nil error to %w printed "%!w()" where the +// reason should have been — seen against a real VM, and it is the last thing +// on screen when an install fails. +// +// The kubeadm path has always got this right through exitReason; this is the +// k3s path being brought to the same place rather than a second way of saying +// it. +func installFailure(what string, code int, err error) error { + return fmt.Errorf("%s failed: %s", what, exitReason(code, err)) +} + func (o Options) channel() string { if o.Channel == "" { return "stable" @@ -40,6 +79,96 @@ func (o Options) channel() string { return o.Channel } +// release renders the environment that tells the k3s installer which build to +// fetch: either an exact version, or a channel to resolve. +// +// A channel is a lookup against update.k3s.io, which is a second service that +// has to be up and correctly certificated. It was neither during live testing +// — it served a Traefik default certificate from all three of its addresses, +// so every `curl -sfL https://get.k3s.io | sh -` on the internet failed TLS +// verification and then failed to download. Pinning a version skips that +// service entirely and fetches straight from the GitHub release, which was +// healthy throughout. +func (o Options) release() string { + if o.Version != "" { + return "INSTALL_K3S_VERSION=" + shellQuote(o.Version) + } + return "INSTALL_K3S_CHANNEL=" + o.channel() +} + +// offline reports whether this install comes from a bundle. +func (o Options) offline() bool { return o.BundleDir != "" } + +// serverInstallCmd builds the first server's install command. +// +// The online and offline forms differ only in where the installer and the +// binary come from; the arguments after `server` are identical, so they are +// built once here rather than in three places that would drift. +func (o Options) serverInstallCmd(t Target, env, tokenArg, initArgs string) string { + if o.offline() { + return offlineInstallCmd(t.Client.SudoPrefix(), strings.TrimSpace(tokenArg)+" ", "server", withSpace(initArgs)) + } + return fmt.Sprintf( + `curl -sfL %s | %s%s%s%s sh -s - server%s`, + o.installURL(), t.Client.SudoPrefix(), env, o.release(), tokenArg, withSpace(initArgs), + ) +} + +func (o Options) joinServerCmd(t Target, env, token, ip string) string { + args := fmt.Sprintf(" --server https://%s:6443%s", ip, withSpace(o.ServerExtraArgs)) + if o.offline() { + return offlineInstallCmd(t.Client.SudoPrefix(), + "K3S_TOKEN="+shellQuote(token)+" ", "server", args) + } + return fmt.Sprintf( + `curl -sfL %s | %s%sK3S_TOKEN=%s %s sh -s - server%s`, + o.installURL(), t.Client.SudoPrefix(), env, shellQuote(token), o.release(), args, + ) +} + +func (o Options) agentInstallCmd(t Target, env, token, ip string) string { + join := fmt.Sprintf("K3S_URL=https://%s:6443 K3S_TOKEN=%s ", ip, shellQuote(token)) + if o.offline() { + return offlineInstallCmd(t.Client.SudoPrefix(), join, "agent", withSpace(o.AgentExtraArgs)) + } + return fmt.Sprintf( + `curl -sfL %s | %s%s%s%s sh -s - agent%s`, + o.installURL(), t.Client.SudoPrefix(), env, join, o.release(), withSpace(o.AgentExtraArgs), + ) +} + +// waitForCloudInit blocks until a freshly booted cloud VM has finished setting +// itself up. +// +// sshd answers well before cloud-init is done. In that window the machine is +// still running apt, which on a fresh Ubuntu image includes replacing +// ca-certificates — and every https download on the box fails TLS +// verification while it does. Installing k3s there fails with "curl failed to +// verify the legitimacy of the server", which reads like a network policy +// problem and is not one. The kubeadm path has it worse: its prerequisites run +// apt straight into cloud-init's dpkg lock. +// +// Seen on real DigitalOcean droplets, roughly 30 seconds after boot. +// +// A machine without cloud-init returns immediately, and a cloud-init that +// never finishes is waited on for five minutes and then left alone: an install +// that proceeds and fails with a real message beats one that hangs forever. +func waitForCloudInit(t Target, progress io.Writer) { + const script = `command -v cloud-init >/dev/null 2>&1 || exit 0 +if command -v timeout >/dev/null 2>&1; then + timeout 300 cloud-init status --wait >/dev/null 2>&1 || true +else + cloud-init status --wait >/dev/null 2>&1 || true +fi` + // A status check first, so the wait is only announced when there is + // actually something to wait for. + out, code, err := t.Client.Run(`command -v cloud-init >/dev/null 2>&1 && cloud-init status 2>/dev/null | head -1 || true`) + if err == nil && code == 0 && strings.Contains(out, "running") && progress != nil { + fmt.Fprintf(progress, "[%s] waiting for cloud-init to finish before installing...\n", t.Node.Host) + } + t.Client.Run(script) +} + // Host is what Setup needs from a connection. *ssh.Client satisfies it; tests // substitute a recorder so the install commands can be asserted without a VM. type Host interface { @@ -70,6 +199,65 @@ func Setup(servers []Target, agents []Target, opts Options) error { fmt.Fprintf(opts.Progress, format+"\n", a...) } } + nodes := append(append([]Target{}, servers...), agents...) + + // Every node is given a chance to finish booting before anything is + // installed on it. + for _, t := range nodes { + waitForCloudInit(t, opts.Progress) + } + + // A node with no route out can borrow this machine's, or be pointed at a + // mirror the operator runs. Neither is needed for an offline bundle, which + // is why both are optional. + var proxies []*nodeProxy + if opts.ViaProxy { + var err error + proxies, err = startProxies(nodes, opts.ProxyAllow, opts.Progress) + if err != nil { + return err + } + defer stopProxies(proxies) + } + if !opts.Mirror.empty() { + for _, t := range nodes { + if err := applyAptMirror(t, opts.Mirror, opts.Progress); err != nil { + return err + } + } + } + + // An offline install has to put the bundle on every node before anything + // is installed anywhere. Doing it per node as we go would leave a cluster + // half built when the last node turns out to be missing a file that was + // never in the bundle to begin with. + if opts.offline() { + m, err := bundle.Load(opts.BundleDir) + if err != nil { + return err + } + progressf("offline install from %s (k3s %s, %s)", opts.BundleDir, m.Version, m.Arch) + all := nodes + // Every node's architecture is checked before any node is uploaded to. + // Finding the mismatch on the last one, after two full transfers, is + // the same waste this check exists to avoid. + for _, t := range all { + if err := checkArch(t, m.Arch); err != nil { + return err + } + } + for _, t := range all { + if err := stageBundle(t, opts.BundleDir, m, opts.Progress); err != nil { + return err + } + } + // The staged copies are only needed until the installer has run. + defer func() { + for _, t := range all { + cleanStage(t) + } + }() + } if len(servers) == 0 { return fmt.Errorf("at least one server is required") } @@ -96,12 +284,14 @@ func Setup(servers []Target, agents []Target, opts Options) error { if opts.Token != "" { tokenArg = " K3S_TOKEN=" + shellQuote(opts.Token) } - cmd := fmt.Sprintf( - `curl -sfL %s | %sINSTALL_K3S_CHANNEL=%s%s sh -s - server%s`, - opts.installURL(), first.Client.SudoPrefix(), opts.channel(), tokenArg, withSpace(initArgs), - ) + // The environment goes on both sides of the pipe. Outside sudo it reaches + // curl, which fetches the installer; inside it reaches the installer + // itself, which fetches the k3s binary — sudo resets the environment, so + // the outer copy alone leaves the larger download unproxied. + env := proxyEnv(proxies, first.Node.Host) + cmd := env + opts.serverInstallCmd(first, env, tokenArg, initArgs) if code, err := streamSudo(first.Client, cmd, opts.Progress); err != nil || code != 0 { - return fmt.Errorf("server install failed (exit %d): %w", code, err) + return installFailure("server install", code, err) } // 2. the join token and the address the others will reach it on @@ -110,21 +300,19 @@ func Setup(servers []Target, agents []Target, opts Options) error { if err != nil { return fmt.Errorf("fetch token: %w", err) } - ip, err := serverInternalIP(first.Client) + ip, err := opts.joinAddress(first, first.Client) if err != nil { - return fmt.Errorf("fetch server IP: %w", err) + return fmt.Errorf("determine the address the other nodes join on: %w", err) } + progressf("[%s] other nodes will join at https://%s:6443", first.Node.Host, ip) // 3. remaining servers join the etcd cluster for _, s := range servers[1:] { progressf("[%s] joining as server (etcd member)...", s.Node.Host) - joinCmd := fmt.Sprintf( - `curl -sfL %s | %sK3S_TOKEN=%s INSTALL_K3S_CHANNEL=%s sh -s - server --server https://%s:6443%s`, - opts.installURL(), s.Client.SudoPrefix(), shellQuote(token), opts.channel(), ip, - withSpace(opts.ServerExtraArgs), - ) + env := proxyEnv(proxies, s.Node.Host) + joinCmd := env + opts.joinServerCmd(s, env, token, ip) if code, err := streamSudo(s.Client, joinCmd, opts.Progress); err != nil || code != 0 { - return fmt.Errorf("server %s join failed (exit %d): %w", s.Node.Host, code, err) + return installFailure("server "+s.Node.Host+" join", code, err) } // Servers are added one at a time on purpose: etcd learners join // sequentially, and adding several at once can cost quorum. @@ -136,13 +324,10 @@ func Setup(servers []Target, agents []Target, opts Options) error { // 4. agents for _, a := range agents { progressf("[%s] installing k3s agent...", a.Node.Host) - joinCmd := fmt.Sprintf( - `curl -sfL %s | %sK3S_URL=https://%s:6443 K3S_TOKEN=%s INSTALL_K3S_CHANNEL=%s sh -s - agent%s`, - opts.installURL(), a.Client.SudoPrefix(), ip, shellQuote(token), opts.channel(), - withSpace(opts.AgentExtraArgs), - ) + env := proxyEnv(proxies, a.Node.Host) + joinCmd := env + opts.agentInstallCmd(a, env, token, ip) if code, err := streamSudo(a.Client, joinCmd, opts.Progress); err != nil || code != 0 { - return fmt.Errorf("agent %s install failed (exit %d): %w", a.Node.Host, code, err) + return installFailure("agent "+a.Node.Host+" install", code, err) } } @@ -207,6 +392,37 @@ func serverInternalIP(server Runner) (string, error) { return ip, nil } +// joinAddress is the address agents and joining servers dial to reach the +// first server. +// +// It is the address from the targets file, not one discovered on the node. +// The operator chose it and k3helper has just proved it works by connecting +// over it, whereas `hostname -I` returns whatever the machine lists first — +// which on a cloud VM is the public address. That address is often the one +// thing the cluster's own network cannot use: an air-gapped node with egress +// blocked can reach its neighbour's private IP and nothing else, so the agents +// sat retrying "failed to get CA certs" against a public IP forever while the +// server ran perfectly well beside them. +// +// Discovery remains the fallback for a local node, which has no host address +// to take. +func (o Options) joinAddress(first Target, client Runner) (string, error) { + return resolveJoinAddress(o.JoinAddress, first, client) +} + +// resolveJoinAddress is shared by the k3s and kubeadm paths, which have the +// same problem and had better not answer it two different ways. +func resolveJoinAddress(override string, first Target, client Runner) (string, error) { + if override != "" { + return override, nil + } + h := strings.TrimSpace(first.Node.Host) + if h != "" && !first.Node.Local && h != "localhost" && h != "127.0.0.1" { + return h, nil + } + return serverInternalIP(client) +} + // waitReady polls until `expected` nodes have registered and all are Ready. // An expected of 0 means "however many are registered, all must be Ready", // which is what the pause between etcd members joining needs. diff --git a/internal/vm/setup_test.go b/internal/vm/setup_test.go index 820938b..c190662 100644 --- a/internal/vm/setup_test.go +++ b/internal/vm/setup_test.go @@ -1,7 +1,16 @@ package vm import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" "io" + "net" + "os" + "path/filepath" + + "github.com/solutionforest/k3helper/internal/bundle" "github.com/solutionforest/k3helper/internal/ssh" "strings" @@ -239,6 +248,9 @@ func (f fakeHost) Run(cmd string) (string, int, error) { case strings.Contains(cmd, "get nodes"): // enough Ready nodes to satisfy any arrangement these tests build return "a=True\nb=True\nc=True\nd=True\n", 0, nil + case strings.Contains(cmd, "print-join-command"): + return "kubeadm join 10.0.0.1:6443 --token abcdef.0123456789abcdef " + + "--discovery-token-ca-cert-hash sha256:deadbeef\n", 0, nil } return "", 0, nil } @@ -328,14 +340,16 @@ func TestHAServersInitialiseThenJoin(t *testing.T) { if strings.Contains(line, "--cluster-init") { t.Errorf("%s must join, not re-initialise: %s", name, line) } - if !strings.Contains(line, "--server https://10.0.0.1:6443") { + // The join address is the first server's host from the targets file, + // not an address discovered on the node. + if !strings.Contains(line, "--server https://s1:6443") { t.Errorf("%s did not join the first server: %s", name, line) } if !strings.Contains(line, "K3S_TOKEN='K10secret::server:node'") { t.Errorf("%s joined without the quoted token: %s", name, line) } } - if !strings.Contains(a1, "K3S_URL=https://10.0.0.1:6443") || strings.Contains(a1, "--server ") { + if !strings.Contains(a1, "K3S_URL=https://s1:6443") || strings.Contains(a1, "--server ") { t.Errorf("the agent should join with K3S_URL, not --server: %s", a1) } } @@ -403,7 +417,7 @@ func TestKubeadmRefusesMultipleServers(t *testing.T) { // The prerequisite script must do the things kubeadm requires and does not do // itself; each omission fails much later and confusingly. func TestKubeadmPrereqScriptCoversTheRequirements(t *testing.T) { - script := kubeadmPrereqScript("v1.31") + script := kubeadmPrereqScript("v1.31", AptMirror{}) for _, want := range []string{ "swapoff -a", // kubelet refuses to start with swap on "br_netfilter", // pod traffic must be visible to iptables @@ -423,7 +437,7 @@ func TestKubeadmPrereqScriptCoversTheRequirements(t *testing.T) { } } // The version must be threaded through, not hardcoded. - if strings.Contains(kubeadmPrereqScript("v1.30"), "stable:/v1.31") { + if strings.Contains(kubeadmPrereqScript("v1.30", AptMirror{}), "stable:/v1.31") { t.Error("the requested version was ignored") } } @@ -494,3 +508,518 @@ func TestKubeadmPrereqRunsAsRoot(t *testing.T) { t.Errorf("script is not fed to a shell: %s", prep) } } + +// A pinned version must skip the channel entirely. Live testing found +// update.k3s.io serving a Traefik default certificate from every one of its +// addresses, which broke `curl -sfL https://get.k3s.io | sh -` worldwide — +// pinning is the way past an outage in a service we do not control. +func TestVersionPinSkipsTheChannel(t *testing.T) { + var rec []string + if err := Setup(fakeTargets(&rec, "s1"), nil, Options{Version: "v1.31.2+k3s1"}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "INSTALL_K3S_VERSION='v1.31.2+k3s1'") { + t.Errorf("version not pinned:\n%s", all) + } + if strings.Contains(all, "INSTALL_K3S_CHANNEL") { + t.Errorf("a pinned version still resolved a channel:\n%s", all) + } +} + +// Without a pin the channel is still used, so existing behaviour is unchanged. +func TestChannelUsedWhenNoVersionPinned(t *testing.T) { + var rec []string + if err := Setup(fakeTargets(&rec, "s1"), nil, Options{}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "INSTALL_K3S_CHANNEL=stable") { + t.Errorf("channel missing:\n%s", all) + } +} + +// The agents and joining servers must honour the pin too, or a cluster ends up +// running two different k3s builds. +func TestVersionPinReachesAgents(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "s1") + agents := fakeTargets(&rec, "a1") + if err := Setup(servers, agents, Options{Version: "v1.31.2+k3s1"}); err != nil { + t.Fatalf("setup: %v", err) + } + for _, line := range rec { + if strings.Contains(line, "sh -s - agent") && !strings.Contains(line, "INSTALL_K3S_VERSION=") { + t.Errorf("agent install did not carry the pinned version: %s", line) + } + } +} + +// A command that ran and exited non-zero has no error to wrap. Passing nil to +// %w printed "%!w()" as the last thing an operator saw when an install +// failed on a real VM. +func TestInstallFailureMessageHasNoFormatVerbLeak(t *testing.T) { + got := installFailure("server install", 1, nil).Error() + if strings.Contains(got, "%!") { + t.Errorf("format verb leaked into the message: %s", got) + } + if !strings.Contains(got, "exit 1") { + t.Errorf("exit code missing from: %s", got) + } + wrapped := installFailure("server install", 0, errors.New("connection reset")) + if !strings.Contains(wrapped.Error(), "connection reset") { + t.Errorf("underlying error lost: %s", wrapped) + } +} + +// The address the other nodes join on comes from the targets file, because +// that is the address the operator chose and the one k3helper has just proved +// works by connecting over it. +// +// Live testing is what turned this up: `hostname -I` returns a cloud VM's +// public address first, and on air-gapped nodes with egress blocked that is +// the one address the cluster cannot use. The agents retried "failed to get CA +// certs" against it indefinitely while the server ran fine next to them. +func TestJoinAddressComesFromTheTargetsFile(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "10.104.0.13") + agents := fakeTargets(&rec, "10.104.0.12") + if err := Setup(servers, agents, Options{}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "K3S_URL=https://10.104.0.13:6443") { + t.Errorf("the agent did not join on the server's configured address:\n%s", all) + } + // 10.0.0.1 is what the fake `hostname -I` reports. + if strings.Contains(all, "https://10.0.0.1:6443") { + t.Errorf("the join address was discovered on the node instead:\n%s", all) + } +} + +// --join-address covers the split case: k3helper reaches the nodes over one +// network and the cluster talks over another. +func TestJoinAddressOverride(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "203.0.113.10") + agents := fakeTargets(&rec, "203.0.113.11") + if err := Setup(servers, agents, Options{JoinAddress: "10.104.0.13"}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "K3S_URL=https://10.104.0.13:6443") { + t.Errorf("the override was ignored:\n%s", all) + } + if strings.Contains(all, "K3S_URL=https://203.0.113.10:6443") { + t.Errorf("the public address was used despite the override:\n%s", all) + } +} + +// A local node has no host address to take, so discovery is still the fallback. +func TestJoinAddressFallsBackToDiscoveryForLocalNode(t *testing.T) { + var rec []string + servers := []Target{{Node: ssh.Node{Host: "localhost", Local: true}, Client: fakeHost{name: "s1", rec: &rec}}} + agents := fakeTargets(&rec, "10.104.0.12") + if err := Setup(servers, agents, Options{}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "K3S_URL=https://10.0.0.1:6443") { + t.Errorf("a local server did not fall back to a discovered address:\n%s", all) + } +} + +// A fresh cloud VM answers sshd before cloud-init has finished with it, and in +// that window apt is still replacing ca-certificates — so every https download +// on the box fails TLS verification. Installing into that window produced +// "curl failed to verify the legitimacy of the server" on real DigitalOcean +// droplets, which reads like a firewall problem and is not one. +func TestSetupWaitsForCloudInitBeforeInstalling(t *testing.T) { + var rec []string + servers := fakeTargets(&rec, "s1") + agents := fakeTargets(&rec, "a1") + if err := Setup(servers, agents, Options{}); err != nil { + t.Fatalf("setup: %v", err) + } + firstInstall, cloudInit := -1, -1 + for i, c := range rec { + if cloudInit < 0 && strings.Contains(c, "cloud-init status --wait") { + cloudInit = i + } + if firstInstall < 0 && strings.Contains(c, "get.k3s.io") { + firstInstall = i + } + } + if cloudInit < 0 { + t.Fatal("cloud-init was never waited on") + } + if firstInstall < 0 { + t.Fatal("nothing was installed") + } + if cloudInit > firstInstall { + t.Errorf("waited for cloud-init at step %d, after installing at step %d", cloudInit, firstInstall) + } +} + +// kubeadm's prerequisites run apt, which collides with cloud-init's dpkg lock +// on a machine that has only just booted. +func TestKubeadmWaitsForCloudInitBeforeApt(t *testing.T) { + var rec []string + if err := SetupKubeadm(fakeTargets(&rec, "s1"), nil, KubeadmOptions{}); err != nil { + t.Fatalf("SetupKubeadm: %v", err) + } + cloudInit, prereq := -1, -1 + for i, c := range rec { + if cloudInit < 0 && strings.Contains(c, "cloud-init status --wait") { + cloudInit = i + } + if prereq < 0 && strings.Contains(c, "base64 -d") { + prereq = i + } + } + if cloudInit < 0 { + t.Fatal("cloud-init was never waited on") + } + if prereq >= 0 && cloudInit > prereq { + t.Errorf("ran the prerequisites at step %d before waiting for cloud-init at step %d", prereq, cloudInit) + } +} + +// A machine without cloud-init must not be held up by the check. +func TestCloudInitWaitIsGuarded(t *testing.T) { + var rec []string + if err := Setup(fakeTargets(&rec, "s1"), nil, Options{}); err != nil { + t.Fatalf("setup: %v", err) + } + var wait string + for _, c := range rec { + if strings.Contains(c, "cloud-init status --wait") { + wait = c + } + } + if !strings.Contains(wait, "command -v cloud-init") { + t.Errorf("the wait is not guarded by a presence check: %s", wait) + } + if !strings.Contains(wait, "timeout 300") { + t.Errorf("a stuck cloud-init would hang the install forever: %s", wait) + } +} + +// uploadHost is a fakeHost that can also receive files, so the offline path +// can be driven without a VM. +type uploadHost struct { + fakeHost + uploaded map[string]int64 + uname string + sums map[string]string +} + +func (u *uploadHost) Run(cmd string) (string, int, error) { + if strings.Contains(cmd, "uname -m") { + return u.uname + "\n", 0, nil + } + if strings.Contains(cmd, "sha256sum") { + // Match the exact staged path, not a substring: "k3s" is also a + // substring of "k3s-airgap-images.tar.zst", so a looser match hands + // back the binary's hash for the image archive. + for name, sum := range u.sums { + if strings.Contains(cmd, "'"+remoteStage+"/"+name+"'") { + return sum + "\n", 0, nil + } + } + return "", 0, nil + } + return u.fakeHost.Run(cmd) +} + +func (u *uploadHost) WriteFileFrom(path string, r io.Reader, mode os.FileMode, progress func(int64)) error { + n, err := io.Copy(io.Discard, r) + if u.uploaded == nil { + u.uploaded = map[string]int64{} + } + u.uploaded[path] = n + return err +} + +// writeBundle creates a bundle on disk for the offline path to install from. +func writeBundle(t *testing.T, arch string) (string, *bundle.Manifest) { + t.Helper() + dir := t.TempDir() + files := map[string]string{ + bundle.BinaryName: "#!/bin/false\n", + bundle.ImagesName: "not really a tarball", + bundle.InstallName: "#!/bin/sh\n", + } + m := &bundle.Manifest{Version: "v1.31.2+k3s1", Arch: arch} + for name, body := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + sum := sha256.Sum256([]byte(body)) + switch name { + case bundle.BinaryName: + m.BinarySHA = hex.EncodeToString(sum[:]) + case bundle.ImagesName: + m.ImagesSHA = hex.EncodeToString(sum[:]) + } + } + data, _ := json.Marshal(m) + if err := os.WriteFile(filepath.Join(dir, bundle.ManifestName), data, 0o644); err != nil { + t.Fatal(err) + } + return dir, m +} + +// A bundle for the wrong architecture is refused before anything is uploaded. +// Finding out afterwards means a few hundred megabytes sent to a machine that +// was never going to run it. +func TestOfflineRefusesWrongArchBeforeUploading(t *testing.T) { + var rec []string + dir, _ := writeBundle(t, "arm64") + host := &uploadHost{fakeHost: fakeHost{name: "s1", rec: &rec}, uname: "x86_64"} + err := Setup([]Target{{Node: ssh.Node{Host: "s1"}, Client: host}}, nil, Options{BundleDir: dir}) + if err == nil { + t.Fatal("an arm64 bundle was accepted on an amd64 node") + } + if !strings.Contains(err.Error(), "arm64") || !strings.Contains(err.Error(), "amd64") { + t.Errorf("the error does not name both architectures: %v", err) + } + if len(host.uploaded) != 0 { + t.Errorf("uploaded %d file(s) before checking the architecture", len(host.uploaded)) + } +} + +// An upload that did not survive the link is caught by its hash, rather than +// becoming a cluster that installs cleanly and then will not run pods. +func TestOfflineDetectsACorruptedUpload(t *testing.T) { + var rec []string + dir, _ := writeBundle(t, "amd64") + host := &uploadHost{ + fakeHost: fakeHost{name: "s1", rec: &rec}, + uname: "x86_64", + sums: map[string]string{bundle.BinaryName: strings.Repeat("00", 32)}, + } + err := Setup([]Target{{Node: ssh.Node{Host: "s1"}, Client: host}}, nil, Options{BundleDir: dir}) + if err == nil { + t.Fatal("a corrupted upload was installed") + } + if !strings.Contains(err.Error(), "corrupted") { + t.Errorf("unhelpful error for a bad hash: %v", err) + } +} + +// The happy path uploads all three files and installs with the download +// skipped, which is the whole point: the node never reaches the internet. +func TestOfflineInstallsWithoutDownloading(t *testing.T) { + var rec []string + dir, m := writeBundle(t, "amd64") + host := &uploadHost{ + fakeHost: fakeHost{name: "s1", rec: &rec}, + uname: "x86_64", + sums: map[string]string{bundle.BinaryName: m.BinarySHA, bundle.ImagesName: m.ImagesSHA}, + } + if err := Setup([]Target{{Node: ssh.Node{Host: "s1"}, Client: host}}, nil, + Options{BundleDir: dir}); err != nil { + t.Fatalf("offline setup: %v", err) + } + if len(host.uploaded) != 3 { + t.Errorf("uploaded %d files, want 3: %v", len(host.uploaded), host.uploaded) + } + all := strings.Join(rec, "\n") + if !strings.Contains(all, "INSTALL_K3S_SKIP_DOWNLOAD=true") { + t.Errorf("the installer was not told to skip the download:\n%s", all) + } + if strings.Contains(all, "get.k3s.io") { + t.Errorf("an offline install still reached for the internet:\n%s", all) + } + // The staged copy is a few hundred megabytes on a real node. + if !strings.Contains(all, "rm -rf '"+remoteStage+"'") { + t.Errorf("the staging directory was left behind:\n%s", all) + } +} + +// A tunnelHost is a fakeHost that can also open a listener, so the --via-proxy +// path can be driven without a VM. +type tunnelHost struct { + fakeHost + l net.Listener +} + +func (h *tunnelHost) ListenRemote(addr string) (net.Listener, error) { + l, err := net.Listen("tcp", "127.0.0.1:0") + h.l = l + return l, err +} + +// --via-proxy must point apt and containerd at the tunnel, and must take both +// settings away again. A node left pointing at a proxy that no longer exists +// cannot reach its own package manager, and "apt hangs on 127.0.0.1" points +// nowhere near k3helper. +func TestViaProxyConfiguresAndCleansUp(t *testing.T) { + var rec []string + host := &tunnelHost{fakeHost: fakeHost{name: "s1", rec: &rec}} + err := Setup([]Target{{Node: ssh.Node{Host: "s1"}, Client: host}}, nil, Options{ViaProxy: true}) + if err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + for _, want := range []string{ + `Acquire::http::Proxy`, + `Acquire::https::Proxy`, + aptProxyConf, + containerdProxyConf, + `HTTP_PROXY=http://127.0.0.1:`, + } { + if !strings.Contains(all, want) { + t.Errorf("the proxy was not configured (%q missing):\n%s", want, all) + } + } + if !strings.Contains(all, "rm -f "+aptProxyConf) { + t.Errorf("the apt proxy config was left on the node:\n%s", all) + } + // The install itself must carry the environment: the k3s installer is + // fetched with curl, which reads http_proxy and nothing else. + var install string + for _, c := range rec { + if strings.Contains(c, "get.k3s.io") { + install = c + } + } + // Once for curl, which fetches the installer, and once after sudo for the + // installer itself, which fetches the k3s binary — sudo resets the + // environment, so one copy is not enough. + if n := strings.Count(install, "http_proxy=http://127.0.0.1:"); n < 2 { + t.Errorf("the proxy environment appears %d time(s); it is needed either side of sudo: %s", n, install) + } +} + +// Cluster-internal traffic must not be sent through the operator's machine. +func TestViaProxyExemptsClusterTraffic(t *testing.T) { + var rec []string + host := &tunnelHost{fakeHost: fakeHost{name: "s1", rec: &rec}} + if err := Setup([]Target{{Node: ssh.Node{Host: "s1"}, Client: host}}, nil, + Options{ViaProxy: true}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + for _, want := range []string{"10.0.0.0/8", ".svc", ".cluster.local"} { + if !strings.Contains(all, want) { + t.Errorf("NO_PROXY does not exempt %s:\n%s", want, all) + } + } +} + +// Without --via-proxy nothing is tunnelled and nothing is written. +func TestNoProxyByDefault(t *testing.T) { + var rec []string + if err := Setup(fakeTargets(&rec, "s1"), nil, Options{}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + if strings.Contains(all, "Acquire::http::Proxy") || strings.Contains(all, "HTTP_PROXY") { + t.Errorf("a proxy was configured without being asked for:\n%s", all) + } +} + +// --apt-mirror rewrites both source formats: Ubuntu 24.04 uses deb822 +// .sources files and everything older uses one-line .list entries. Missing one +// leaves half the sources pointing at an archive the node cannot reach. +func TestAptMirrorRewritesBothSourceFormats(t *testing.T) { + script := aptMirrorScript(AptMirror{URL: "https://nexus.corp/repository/ubuntu"}) + for _, want := range []string{ + "/etc/apt/sources.list.d/*.sources", + "/etc/apt/sources.list.d/*.list", + "/etc/apt/sources.list", + "k3helper.bak", + "nexus.corp/repository/ubuntu", + } { + if !strings.Contains(script, want) { + t.Errorf("the rewrite does not cover %q:\n%s", want, script) + } + } +} + +func TestAptMirrorIsOptional(t *testing.T) { + if s := aptMirrorScript(AptMirror{}); s != "" { + t.Errorf("an empty mirror produced a script:\n%s", s) + } + if !(AptMirror{}).empty() { + t.Error("an empty AptMirror does not report itself empty") + } + if (AptMirror{K8sRepo: "https://nexus.corp/k8s"}).empty() { + t.Error("a mirror with only a Kubernetes repo reports itself empty") + } +} + +// The Kubernetes packages come from their own service, mirrored separately — +// a site may have one mirror and not the other. +func TestK8sAptRepoOverride(t *testing.T) { + script := kubeadmPrereqScript("v1.31", AptMirror{K8sRepo: "https://nexus.corp/repository/k8s"}) + if !strings.Contains(script, "https://nexus.corp/repository/k8s/Release.key") { + t.Errorf("the repository key is still fetched upstream:\n%s", script) + } + if strings.Contains(script, "pkgs.k8s.io") { + t.Errorf("pkgs.k8s.io is still referenced after an override:\n%s", script) + } +} + +func TestK8sAptRepoDefaultsUpstream(t *testing.T) { + script := kubeadmPrereqScript("v1.31", AptMirror{}) + if !strings.Contains(script, "https://pkgs.k8s.io/core:/stable:/v1.31/deb/") { + t.Errorf("the default repository is wrong:\n%s", script) + } +} + +// The proxy lives inside a running k3helper. If the operator's machine goes +// away mid-install, apt has no timeout of its own: it waits on the dead tunnel +// forever, holding the apt lock. One was found still holding it 31 minutes +// later, failing every later run on that node with a lock error that pointed +// nowhere near the cause. +func TestViaProxyGivesAptATimeout(t *testing.T) { + var rec []string + host := &tunnelHost{fakeHost: fakeHost{name: "s1", rec: &rec}} + if err := Setup([]Target{{Node: ssh.Node{Host: "s1"}, Client: host}}, nil, + Options{ViaProxy: true}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + for _, want := range []string{ + `Acquire::http::Timeout`, + `Acquire::https::Timeout`, + `Acquire::http::Pipeline-Depth "0"`, + } { + if !strings.Contains(all, want) { + t.Errorf("the apt configuration is missing %s:\n%s", want, all) + } + } +} + +// Cluster traffic must never go through the tunnel. kubeadm talks to the API +// server it has just started, and with only localhost exempted it sent that +// through the proxy — which refused the node's own address, because an +// allowlist of internet hosts does not contain it, and the install failed. +func TestViaProxyExemptsTheClusterFromItself(t *testing.T) { + var rec []string + host := &tunnelHost{fakeHost: fakeHost{name: "s1", rec: &rec}} + if err := Setup([]Target{{Node: ssh.Node{Host: "10.104.0.5"}, Client: host}}, nil, + Options{ViaProxy: true}); err != nil { + t.Fatalf("setup: %v", err) + } + all := strings.Join(rec, "\n") + // The private ranges cover a node's own address without having to know it. + for _, want := range []string{"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", ".cluster.local"} { + if !strings.Contains(all, want) { + t.Errorf("NO_PROXY does not exempt %s:\n%s", want, all) + } + } + // And it has to reach the commands, not only containerd's unit file. + var install string + for _, c := range rec { + if strings.Contains(c, "get.k3s.io") { + install = c + } + } + if !strings.Contains(install, "no_proxy=") || !strings.Contains(install, "10.0.0.0/8") { + t.Errorf("the install command carries no cluster exemption: %s", install) + } +} diff --git a/internal/vm/viaproxy.go b/internal/vm/viaproxy.go new file mode 100644 index 0000000..8cfd94a --- /dev/null +++ b/internal/vm/viaproxy.go @@ -0,0 +1,240 @@ +package vm + +import ( + "fmt" + "io" + "net" + "strings" + "sync" + + "github.com/solutionforest/k3helper/internal/proxy" +) + +// This file lends a node the operator's internet connection for the length of +// an install. +// +// k3s can be installed from a bundle because it is one static binary and one +// image archive. kubeadm cannot: it needs apt packages from a distribution +// mirror and images from registry.k8s.io, and neither fits in a file you carry +// in. The machine running k3helper usually can reach both, and it already +// holds an SSH connection to every node — so the connection carries the +// traffic backwards, for as long as the install takes and no longer. +// +// This is deliberately not the default. An air-gapped network is air-gapped on +// purpose, and in some environments opening even a temporary, proxied, +// allowlisted route would be a policy breach regardless of how briefly it +// existed. It happens only when asked for by name. + +// Tunneler is a Host that can open a listener on the node. Kept apart from +// Host because only this path needs it. +type Tunneler interface { + ListenRemote(addr string) (net.Listener, error) +} + +// nodeProxy is a live tunnel to one node. +type nodeProxy struct { + host string + port int + listener net.Listener + server *proxy.Server + client Host +} + +// aptProxyConf is where the apt configuration is written. The name sorts late +// so it wins over anything already there, and says who wrote it. +const aptProxyConf = "/etc/apt/apt.conf.d/99-k3helper-proxy" + +// containerdProxyConf is a systemd drop-in, which is how containerd is given +// an environment at all: it does not read the invoking shell's. +const containerdProxyConf = "/etc/systemd/system/containerd.service.d/k3helper-proxy.conf" + +// startNodeProxy opens the tunnel and points the node's package manager and +// container runtime at it. +func startNodeProxy(t Target, allow []string, progress io.Writer) (*nodeProxy, error) { + tun, ok := t.Client.(Tunneler) + if !ok { + return nil, fmt.Errorf("node %s: this connection cannot open a tunnel", t.Node.Host) + } + // Port 0: the node picks a free one. A fixed port collides with whatever + // is already listening on a machine we do not own. + l, err := tun.ListenRemote("127.0.0.1:0") + if err != nil { + return nil, fmt.Errorf("node %s: %w", t.Node.Host, err) + } + port := 0 + if a, ok := l.Addr().(*net.TCPAddr); ok { + port = a.Port + } + if port == 0 { + l.Close() + return nil, fmt.Errorf("node %s: the tunnel did not report a port", t.Node.Host) + } + + // One line per refused host, not per refused request: apt retries, and a + // single unreachable repository otherwise fills the install log with the + // same sentence eight times. + var refusedMu sync.Mutex + refused := map[string]bool{} + srv := &proxy.Server{ + Allow: allow, + OnRefuse: func(host string) { + refusedMu.Lock() + first := !refused[host] + refused[host] = true + refusedMu.Unlock() + if first && progress != nil { + fmt.Fprintf(progress, + "[%s] proxy refused %s — not on the allowlist; add it with --proxy-allow %s if the install needs it\n", + t.Node.Host, host, host) + } + }, + } + go srv.Serve(l) + + p := &nodeProxy{host: t.Node.Host, port: port, listener: l, server: srv, client: t.Client} + if err := p.configure(); err != nil { + p.Stop() + return nil, err + } + if progress != nil { + fmt.Fprintf(progress, "[%s] lending this machine's connection on 127.0.0.1:%d for the install\n", + t.Node.Host, port) + } + return p, nil +} + +// configure writes the proxy settings the install actually reads. +func (p *nodeProxy) configure() error { + url := fmt.Sprintf("http://127.0.0.1:%d", p.port) + + // apt needs both: https goes through the same proxy with CONNECT. + // Pipelining off, and a timeout on. + // + // apt sends several requests down one connection without waiting for the + // replies, and that is the first thing to disable against any proxy it + // finds flaky — over an SSH channel, index fetches were timing out. + // + // The timeout matters more. This proxy lives inside a running k3helper: if + // the operator's machine goes away mid-install — a closed laptop, a + // dropped link, a Ctrl-C — apt has no timeout of its own and waits on the + // dead tunnel forever, holding the apt lock while it does. One was found + // still holding it after 31 minutes, which then failed every later run on + // that node with a lock error that pointed nowhere near the cause. + apt := fmt.Sprintf(`Acquire::http::Proxy "%s"; +Acquire::https::Proxy "%s"; +Acquire::http::Pipeline-Depth "0"; +Acquire::Retries "3"; +Acquire::http::Timeout "30"; +Acquire::https::Timeout "30"; +`, url, url) + if out, code, err := p.client.SudoRun(fmt.Sprintf( + `mkdir -p /etc/apt/apt.conf.d && printf %%s %s > %s`, + shellQuote(apt), aptProxyConf)); err != nil || code != 0 { + return fmt.Errorf("node %s: write apt proxy config (exit %d): %s", p.host, code, out) + } + + // containerd is a service: it reads its environment from systemd, not from + // whatever shell started the install. + drop := fmt.Sprintf(`[Service] +Environment="HTTP_PROXY=%s" +Environment="HTTPS_PROXY=%s" +Environment="NO_PROXY=%s" +`, url, url, noProxy) + if out, code, err := p.client.SudoRun(fmt.Sprintf( + `mkdir -p %s && printf %%s %s > %s && systemctl daemon-reload && systemctl restart containerd 2>/dev/null || true`, + dirOf(containerdProxyConf), shellQuote(drop), containerdProxyConf)); err != nil || code != 0 { + return fmt.Errorf("node %s: write containerd proxy config (exit %d): %s", p.host, code, out) + } + return nil +} + +// noProxy is everything that must not go through the tunnel. +// +// Cluster traffic is the point of this list. kubeadm talks to the API server +// it has just started, and with only localhost exempted it sent that through +// the proxy — which refused it, because the node's own address is not on an +// allowlist of internet hosts, and the install failed. Routing a node's +// conversation with itself through the operator's laptop would be absurd even +// if it worked. +// +// The private ranges cover the node's own addresses without having to know +// them. Go's proxy resolution understands CIDR here, and every tool in this +// install is Go. +const noProxy = "localhost,127.0.0.1,::1," + + "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16," + + ".svc,.svc.cluster.local,.cluster.local" + +// Env is the proxy environment for commands run during the install, for the +// ones that read it directly — curl fetching a repository key, kubeadm +// fetching a version marker. +func (p *nodeProxy) Env() string { + url := fmt.Sprintf("http://127.0.0.1:%d", p.port) + // Both cases: Go reads the upper-case names, curl and apt the lower-case. + return fmt.Sprintf( + `http_proxy=%s https_proxy=%s HTTP_PROXY=%s HTTPS_PROXY=%s `+ + `no_proxy=%s NO_PROXY=%s `, + url, url, url, url, shellQuote(noProxy), shellQuote(noProxy)) +} + +// Stop closes the tunnel and removes every trace of it from the node. +// +// The removal matters more than the close: a node left pointing at a proxy +// that no longer exists cannot reach its own package manager afterwards, and +// the symptom — apt hanging on 127.0.0.1 — points nowhere near k3helper. +func (p *nodeProxy) Stop() { + if p.client != nil { + p.client.SudoRun(fmt.Sprintf( + `rm -f %s %s; systemctl daemon-reload 2>/dev/null; systemctl restart containerd 2>/dev/null || true`, + aptProxyConf, containerdProxyConf)) + } + if p.server != nil { + p.server.Close() + } + if p.listener != nil { + p.listener.Close() + } +} + +// startProxies opens a tunnel to every node, unwinding the ones already open +// if a later node fails. +func startProxies(targets []Target, allow []string, progress io.Writer) ([]*nodeProxy, error) { + var out []*nodeProxy + for _, t := range targets { + p, err := startNodeProxy(t, allow, progress) + if err != nil { + stopProxies(out) + return nil, err + } + out = append(out, p) + } + return out, nil +} + +func stopProxies(ps []*nodeProxy) { + for _, p := range ps { + p.Stop() + } +} + +func dirOf(path string) string { + if i := strings.LastIndex(path, "/"); i > 0 { + return path[:i] + } + return "/" +} + +// proxyEnv is the environment prefix for a command on one node, empty when no +// tunnel is open for it. +// +// The prefix is needed on top of the apt and containerd configuration because +// some of the install is neither: the kubeadm prerequisites fetch a repository +// key with curl, which reads http_proxy from its own environment and nothing +// else. +func proxyEnv(ps []*nodeProxy, host string) string { + for _, p := range ps { + if p.host == host { + return p.Env() + } + } + return "" +} diff --git a/internal/web/page.go b/internal/web/page.go new file mode 100644 index 0000000..ff5409c --- /dev/null +++ b/internal/web/page.go @@ -0,0 +1,85 @@ +package web + +// The page shell. It is a Go string rather than an embedded file only because +// the token has to be templated into it — everything else the browser needs +// lives in ui/ and is served as static assets. +const pageHTML = ` + + + + +{{.Cluster}} — k3helper + + + +
+
☸ k3helper
+
+ {{.Cluster}} + {{.Reached}} +
+
+
+ + + {{.Version}} +
+ + + +
+
+
+
+
+
+
+
+ + + +
read-only · nothing here changes the cluster
+ + + +` + +// unauthorisedHTML is what a browser gets without the token. It explains +// rather than just refusing: the usual way to arrive here is a bookmark from a +// previous run, whose token died with that process. +const unauthorisedHTML = ` + +k3helper + +
+

This page needs the token k3helper printed when it started.

+

Each run makes a new one, so a bookmark from a previous session will not + work. Look for the line beginning http://127.0.0.1: in the + terminal where k3helper serve is running, and open that.

+
+` diff --git a/internal/web/server.go b/internal/web/server.go new file mode 100644 index 0000000..4fb3664 --- /dev/null +++ b/internal/web/server.go @@ -0,0 +1,566 @@ +// Package web serves the browser interface. +// +// It is the same binary, the same data layer and the same transports as the +// CLI and the TUI: `serve` opens a connection to the cluster exactly as +// `doctor` does, and the HTTP handlers below are thin wrappers over the same +// functions. That is the point of it — a second way to look at a cluster, not +// a second implementation of looking at one. +// +// The assets are embedded, so this stays one file to copy onto a machine, and +// the same file runs on Windows, macOS, Linux and in a container. +// +// It is read-only. Everything here lists, describes or tails; nothing applies, +// deletes or restarts. A browser interface that can change a production +// cluster deserves rather more thought about who is holding it than a +// localhost token provides, and none of that is needed to make the cluster +// legible — which is what this is for. +package web + +import ( + "embed" + "encoding/json" + "fmt" + "html/template" + "io/fs" + "log" + "net/http" + "strconv" + "strings" + "sync" + "time" + + "github.com/solutionforest/k3helper/internal/check" + "github.com/solutionforest/k3helper/internal/config" + "github.com/solutionforest/k3helper/internal/kube" + "github.com/solutionforest/k3helper/internal/ssh" + "github.com/solutionforest/k3helper/internal/transport" + "github.com/solutionforest/k3helper/internal/troubleshoot" +) + +//go:embed ui +var uiFS embed.FS + +// Server holds what every request needs. +type Server struct { + // Targets is the cluster being served. + Targets *config.Targets + // Token must accompany every request. Generated per launch by the caller. + Token string + // Version is shown in the header. + Version string + + // cache holds the last successful answer per endpoint, so a browser + // polling four panels does not open four SSH connections a second. + cache sync.Map +} + +// cached is one memoised answer. +type cached struct { + at time.Time + body any + err error +} + +// freshness is how long an answer is reused. +// +// A cluster does not change meaningfully inside two seconds, and every read +// here costs an SSH round trip to a machine that may be on the other side of +// the world — the live tests ran against Singapore from Hong Kong, where a +// pod list took the best part of a second. Without this, four panels +// refreshing independently would keep a connection permanently busy. +const freshness = 2 * time.Second + +// Handler builds the router. +func (s *Server) Handler() http.Handler { + mux := http.NewServeMux() + + sub, err := fs.Sub(uiFS, "ui") + if err != nil { + panic("web: embedded assets are missing: " + err.Error()) + } + assets := http.FileServer(http.FS(sub)) + + mux.HandleFunc("/", s.page) + mux.Handle("/static/", http.StripPrefix("/static/", assets)) + + mux.HandleFunc("/api/summary", s.guard(s.summary)) + mux.HandleFunc("/api/pods", s.guard(s.pods)) + mux.HandleFunc("/api/nodes", s.guard(s.nodes)) + mux.HandleFunc("/api/events", s.guard(s.events)) + mux.HandleFunc("/api/workloads", s.guard(s.workloads)) + mux.HandleFunc("/api/doctor", s.guard(s.doctor)) + mux.HandleFunc("/api/checks", s.guard(s.checks)) + mux.HandleFunc("/api/logs", s.guard(s.logs)) + mux.HandleFunc("/api/describe", s.guard(s.describe)) + + return mux +} + +// guard rejects anything without the session token and refuses to be framed. +// +// The token is the whole of the authentication, which is why the server binds +// to loopback unless told otherwise: it is a key for the person who started +// the process, not a login for a service on a network. +func (s *Server) guard(h http.HandlerFunc) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + http.Error(w, "read-only", http.StatusMethodNotAllowed) + return + } + if !s.authorised(r) { + http.Error(w, "unauthorised: this page needs the token k3helper printed when it started", + http.StatusUnauthorized) + return + } + w.Header().Set("Cache-Control", "no-store") + h(w, r) + } +} + +func (s *Server) authorised(r *http.Request) bool { + if s.Token == "" { + return false + } + if t := r.Header.Get("X-K3helper-Token"); t == s.Token { + return true + } + return r.URL.Query().Get("token") == s.Token +} + +// --- pages ------------------------------------------------------------------- + +var pageTmpl = template.Must(template.New("page").Parse(pageHTML)) + +func (s *Server) page(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/" { + http.NotFound(w, r) + return + } + if !s.authorised(r) { + w.WriteHeader(http.StatusUnauthorized) + fmt.Fprint(w, unauthorisedHTML) + return + } + // The token reaches the page in its HTML rather than staying in the URL, + // so it is not carried into the Referer of anything the page loads. + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.Header().Set("Referrer-Policy", "no-referrer") + w.Header().Set("X-Frame-Options", "DENY") + w.Header().Set("Content-Security-Policy", + "default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self'; img-src 'self' data:") + pageTmpl.Execute(w, map[string]string{ + "Token": s.Token, + "Cluster": s.Targets.Cluster, + "Reached": s.Targets.Mode().String(), + "Version": s.Version, + }) +} + +// --- api --------------------------------------------------------------------- + +// withCluster opens a connection, runs f, and closes it. +// +// One connection per request rather than one held open: `doctor` works the +// same way, and for the same reason. A connection kept from startup reports +// the cluster as it was when the connection was made, and a node going away is +// one of the things worth seeing. +func (s *Server) withCluster(f func(transport.Cluster) (any, error)) (any, error) { + c, err := transport.Server(s.Targets) + if err != nil { + return nil, err + } + defer c.Close() + return f(c) +} + +// memo runs f at most once per freshness window. +func (s *Server) memo(key string, f func() (any, error)) (any, error) { + if v, ok := s.cache.Load(key); ok { + e := v.(cached) + if time.Since(e.at) < freshness { + return e.body, e.err + } + } + body, err := f() + s.cache.Store(key, cached{at: time.Now(), body: body, err: err}) + return body, err +} + +func writeJSON(w http.ResponseWriter, body any, err error) { + w.Header().Set("Content-Type", "application/json") + if err != nil { + w.WriteHeader(http.StatusBadGateway) + json.NewEncoder(w).Encode(map[string]string{"error": err.Error()}) + return + } + json.NewEncoder(w).Encode(body) +} + +type podRow struct { + Namespace string `json:"namespace"` + Name string `json:"name"` + Ready string `json:"ready"` + Status string `json:"status"` + Restarts int `json:"restarts"` + Node string `json:"node"` + Age string `json:"age"` + Healthy bool `json:"healthy"` +} + +func (s *Server) pods(w http.ResponseWriter, r *http.Request) { + ns := r.URL.Query().Get("ns") + body, err := s.memo("pods:"+ns, func() (any, error) { + return s.withCluster(func(c transport.Cluster) (any, error) { + pods, err := kube.ListPods(c, ns) + if err != nil { + return nil, err + } + out := make([]podRow, 0, len(pods)) + for _, p := range pods { + out = append(out, podRow{ + Namespace: p.Namespace, Name: p.Name, Ready: p.Ready, + Status: p.Status, Restarts: p.Restarts, Node: p.Node, + Age: kube.ShortAge(p.Age), Healthy: podHealthy(p), + }) + } + return out, nil + }) + }) + writeJSON(w, body, err) +} + +// podHealthy is what colours a row. A pod is healthy when it is running with +// every container ready, or has finished successfully — anything else is worth +// the operator's eye, which is the whole reason for a colour. +func podHealthy(p kube.Pod) bool { + if p.Status == "Succeeded" || p.Status == "Completed" { + return true + } + if p.Status != "Running" { + return false + } + a, b, ok := strings.Cut(p.Ready, "/") + return ok && a == b +} + +type nodeRow struct { + Name string `json:"name"` + Status string `json:"status"` + Roles string `json:"roles"` + Version string `json:"version"` + Age string `json:"age"` + IP string `json:"ip"` + OS string `json:"os"` + Ready bool `json:"ready"` +} + +func (s *Server) nodes(w http.ResponseWriter, r *http.Request) { + body, err := s.memo("nodes", func() (any, error) { + return s.withCluster(func(c transport.Cluster) (any, error) { + ns, err := kube.ListNodes(c) + if err != nil { + return nil, err + } + out := make([]nodeRow, 0, len(ns)) + for _, n := range ns { + out = append(out, nodeRow{ + Name: n.Name, Status: n.Status, Roles: n.Roles, Version: n.Version, + Age: kube.ShortAge(n.Age), IP: n.InternalIP, OS: n.OSImage, + Ready: n.Status == "Ready", + }) + } + return out, nil + }) + }) + writeJSON(w, body, err) +} + +type eventRow struct { + Namespace string `json:"namespace"` + Type string `json:"type"` + Reason string `json:"reason"` + Object string `json:"object"` + Message string `json:"message"` + Age string `json:"age"` + Count int `json:"count"` + Warning bool `json:"warning"` +} + +func (s *Server) events(w http.ResponseWriter, r *http.Request) { + ns := r.URL.Query().Get("ns") + body, err := s.memo("events:"+ns, func() (any, error) { + return s.withCluster(func(c transport.Cluster) (any, error) { + evs, err := kube.ListEvents(c, ns) + if err != nil { + return nil, err + } + out := make([]eventRow, 0, len(evs)) + for _, e := range evs { + out = append(out, eventRow{ + Namespace: e.Namespace, Type: e.Type, Reason: e.Reason, + Object: e.Object, Message: e.Message, Age: kube.ShortAge(e.Age), + Count: e.Count, Warning: e.Type == "Warning", + }) + } + return out, nil + }) + }) + writeJSON(w, body, err) +} + +type workloadRow struct { + Namespace string `json:"namespace"` + Kind string `json:"kind"` + Name string `json:"name"` + Ready string `json:"ready"` + Age string `json:"age"` + Images string `json:"images"` + Healthy bool `json:"healthy"` +} + +func (s *Server) workloads(w http.ResponseWriter, r *http.Request) { + ns := r.URL.Query().Get("ns") + body, err := s.memo("workloads:"+ns, func() (any, error) { + return s.withCluster(func(c transport.Cluster) (any, error) { + var out []workloadRow + for _, kind := range []string{"deployments", "statefulsets", "daemonsets"} { + ws, err := kube.ListWorkloads(c, kind, ns) + if err != nil { + // One kind failing is not a reason to show none: a cluster + // with no statefulsets should still list its deployments. + continue + } + for _, wl := range ws { + out = append(out, workloadRow{ + Namespace: wl.Namespace, Kind: wl.Kind, Name: wl.Name, + Ready: wl.Ready, Age: kube.ShortAge(wl.Age), Images: wl.Images, + Healthy: readyAll(wl.Ready), + }) + } + } + return out, nil + }) + }) + writeJSON(w, body, err) +} + +func readyAll(ratio string) bool { + a, b, ok := strings.Cut(ratio, "/") + return ok && a == b && a != "0" +} + +type finding struct { + ID string `json:"id"` + Title string `json:"title"` + Confidence int `json:"confidence"` + Remediation string `json:"remediation"` + Informational bool `json:"informational"` +} + +type doctorReport struct { + Findings []finding `json:"findings"` + Unreachable []string `json:"unreachable"` + ProbeErrors map[string]string `json:"probe_errors"` + Healthy bool `json:"healthy"` +} + +func (s *Server) doctor(w http.ResponseWriter, r *http.Request) { + // A diagnosis is the most expensive thing here — it opens a connection per + // node and runs a dozen probes on each — so it is cached for longer than a + // list. Nobody needs a fresh one every two seconds. + body, err := s.memoFor("doctor", 15*time.Second, func() (any, error) { + c, err := transport.Server(s.Targets) + if err != nil { + return nil, err + } + defer c.Close() + hosts, failures, closeHosts := transport.Hosts(s.Targets, c, transport.ServerName(s.Targets)) + defer closeHosts() + + var unreachable []troubleshoot.UnreachableNode + var names []string + for _, f := range failures { + unreachable = append(unreachable, troubleshoot.UnreachableNode{Name: f.Name, Reason: f.Reason}) + names = append(names, f.Name+": "+f.Reason) + } + var serverNames []string + for _, n := range s.Targets.Servers() { + serverNames = append(serverNames, n.Name) + } + ev := troubleshoot.Gatherer{ + Server: c, Hosts: hosts, Unreachable: unreachable, ServerNodes: serverNames, + NoHostLayer: s.Targets.Mode() == config.ModeKubeconfig, + }.Collect() + + ds := troubleshoot.Diagnose(ev) + out := doctorReport{ + Unreachable: names, + ProbeErrors: ev.ProbeErrors, + Healthy: troubleshoot.OnlyInformational(ds), + Findings: make([]finding, 0, len(ds)), + } + for _, d := range ds { + out.Findings = append(out.Findings, finding{ + ID: d.SignatureID, Title: d.Title, Confidence: d.Confidence, + Remediation: d.Remediation, Informational: d.Informational(), + }) + } + return out, nil + }) + writeJSON(w, body, err) +} + +// memoFor is memo with an explicit window. +func (s *Server) memoFor(key string, window time.Duration, f func() (any, error)) (any, error) { + if v, ok := s.cache.Load(key); ok { + e := v.(cached) + if time.Since(e.at) < window { + return e.body, e.err + } + } + body, err := f() + s.cache.Store(key, cached{at: time.Now(), body: body, err: err}) + return body, err +} + +type checkRow struct { + Node string `json:"node"` + Role string `json:"role"` + ID string `json:"id"` + Name string `json:"name"` + Status string `json:"status"` + Summary string `json:"summary"` + Remediation string `json:"remediation"` +} + +func (s *Server) checks(w http.ResponseWriter, r *http.Request) { + body, err := s.memoFor("checks", 15*time.Second, func() (any, error) { + var out []checkRow + if s.Targets.Mode() == config.ModeKubeconfig { + // The host layer is not missing, it is not there: say so with the + // same words the CLI uses rather than showing an empty panel. + for _, res := range check.SkippedHostResults("server") { + out = append(out, checkRow{ + Node: transport.ServerName(s.Targets), ID: res.ID, Name: res.Name, + Status: res.Status.String(), Summary: res.Summary, + }) + } + return out, nil + } + for _, node := range s.Targets.Nodes { + client, err := ssh.Dial(node.SSH()) + if err != nil { + out = append(out, checkRow{ + Node: node.Name, Role: node.Role, ID: "ssh.connect", + Name: "SSH connectivity", Status: "FAIL", + Summary: "unreachable: " + err.Error(), + }) + continue + } + runner := check.NewRunner(check.HostChecks(node.Role)...) + for _, res := range runner.RunAll(check.Context{Exec: client, Node: node.Name}) { + out = append(out, checkRow{ + Node: node.Name, Role: node.Role, ID: res.ID, Name: res.Name, + Status: res.Status.String(), Summary: res.Summary, + Remediation: res.Remediation, + }) + } + client.Close() + } + return out, nil + }) + writeJSON(w, body, err) +} + +type summary struct { + Cluster string `json:"cluster"` + Reached string `json:"reached"` + Nodes int `json:"nodes"` + NodesReady int `json:"nodes_ready"` + Pods int `json:"pods"` + PodsIssue int `json:"pods_issue"` + Version string `json:"version"` + Error string `json:"error,omitempty"` +} + +func (s *Server) summary(w http.ResponseWriter, r *http.Request) { + body, err := s.memo("summary", func() (any, error) { + out := summary{ + Cluster: s.Targets.Cluster, + Reached: s.Targets.Mode().String(), + Version: s.Version, + } + _, err := s.withCluster(func(c transport.Cluster) (any, error) { + ns, err := kube.ListNodes(c) + if err != nil { + return nil, err + } + out.Nodes = len(ns) + for _, n := range ns { + if n.Status == "Ready" { + out.NodesReady++ + } + } + pods, err := kube.ListPods(c, "") + if err != nil { + return nil, err + } + out.Pods = len(pods) + for _, p := range pods { + if !podHealthy(p) { + out.PodsIssue++ + } + } + return nil, nil + }) + if err != nil { + // Reported in the body rather than as a failed request: the header + // should still say which cluster could not be reached. + out.Error = err.Error() + } + return out, nil + }) + writeJSON(w, body, err) +} + +func (s *Server) logs(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query() + ns, pod := q.Get("ns"), q.Get("pod") + if ns == "" || pod == "" { + http.Error(w, "ns and pod are required", http.StatusBadRequest) + return + } + tail := 200 + if n, err := strconv.Atoi(q.Get("tail")); err == nil && n > 0 && n <= 5000 { + tail = n + } + body, err := s.withCluster(func(c transport.Cluster) (any, error) { + out, err := kube.Logs(c, ns, pod, tail, q.Get("previous") == "1") + if err != nil { + return nil, err + } + return map[string]string{"logs": out}, nil + }) + writeJSON(w, body, err) +} + +func (s *Server) describe(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query() + kind, ns, name := q.Get("kind"), q.Get("ns"), q.Get("name") + if kind == "" || name == "" { + http.Error(w, "kind and name are required", http.StatusBadRequest) + return + } + body, err := s.withCluster(func(c transport.Cluster) (any, error) { + out, err := kube.Describe(c, kind, ns, name) + if err != nil { + return nil, err + } + return map[string]string{"text": out}, nil + }) + writeJSON(w, body, err) +} + +// Logf is the request log, quiet by default because the interesting output is +// the URL printed at startup. +func Logf(format string, a ...any) { log.Printf(format, a...) } diff --git a/internal/web/server_test.go b/internal/web/server_test.go new file mode 100644 index 0000000..7a87994 --- /dev/null +++ b/internal/web/server_test.go @@ -0,0 +1,225 @@ +package web + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/solutionforest/k3helper/internal/config" + "github.com/solutionforest/k3helper/internal/kube" +) + +func testServer() *Server { + return &Server{ + Targets: &config.Targets{Cluster: "demo", Nodes: []config.Node{ + {Name: "server", Role: "server", Host: "10.0.0.1", User: "root"}, + }}, + Token: "s3cret", + Version: "test", + } +} + +// The token is the whole of the authentication, so every path that reveals +// anything about the cluster has to want it. +func TestEveryAPIPathNeedsTheToken(t *testing.T) { + h := testServer().Handler() + for _, path := range []string{ + "/api/summary", "/api/pods", "/api/nodes", "/api/events", + "/api/workloads", "/api/doctor", "/api/checks", + "/api/logs?ns=default&pod=web", "/api/describe?kind=pod&name=web", + } { + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest("GET", path, nil)) + if w.Code != http.StatusUnauthorized { + t.Errorf("%s answered %d without a token, want 401", path, w.Code) + } + } +} + +func TestPageNeedsTheTokenAndExplainsWhy(t *testing.T) { + h := testServer().Handler() + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest("GET", "/", nil)) + if w.Code != http.StatusUnauthorized { + t.Fatalf("the page answered %d without a token", w.Code) + } + // The usual way to arrive here is a bookmark from a previous run, whose + // token died with that process. Saying so beats a bare 401. + if !strings.Contains(w.Body.String(), "printed when it started") { + t.Errorf("the refusal does not say where to find the token:\n%s", w.Body.String()) + } +} + +func TestTokenAcceptedInHeaderOrQuery(t *testing.T) { + h := testServer().Handler() + + w := httptest.NewRecorder() + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("X-K3helper-Token", "s3cret") + h.ServeHTTP(w, r) + if w.Code != http.StatusOK { + t.Errorf("header token rejected: %d", w.Code) + } + + w = httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest("GET", "/?token=s3cret", nil)) + if w.Code != http.StatusOK { + t.Errorf("query token rejected: %d", w.Code) + } +} + +// An empty token would otherwise match a request that carries none. +func TestEmptyTokenAuthorisesNobody(t *testing.T) { + s := testServer() + s.Token = "" + w := httptest.NewRecorder() + s.Handler().ServeHTTP(w, httptest.NewRequest("GET", "/api/pods", nil)) + if w.Code != http.StatusUnauthorized { + t.Errorf("an empty token authorised a request: %d", w.Code) + } +} + +// This interface is read-only, and says so by refusing anything else. It is +// the smallest way to be sure a browser tab cannot change a production +// cluster. +func TestWritesAreRefused(t *testing.T) { + h := testServer().Handler() + for _, method := range []string{"POST", "PUT", "PATCH", "DELETE"} { + w := httptest.NewRecorder() + r := httptest.NewRequest(method, "/api/pods?token=s3cret", nil) + h.ServeHTTP(w, r) + if w.Code != http.StatusMethodNotAllowed { + t.Errorf("%s answered %d, want 405", method, w.Code) + } + } +} + +// The token is in the page, so nothing the page loads may carry it away in a +// Referer, and nothing may frame it. +func TestPageSetsItsHeaders(t *testing.T) { + h := testServer().Handler() + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest("GET", "/?token=s3cret", nil)) + for header, want := range map[string]string{ + "Referrer-Policy": "no-referrer", + "X-Frame-Options": "DENY", + "Content-Security-Policy": "default-src 'none'", + } { + if got := w.Header().Get(header); !strings.Contains(got, want) { + t.Errorf("%s = %q, want it to contain %q", header, got, want) + } + } +} + +func TestAssetsAreEmbedded(t *testing.T) { + h := testServer().Handler() + for _, path := range []string{"/static/app.css", "/static/app.js"} { + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest("GET", path, nil)) + if w.Code != http.StatusOK { + t.Errorf("%s is not embedded in the binary: %d", path, w.Code) + } + if w.Body.Len() == 0 { + t.Errorf("%s is empty", path) + } + } +} + +// A pod is healthy when it is running with every container ready, or has +// finished. Anything else is worth the operator's eye, which is the whole +// reason the row is coloured. +func TestPodHealthy(t *testing.T) { + tests := []struct { + pod kube.Pod + want bool + }{ + {kube.Pod{Status: "Running", Ready: "1/1"}, true}, + {kube.Pod{Status: "Running", Ready: "3/3"}, true}, + {kube.Pod{Status: "Running", Ready: "1/2"}, false}, + {kube.Pod{Status: "Running", Ready: "0/1"}, false}, + {kube.Pod{Status: "Succeeded", Ready: "0/1"}, true}, + {kube.Pod{Status: "CrashLoopBackOff", Ready: "0/1"}, false}, + {kube.Pod{Status: "ImagePullBackOff", Ready: "0/1"}, false}, + {kube.Pod{Status: "Pending", Ready: "0/1"}, false}, + // A malformed ratio must not read as healthy. + {kube.Pod{Status: "Running", Ready: ""}, false}, + } + for _, tc := range tests { + if got := podHealthy(tc.pod); got != tc.want { + t.Errorf("podHealthy(%s %s) = %v, want %v", tc.pod.Status, tc.pod.Ready, got, tc.want) + } + } +} + +func TestReadyAll(t *testing.T) { + for ratio, want := range map[string]bool{ + "3/3": true, "1/1": true, "2/3": false, "0/0": false, "": false, "x": false, + } { + if got := readyAll(ratio); got != want { + t.Errorf("readyAll(%q) = %v, want %v", ratio, got, want) + } + } +} + +// Every read here is an SSH round trip — the live tests were talking to +// Singapore — and a browser polls several panels at once. Without the memo +// they would each open their own connection. +func TestMemoReusesAnAnswer(t *testing.T) { + s := testServer() + calls := 0 + f := func() (any, error) { calls++; return calls, nil } + + for i := 0; i < 5; i++ { + if _, err := s.memo("k", f); err != nil { + t.Fatal(err) + } + } + if calls != 1 { + t.Errorf("the answer was computed %d times inside one window, want 1", calls) + } + + // And it does expire, or the page would never update. + s.cache.Store("k", cached{at: time.Now().Add(-time.Hour), body: 1}) + if _, err := s.memo("k", f); err != nil { + t.Fatal(err) + } + if calls != 2 { + t.Errorf("a stale answer was reused; calls = %d", calls) + } +} + +// A cluster that cannot be reached still has a name, and the header should +// show it along with the reason rather than the page failing entirely. +func TestSummaryReportsUnreachableInTheBody(t *testing.T) { + s := testServer() + w := httptest.NewRecorder() + s.Handler().ServeHTTP(w, httptest.NewRequest("GET", "/api/summary?token=s3cret", nil)) + if w.Code != http.StatusOK { + t.Fatalf("summary answered %d for an unreachable cluster, want 200 with the reason", w.Code) + } + body := w.Body.String() + if !strings.Contains(body, `"cluster":"demo"`) { + t.Errorf("the cluster name is missing: %s", body) + } + if !strings.Contains(body, `"error"`) { + t.Errorf("an unreachable cluster reported no error: %s", body) + } +} + +func TestLogsAndDescribeNeedTheirArguments(t *testing.T) { + h := testServer().Handler() + for _, path := range []string{ + "/api/logs?token=s3cret", + "/api/logs?token=s3cret&ns=default", + "/api/describe?token=s3cret", + "/api/describe?token=s3cret&kind=pod", + } { + w := httptest.NewRecorder() + h.ServeHTTP(w, httptest.NewRequest("GET", path, nil)) + if w.Code != http.StatusBadRequest { + t.Errorf("%s answered %d, want 400", path, w.Code) + } + } +} diff --git a/internal/web/ui/app.css b/internal/web/ui/app.css new file mode 100644 index 0000000..606760e --- /dev/null +++ b/internal/web/ui/app.css @@ -0,0 +1,141 @@ +/* The interface is deliberately plain: dense tables, one accent colour, and + status carried by colour in a single column rather than by decorating whole + rows. An operator reads this while something is broken. */ + +:root { + --bg: #0d1117; + --panel: #161b22; + --line: #21262d; + --text: #c9d1d9; + --dim: #8b949e; + --ok: #3fb950; + --warn: #d29922; + --bad: #f85149; + --accent: #58a6ff; + --mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace; +} + +* { box-sizing: border-box; } + +body { + margin: 0; + background: var(--bg); + color: var(--text); + font: 14px/1.5 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif; +} + +header { + display: flex; + align-items: center; + gap: 1rem; + padding: .65rem 1rem; + border-bottom: 1px solid var(--line); + background: var(--panel); + position: sticky; + top: 0; + z-index: 3; +} +.brand { font-weight: 700; letter-spacing: .01em; } +.cluster { display: flex; align-items: center; gap: .5rem; } +.chip { + font-size: 11px; color: var(--dim); border: 1px solid var(--line); + padding: .05rem .4rem; border-radius: 999px; +} +.vitals { display: flex; gap: 1rem; color: var(--dim); font-size: 13px; } +.vitals b { color: var(--text); font-weight: 600; } +.spacer { flex: 1; } +.ns { color: var(--dim); font-size: 12px; display: flex; gap: .4rem; align-items: center; } +.ns input { + background: var(--bg); border: 1px solid var(--line); color: var(--text); + border-radius: 5px; padding: .25rem .5rem; width: 10rem; font: inherit; font-size: 13px; +} +.ns input:focus { outline: none; border-color: var(--accent); } +.status { font-size: 12px; color: var(--dim); min-width: 4.5rem; text-align: right; } +.status.err { color: var(--bad); } +.version { font-size: 12px; color: var(--dim); } + +nav { + display: flex; gap: .25rem; padding: 0 1rem; + border-bottom: 1px solid var(--line); background: var(--panel); + position: sticky; top: 49px; z-index: 2; +} +nav button { + background: none; border: none; border-bottom: 2px solid transparent; + color: var(--dim); padding: .55rem .8rem; cursor: pointer; font: inherit; +} +nav button:hover { color: var(--text); } +nav button.on { color: var(--text); border-bottom-color: var(--accent); } + +main { padding: 1rem; } +.tab { display: none; } +.tab.on { display: block; } + +table { width: 100%; border-collapse: collapse; font-size: 13px; } +th { + text-align: left; font-weight: 600; color: var(--dim); font-size: 11px; + text-transform: uppercase; letter-spacing: .04em; + padding: .35rem .6rem; border-bottom: 1px solid var(--line); + position: sticky; top: 84px; background: var(--bg); +} +td { padding: .35rem .6rem; border-bottom: 1px solid var(--line); vertical-align: top; } +tbody tr:hover { background: #1c2128; } +td.mono, .mono { font-family: var(--mono); } +td.num { text-align: right; font-variant-numeric: tabular-nums; } +.dim { color: var(--dim); } +.ok { color: var(--ok); } +.warn { color: var(--warn); } +.bad { color: var(--bad); } +.msg { max-width: 60ch; } + +.cards { display: grid; gap: .75rem; grid-template-columns: repeat(auto-fill, minmax(23rem, 1fr)); } +.card { background: var(--panel); border: 1px solid var(--line); border-radius: 8px; padding: .8rem 1rem; } +.card h3 { margin: 0 0 .5rem; font-size: 13px; display: flex; gap: .5rem; align-items: baseline; } +.card h3 .role { color: var(--dim); font-weight: 400; font-size: 12px; } +.card .line { display: flex; gap: .6rem; padding: .12rem 0; font-size: 13px; } +.card .line .what { color: var(--dim); min-width: 11rem; } +.fix { color: var(--warn); font-size: 12px; padding: .1rem 0 .3rem 1.2rem; } + +.finding { background: var(--panel); border: 1px solid var(--line); border-left-width: 3px; border-radius: 6px; margin-bottom: .5rem; } +.finding summary { cursor: pointer; padding: .55rem .8rem; display: flex; gap: .75rem; align-items: baseline; } +.finding summary::-webkit-details-marker { display: none; } +.finding .conf { font-variant-numeric: tabular-nums; color: var(--dim); min-width: 3rem; } +.finding .sig { font-family: var(--mono); font-size: 12px; color: var(--dim); } +.finding p { margin: 0; padding: 0 .8rem .7rem 4.5rem; color: var(--dim); font-size: 13px; } +.finding.sev-bad { border-left-color: var(--bad); } +.finding.sev-warn { border-left-color: var(--warn); } +.finding.sev-note { border-left-color: var(--dim); } + +.banner { padding: .6rem .8rem; border-radius: 6px; margin-bottom: .75rem; font-size: 13px; } +.banner.good { background: #0f2a17; color: var(--ok); border: 1px solid #1f6f36; } +.banner.bad { background: #2a1214; color: #ffa198; border: 1px solid #7d2b28; } + +button.link { + background: none; border: none; color: var(--accent); cursor: pointer; + font: inherit; font-size: 12px; padding: 0; +} +button.link:hover { text-decoration: underline; } + +.drawer { + position: fixed; inset: auto 0 0 0; height: 62vh; background: var(--panel); + border-top: 1px solid var(--line); display: flex; flex-direction: column; z-index: 5; + box-shadow: 0 -8px 24px rgba(1, 4, 9, .6); +} +/* `display: flex` beats the hidden attribute's own display: none, so the + drawer sat open over the page with nothing in it. */ +.drawer[hidden] { display: none; } +.drawer-head { + display: flex; align-items: center; gap: 1rem; + padding: .5rem .8rem; border-bottom: 1px solid var(--line); +} +.drawer-actions { margin-left: auto; display: flex; gap: .5rem; } +.drawer-head button { + background: var(--bg); border: 1px solid var(--line); color: var(--text); + border-radius: 5px; padding: .2rem .6rem; cursor: pointer; font: inherit; font-size: 12px; +} +.drawer pre { + margin: 0; padding: .8rem; overflow: auto; flex: 1; + font-family: var(--mono); font-size: 12px; line-height: 1.45; white-space: pre; +} + +footer { padding: .75rem 1rem 2rem; color: var(--dim); font-size: 12px; } +.empty { color: var(--dim); padding: 1.5rem .6rem; } diff --git a/internal/web/ui/app.js b/internal/web/ui/app.js new file mode 100644 index 0000000..ee39046 --- /dev/null +++ b/internal/web/ui/app.js @@ -0,0 +1,338 @@ +// The whole client. No framework and no build step, because the promise of +// this tool is one file you copy onto a machine — a node_modules directory +// would be a strange thing to find inside it. + +const TOKEN = document.body.dataset.token; +const $ = (sel) => document.querySelector(sel); + +// Everything is rendered by building text and setting textContent. Nothing +// from the cluster is ever interpolated into HTML: pod names, event messages +// and container logs are attacker-influenced in the general case, and a +// diagnosis tool that could be made to run script by a pod name would be an +// odd thing to hand an operator. +function el(tag, opts = {}, kids = []) { + const n = document.createElement(tag); + if (opts.class) n.className = opts.class; + if (opts.text !== undefined) n.textContent = opts.text; + if (opts.title) n.title = opts.title; + if (opts.attrs) for (const [k, v] of Object.entries(opts.attrs)) n.setAttribute(k, v); + if (opts.on) for (const [k, v] of Object.entries(opts.on)) n.addEventListener(k, v); + for (const k of kids) if (k) n.append(k); + return n; +} + +async function api(path, params = {}) { + const u = new URL(path, location.origin); + for (const [k, v] of Object.entries(params)) if (v) u.searchParams.set(k, v); + const r = await fetch(u, { headers: { "X-K3helper-Token": TOKEN } }); + if (!r.ok) { + let detail = r.statusText; + try { detail = (await r.json()).error || detail; } catch (_) { /* not json */ } + throw new Error(detail); + } + return r.json(); +} + +// --- state ------------------------------------------------------------------- + +let tab = "overview"; +let ns = ""; +let timer = null; +let inflight = false; + +function setStatus(text, bad = false) { + const s = $("#status"); + s.textContent = text; + s.classList.toggle("err", bad); +} + +// --- tables ------------------------------------------------------------------ + +function table(cols, rows, render) { + if (!rows || rows.length === 0) { + return el("div", { class: "empty", text: "nothing here" }); + } + const thead = el("thead", {}, [el("tr", {}, cols.map((c) => el("th", { text: c })))]); + const tbody = el("tbody", {}, rows.map(render)); + return el("table", {}, [thead, tbody]); +} + +function statusCell(text, healthy) { + return el("td", { class: healthy ? "ok" : "bad", text }); +} + +// --- views ------------------------------------------------------------------- + +const views = { + async overview(root) { + const [sum, checks, doc] = await Promise.all([ + api("/api/summary"), + api("/api/checks"), + api("/api/doctor"), + ]); + + root.replaceChildren(); + + if (sum.error) { + root.append(el("div", { class: "banner bad", text: "cannot reach the cluster: " + sum.error })); + } else if (doc.healthy) { + root.append(el("div", { class: "banner good", text: "✓ no issues detected — cluster looks healthy" })); + } else { + const real = doc.findings.filter((f) => !f.informational).length; + root.append(el("div", { + class: "banner bad", + text: `${real} issue${real === 1 ? "" : "s"} found — see the Doctor tab`, + })); + } + + // One card per node, or per result group: a kubeconfig cluster has no + // nodes in its targets file and files its skipped checks under the + // context, and an empty page would read as "all clear". + const byNode = new Map(); + for (const c of checks) { + if (!byNode.has(c.node)) byNode.set(c.node, []); + byNode.get(c.node).push(c); + } + const cards = el("div", { class: "cards" }); + for (const [node, rows] of byNode) { + const card = el("div", { class: "card" }, [ + el("h3", {}, [ + el("span", { text: node }), + rows[0].role ? el("span", { class: "role", text: rows[0].role }) : null, + ]), + ]); + for (const r of rows) { + const mark = { OK: "✓", WARN: "!", FAIL: "✗", SKIP: "–" }[r.status] || "?"; + const cls = { OK: "ok", WARN: "warn", FAIL: "bad", SKIP: "dim" }[r.status] || "dim"; + card.append(el("div", { class: "line" }, [ + el("span", { class: cls, text: mark }), + el("span", { class: "what", text: r.name }), + el("span", { text: r.summary }), + ])); + if (r.remediation && (r.status === "FAIL" || r.status === "WARN")) { + card.append(el("div", { class: "fix", text: "↳ " + r.remediation })); + } + } + cards.append(card); + } + root.append(cards); + }, + + async pods(root) { + const rows = await api("/api/pods", { ns }); + root.replaceChildren(table( + ["namespace", "pod", "ready", "status", "restarts", "node", "age", ""], + rows, + (p) => el("tr", {}, [ + el("td", { class: "dim", text: p.namespace }), + el("td", { class: "mono", text: p.name }), + el("td", { text: p.ready }), + statusCell(p.status, p.healthy), + el("td", { class: p.restarts > 0 ? "num warn" : "num dim", text: String(p.restarts) }), + el("td", { class: "dim", text: p.node }), + el("td", { class: "dim", text: p.age }), + el("td", {}, [ + el("button", { class: "link", text: "logs", on: { click: () => showLogs(p) } }), + el("span", { text: " " }), + el("button", { + class: "link", text: "describe", + on: { click: () => showDescribe("pod", p.namespace, p.name) }, + }), + ]), + ]), + )); + }, + + async workloads(root) { + const rows = await api("/api/workloads", { ns }); + root.replaceChildren(table( + ["namespace", "kind", "name", "ready", "age", "images"], + rows, + (wl) => el("tr", {}, [ + el("td", { class: "dim", text: wl.namespace }), + el("td", { class: "dim", text: wl.kind }), + el("td", { class: "mono", text: wl.name }), + el("td", { class: wl.healthy ? "ok" : "warn", text: wl.ready }), + el("td", { class: "dim", text: wl.age }), + el("td", { class: "mono dim", text: wl.images }), + ]), + )); + }, + + async nodes(root) { + const rows = await api("/api/nodes"); + root.replaceChildren(table( + ["name", "status", "roles", "version", "internal ip", "os", "age", ""], + rows, + (n) => el("tr", {}, [ + el("td", { class: "mono", text: n.name }), + statusCell(n.status, n.ready), + el("td", { class: "dim", text: n.roles }), + el("td", { class: "dim", text: n.version }), + el("td", { class: "mono dim", text: n.ip }), + el("td", { class: "dim", text: n.os }), + el("td", { class: "dim", text: n.age }), + el("td", {}, [el("button", { + class: "link", text: "describe", + on: { click: () => showDescribe("node", "", n.name) }, + })]), + ]), + )); + }, + + async events(root) { + const rows = await api("/api/events", { ns }); + root.replaceChildren(table( + ["age", "type", "reason", "object", "message", "count"], + rows, + (e) => el("tr", {}, [ + el("td", { class: "dim", text: e.age }), + el("td", { class: e.warning ? "warn" : "dim", text: e.type }), + el("td", { text: e.reason }), + el("td", { class: "mono dim", text: e.object }), + el("td", { class: "msg", text: e.message }), + el("td", { class: "num dim", text: String(e.count) }), + ]), + )); + }, + + async doctor(root) { + const rep = await api("/api/doctor"); + root.replaceChildren(); + + for (const u of rep.unreachable || []) { + root.append(el("div", { class: "banner bad", text: "node unreachable — " + u })); + } + if (rep.healthy) { + root.append(el("div", { class: "banner good", text: "✓ no issues detected — cluster looks healthy" })); + } + for (const f of rep.findings) { + // A note about what could not be looked at is not a fault, and is not + // coloured like one. + const sev = f.informational ? "note" : f.confidence >= 70 ? "bad" : "warn"; + root.append(el("details", { class: "finding sev-" + sev }, [ + el("summary", {}, [ + el("span", { class: "conf", text: f.confidence + "%" }), + el("span", { text: f.title }), + el("span", { class: "sig", text: f.id }), + ]), + el("p", { text: f.remediation }), + ])); + } + const probes = Object.entries(rep.probe_errors || {}); + if (probes.length) { + root.append(el("div", { class: "empty", text: "could not gather: " + probes.map(([k, v]) => `${k} (${v})`).join("; ") })); + } + }, +}; + +// --- drawer ------------------------------------------------------------------ + +let drawerReload = null; + +function openDrawer(title, body, opts = {}) { + $("#drawer-title").textContent = title; + $("#drawer-body").textContent = body; + $("#drawer-prev").hidden = !opts.onPrevious; + drawerReload = opts.onPrevious || null; + $("#drawer").hidden = false; +} + +async function showLogs(p, previous = false) { + openDrawer(`${p.namespace}/${p.name}${previous ? " (previous container)" : ""}`, "loading…", { + onPrevious: previous ? null : () => showLogs(p, true), + }); + try { + const r = await api("/api/logs", { ns: p.namespace, pod: p.name, previous: previous ? "1" : "" }); + // Logs arrive newest-last, which is what a terminal shows; scroll there. + $("#drawer-body").textContent = r.logs || "(no output)"; + $("#drawer-body").scrollTop = $("#drawer-body").scrollHeight; + } catch (e) { + $("#drawer-body").textContent = "could not read logs: " + e.message; + } +} + +async function showDescribe(kind, namespace, name) { + openDrawer(`${kind}/${name}`, "loading…"); + try { + const r = await api("/api/describe", { kind, ns: namespace, name }); + $("#drawer-body").textContent = r.text || "(nothing)"; + $("#drawer-body").scrollTop = 0; + } catch (e) { + $("#drawer-body").textContent = "could not describe: " + e.message; + } +} + +$("#drawer-close").addEventListener("click", () => { $("#drawer").hidden = true; }); +$("#drawer-prev").addEventListener("click", () => drawerReload && drawerReload()); +document.addEventListener("keydown", (e) => { + if (e.key === "Escape") $("#drawer").hidden = true; +}); + +// --- refresh ----------------------------------------------------------------- + +async function refresh() { + // One request at a time. A slow cluster — every read here is an SSH round + // trip, and the live tests were talking to Singapore — must not have a + // second poll stacked on top of the first. + if (inflight) return; + inflight = true; + setStatus("…"); + try { + await views[tab]($("#" + tab)); + await vitals(); + setStatus("updated " + new Date().toLocaleTimeString()); + } catch (e) { + setStatus(e.message, true); + } finally { + inflight = false; + } +} + +async function vitals() { + const s = await api("/api/summary"); + const v = $("#vitals"); + v.replaceChildren( + el("span", {}, [el("b", { text: `${s.nodes_ready}/${s.nodes}` }), el("span", { text: " nodes ready" })]), + el("span", {}, [ + el("b", { class: s.pods_issue ? "warn" : "", text: `${s.pods - s.pods_issue}/${s.pods}` }), + el("span", { text: " pods healthy" }), + ]), + ); +} + +function select(name, push = true) { + if (!views[name]) name = "overview"; + tab = name; + for (const b of document.querySelectorAll("#tabs button")) b.classList.toggle("on", b.dataset.tab === name); + for (const s of document.querySelectorAll(".tab")) s.classList.toggle("on", s.id === name); + // The tab lives in the fragment, so a view can be linked to and survives a + // reload. The token stays in the page rather than the URL either way. + if (push && location.hash !== "#" + name) history.replaceState(null, "", "#" + name); + refresh(); +} + +for (const b of document.querySelectorAll("#tabs button")) { + b.addEventListener("click", () => select(b.dataset.tab)); +} +window.addEventListener("hashchange", () => select(location.hash.slice(1), false)); + +let nsDebounce = null; +$("#ns").addEventListener("input", (e) => { + clearTimeout(nsDebounce); + nsDebounce = setTimeout(() => { ns = e.target.value.trim(); refresh(); }, 250); +}); + +// Polling stops when the tab is hidden. Left open on a second monitor +// overnight, this would otherwise SSH into a production cluster every five +// seconds until morning. +document.addEventListener("visibilitychange", () => { + clearInterval(timer); + if (!document.hidden) { + refresh(); + timer = setInterval(refresh, 5000); + } +}); + +select(location.hash.slice(1) || "overview", false); +timer = setInterval(refresh, 5000); diff --git a/test/do/REPORT.md b/test/do/REPORT.md new file mode 100644 index 0000000..08c205c --- /dev/null +++ b/test/do/REPORT.md @@ -0,0 +1,308 @@ +# Live test report — k3helper against real VMs + +**Date:** 2026-09-10 · **Provider:** DigitalOcean, `sgp1` (Singapore) +**Nodes:** 7 droplets, Ubuntu 24.04, `s-2vcpu-4gb` (jump host `s-2vcpu-2gb`) +**k3s:** v1.36.4+k3s1 · **Cost:** ~US$0.35 total · **All droplets destroyed at the end** + +Three scenarios, each on its own pair of fresh VMs. Every screenshot below is a +real capture of the terminal against the live cluster, not a mockup. + +--- + +## Scenario 1 — a cluster k3helper did not build + +**Question:** can we point this at a client's existing Kubernetes? + +Two droplets. k3s installed **by hand** with the upstream installer — no +k3helper involved in building anything. Then a workload and two deliberate +faults were deployed, and k3helper was pointed at the result. + +```bash +# by hand, as a client would have done it +curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.4+k3s1 sh - +curl -sfL https://get.k3s.io | K3S_URL=... K3S_TOKEN=... sh - # agent +``` + +**Result: works, with no adoption step.** `check` detected the distribution per +node from its unit files — `k3s` on the server, `k3s-agent` on the agent — +without being told which was which. + +``` +== node server (server) == == node agent1 (agent) == +✓ OK disk 5% used ✓ OK disk 4% used +✓ OK 2534MB memory available ✓ OK 3320MB memory available +✓ OK swap disabled ✓ OK swap disabled +✓ OK cgroup controllers present ✓ OK cgroup controllers present +✓ OK k3s services active ✓ OK k3s services active + (k3s=active) (k3s-agent=active) +``` + +`doctor` found every fault that was planted, and ranked them: + +![doctor](screenshots/doctor.png) + +Worth noting what it did *not* do: it separated `registry.unreachable` (83%) +from the generic `pod.imagepull` (55%). The image was `registry.invalid/nope`, +so the pull never got as far as an answer — a different fix from a bad tag or a +missing credential, and it said so. + +![dashboard](screenshots/dash.png) + +![pods](screenshots/pods.png) + +![xray](screenshots/xray.png) + +### The same cluster with no SSH at all + +The v0.5.0 kubeconfig work, tested the way a managed cluster actually arrives — +a kubeconfig and nothing else: + +```bash +k3helper init --cluster client-managed --kubeconfig ./kubeconfig.yaml +k3helper doctor -t s1-kube.yaml # exit 2 +``` + +Same four faults found, plus an honest note that the host layer was not looked +at. Nothing was silently skipped. + +![kubeconfig mode](screenshots/kube.png) + +--- + +## Scenario 2 — a cluster k3helper builds + +Two fresh droplets, then: + +```bash +k3helper init --server --agent --user root --key ... +k3helper vm setup -t targets.yaml --k3s-version v1.36.4+k3s1 +``` + +**Result: `✓ cluster ready` in 32 seconds**, both nodes Ready. `check` came back +all-green, and `doctor` reported two findings immediately after the install +(Traefik still starting) that cleared on their own within 75 seconds: + +``` +doctor exit=0 +✓ no issues detected — cluster looks healthy +``` + +No false positives on a healthy cluster it had just built. + +**But the first attempt failed**, and that turned out to matter — see finding 1. + +--- + +## Scenario 3 — nodes with no internet + +**This needed code that did not exist.** `vm setup` piped `curl https://get.k3s.io` +into a shell, which an air-gapped node cannot do. + +Set-up: two droplets with **egress blocked at the DigitalOcean firewall** — +outbound allowed only within the VPC subnet — plus one jump host with internet, +standing in for an engineer's laptop inside the client network. The block is at +the provider's edge, not a rule on the host that an install script could undo. + +Proof the nodes were actually cut off, before and after: + +``` +node 1: github=000 get.k3s.io=000 +node 2: github=000 get.k3s.io=000 +``` + +### What was built + +```bash +# on the jump host (has internet) — 8.3 seconds +k3helper bundle k3s --version v1.36.4+k3s1 --arch amd64 -o /root/k3s-bundle + k3s 75MB 835873f37245 + k3s-airgap-images.tar.zst 184MB 9024613e2d46 + install.sh 37KB e5cc3b3d9dfc +✓ bundle ready + +# same jump host, reaching the cut-off nodes over the private network +k3helper vm setup -t /root/targets.yaml --bundle /root/k3s-bundle + [10.104.0.13] uploading k3s-airgap-images.tar.zst (184MB)... + [INFO] Skipping k3s download and verify ← never touched the internet + [10.104.0.13] other nodes will join at https://10.104.0.13:6443 + waiting for 2 node(s) to become ready... + ✓ cluster ready +``` + +Every asset is verified against the release's own sha256 manifest before it +goes anywhere. + +**Result:** + +``` +NAME STATUS ROLES AGE VERSION +s3air-1 Ready control-plane 30s v1.36.4+k3s1 +s3air-2 Ready 19s v1.36.4+k3s1 + +egress test: github=000 k3s.io=000 +``` + +![air-gapped cluster](screenshots/airgap.png) + +`doctor` on the air-gapped cluster: `✓ no issues detected`, exit 0. + +--- + +## What live testing found that unit tests could not + +### 1. The k3s channel service was down, worldwide + +`update.k3s.io` served a **Traefik default certificate from all three of its +IP addresses** — reproducible from DigitalOcean *and* from a laptop in Hong +Kong. Every `curl -sfL https://get.k3s.io | sh -` on the internet was failing +TLS verification, and `vm setup` failed with it: + +``` +[INFO] Finding release for channel stable +curl: (60) SSL certificate problem: self-signed certificate +[ERROR] Download failed +``` + +Not our bug, but our problem: a tool that installs k3s should not be defeated +by a lookup service it does not control. + +**Fixed:** `--k3s-version` pins a release and skips the channel entirely, +fetching from the GitHub release, which stayed healthy throughout. + +### 2. Agents joined on the wrong address + +The address agents dial to reach the server was discovered with `hostname -I`, +which lists a cloud VM's **public** address first. On the air-gapped nodes that +is the one address the network cannot reach: + +``` +level=error msg="Failed to validate connection to cluster at +https://165.22.58.110:6443: failed to get CA certs ... context deadline exceeded" +``` + +The agents retried that forever while the server ran perfectly well beside +them. The targets file had said `10.104.0.13` — the address k3helper itself had +just used to SSH in. + +**Fixed:** the join address now comes from the targets file. `--join-address` +covers the case where the two networks genuinely differ (SSH over public, join +over private). + +### 3. `%!w()` in the failure message + +When the install above failed, the last thing on screen was: + +``` +Error: server install failed (exit 1): %!w() +``` + +A command that ran and exited non-zero has no error to wrap, and `%w` printed +its own failure where the reason should have been. + +**Fixed**, with a test that fails if a format verb ever leaks into that message +again. + +### 4. A healthy managed cluster scored 0% + +Caught in a screenshot, not by a test. Every host check is skipped on a +kubeconfig cluster, and the TUI counted a skip as "not OK" — putting +`score 0%` at the top of a screen showing a perfectly healthy cluster. + +**Fixed:** a skip is excluded from both halves of the fraction, so it reads +`score n/a`. + +### 5. cloud-init races the installer + +A fresh Ubuntu image is still replacing `ca-certificates` when sshd starts +answering, and every https download on the box fails TLS verification while it +does. An install in that window dies with "curl failed to verify the legitimacy +of the server", which reads like a firewall problem and is not one. It bit this +run about 30 seconds after boot. + +**Fixed in the product**, not just the test tooling: both install paths now +wait for cloud-init before touching the machine — guarded by a presence check +so a node without cloud-init is not delayed, and bounded at five minutes so a +stuck one cannot hang the install. The kubeadm path needed it more than k3s: +its prerequisites run `apt` straight into cloud-init's dpkg lock. + +### 6. Confirmed working, unchanged + +- **The crashloop fix from v0.5.0 earned itself.** The pod was caught with its + container `terminated`, not `waiting: CrashLoopBackOff` — precisely the + sampling window the old code missed. The restart count caught it. + +--- + +## A second pass: what the audit found after the fixes + +Each fault above is an instance of a class, so the codebase was searched for +siblings. Six more, none of which had been exercised live. + +**The same join-address bug exists in the kubeadm path.** `kubeadm init` +defaults the API server's advertise address to the default route's interface — +the public one on a cloud VM — and the join command handed to every agent is +built from it. Identical failure, different installer. Fixed the same way, and +the resolver is now shared rather than implemented twice. Found by reading, not +by running: the kubeadm path was never live-tested here. + +**The dashboard was blank for a kubeconfig cluster.** It iterated the targets +file's nodes, and a kubeconfig cluster has none — so the skipped host checks +and their reason never reached the screen. An empty dashboard reads as "all +clear", which is the one thing those skipped results exist to prevent. This is +the same bug as the 0% score, one layer up. + +![kubeconfig dashboard](screenshots/dashkube.png) + +*The kubeconfig dashboard after the fix. Before it, this screen was empty.* + +**Skipped checks were invisible in the counter**, which read `0 ok, 0 warn, +0 fail` on a cluster where six checks had been deliberately skipped. Now says +`6 skipped`. + +**`--bundle` with `--distro kubeadm` was silently ignored.** An operator asking +for an offline install would have got an online one, and found out on an +air-gapped node at the worst possible moment. Now refused with the reason. +`--bundle` with `--k3s-version` is refused too — they contradict. + +**The bundle's architecture was recorded but never checked.** The manifest +carries it and a comment claimed it prevented installing an arm64 build on an +amd64 node; nothing compared the two, so the first sign would have been "cannot +execute binary file" after a 260MB upload. Now checked on every node before any +node is uploaded to. + +**Uploads were neither verified nor cleaned up.** 260MB over a link that may be +a tunnel, with the hashes already in the manifest and unused. A truncated k3s +binary fails immediately; a truncated image archive fails much later, as pods +that will not start on a cluster that installed cleanly. Now hashed on the node +after upload, and the staging copy is removed once the installer has run rather +than left on every node's disk. + +**One of my own:** `installFailure` duplicated `exitReason`, which the kubeadm +path had been using correctly all along. Consolidated. + +--- + +## Reproducing this + +```bash +test/do/do.sh up s1 2 # two droplets, sgp1 +test/do/do.sh cut s1 # block egress at the provider firewall +test/do/do.sh mend s1 # restore it +test/do/do.sh list # everything still costing money +test/do/do.sh down s1 # destroy +``` + +The token is read from a gitignored `.env.production` and never passed on a +command line, where `ps` would show it to any other user on the machine. +Everything created is tagged `k3helper-test`, so `down` cannot touch a droplet +this tool did not create. + +## Still open + +- **The web GUI is not built.** These are the TUI. `k3helper serve` — the + browser UI that would also run in Docker — is the next piece. +- **Workload images in an air-gapped cluster.** The bundle covers the cluster's + own images. Client images still need an internal registry, which k3helper can + already configure but which this test did not exercise. +- **Bundle fan-out.** Each node is uploaded to directly. Seeding one node and + having it distribute to the rest would save N× the transfer on a slow link. diff --git a/test/do/do.sh b/test/do/do.sh new file mode 100755 index 0000000..ef18f8d --- /dev/null +++ b/test/do/do.sh @@ -0,0 +1,261 @@ +#!/usr/bin/env bash +# DigitalOcean droplets for live testing, driven straight off the v2 API. +# +# test/do/do.sh up [size] create droplets, wait, print IPs +# test/do/do.sh ips the IPs again +# test/do/do.sh ssh [cmd] ssh to the nth droplet +# test/do/do.sh cut block outbound internet (airgap) +# test/do/do.sh mend restore outbound internet +# test/do/do.sh down destroy droplets and firewalls +# test/do/do.sh list everything this tool created +# +# The token comes from .env.production, which is gitignored and stays out of +# every command line here — an argument is visible in `ps` to any other user +# on the machine, and would end up in shell history. +# +# Everything is tagged k3helper-test so `down` can never touch a droplet this +# script did not create, and `list` can show what is still costing money. +set -euo pipefail + +ROOT=$(cd "$(dirname "$0")/../.." && pwd) +STATE="$ROOT/test/do/.state" +KEY="$STATE/id_ed25519" +TAG=k3helper-test +REGION="${DO_REGION:-sgp1}" +IMAGE="${DO_IMAGE:-ubuntu-24-04-x64}" +SIZE_DEFAULT="${DO_SIZE:-s-2vcpu-4gb}" + +mkdir -p "$STATE" +chmod 700 "$STATE" + +if [ -z "${DO_API_KEY:-}" ]; then + [ -f "$ROOT/.env.production" ] || { echo "error: no .env.production and DO_API_KEY unset" >&2; exit 1; } + set -a; . "$ROOT/.env.production"; set +a +fi +[ -n "${DO_API_KEY:-}" ] || { echo "error: DO_API_KEY is empty" >&2; exit 1; } + +api() { + local method=$1 path=$2; shift 2 + curl -sS -X "$method" \ + -H "Authorization: Bearer $DO_API_KEY" \ + -H "Content-Type: application/json" \ + "https://api.digitalocean.com/v2$path" "$@" +} + +# jq is not assumed; python3 is already required by the sandbox tooling. +pyq() { python3 -c "$1"; } + +log() { printf '%s\n' "$*" >&2; } + +# --- ssh key ----------------------------------------------------------------- +# A throwaway key per checkout, never the developer's own: these droplets are +# public and short-lived, and the key ends up in a state directory. +ensure_key() { + if [ ! -f "$KEY" ]; then + ssh-keygen -q -t ed25519 -f "$KEY" -N '' -C k3helper-test + log "generated $KEY" + fi + local fp + fp=$(ssh-keygen -lf "$KEY.pub" | awk '{print $2}' | sed 's/^SHA256://') + local id + id=$(api GET "/account/keys?per_page=200" | pyq " +import json,sys +ks=json.load(sys.stdin)['ssh_keys'] +print(next((str(k['id']) for k in ks if k['name']=='k3helper-test'), '')) +") + if [ -z "$id" ]; then + # The payload goes through a file rather than a nested command + # substitution: a public key contains spaces, and quoting it through two + # levels of shell and one of python is how it silently arrived empty — + # which produced droplets with no key on them and no way in. + KEY_PUB="$KEY.pub" python3 -c " +import json,os +print(json.dumps({'name':'k3helper-test','public_key':open(os.environ['KEY_PUB']).read().strip()})) +" > "$STATE/key.json" + id=$(api POST "/account/keys" -d "@$STATE/key.json" | pyq " +import json,sys +d=json.load(sys.stdin) +k=d.get('ssh_key') +if not k: raise SystemExit('ssh key upload failed: ' + json.dumps(d)) +print(k['id']) +") + log "uploaded ssh key to DigitalOcean (id $id)" + fi + [ -n "$id" ] || { log "error: no ssh key id; refusing to create droplets nobody can log into"; exit 1; } + echo "$id" +} + +SSH_OPTS=(-i "$KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null + -o LogLevel=ERROR -o ConnectTimeout=10 -o BatchMode=yes) + +# --- lifecycle --------------------------------------------------------------- +cmd_up() { + local prefix=$1 count=${2:-2} size=${3:-$SIZE_DEFAULT} + local keyid; keyid=$(ensure_key) + local names=() + for i in $(seq 1 "$count"); do names+=("\"$prefix-$i\""); done + + log "creating $count x $size in $REGION ($IMAGE)..." + api POST "/droplets" -d "{ + \"names\": [$(IFS=,; echo "${names[*]}")], + \"region\": \"$REGION\", + \"size\": \"$size\", + \"image\": \"$IMAGE\", + \"ssh_keys\": [$keyid], + \"tags\": [\"$TAG\", \"$prefix\"] + }" > "$STATE/$prefix.create.json" + + if grep -q '"id":"' "$STATE/$prefix.create.json" 2>/dev/null; then + log "create failed:"; cat "$STATE/$prefix.create.json" >&2; exit 1 + fi + + log "waiting for droplets to come up..." + local tries=0 + while [ $tries -lt 60 ]; do + api GET "/droplets?tag_name=$prefix&per_page=200" > "$STATE/$prefix.json" + local ready + ready=$(pyq " +import json +ds=json.load(open('$STATE/$prefix.json'))['droplets'] +up=[d for d in ds if d['status']=='active' and any(n['type']=='public' for n in d['networks']['v4'])] +print(len(up)) +") + [ "$ready" = "$count" ] && break + sleep 5; tries=$((tries+1)) + done + + log "waiting for sshd..." + for ip in $(cmd_ips "$prefix"); do + local t=0 + until ssh "${SSH_OPTS[@]}" "root@$ip" true 2>/dev/null; do + t=$((t+1)); [ $t -gt 40 ] && { log "error: $ip never accepted ssh"; exit 1; } + sleep 5 + done + done + + # sshd answers well before the machine is finished with itself. cloud-init is + # still running apt, and on a fresh Ubuntu image that includes replacing + # ca-certificates — during which every https download on the box fails TLS + # verification. Installing k3s in that window fails with "curl failed to + # verify the legitimacy of the server", which looks like a network policy + # problem and is not one. + log "waiting for cloud-init to finish..." + for ip in $(cmd_ips "$prefix"); do + ssh "${SSH_OPTS[@]}" "root@$ip" "cloud-init status --wait >/dev/null 2>&1 || true" || true + done + cmd_ips "$prefix" +} + +cmd_ips() { + api GET "/droplets?tag_name=$1&per_page=200" | pyq " +import json,sys +ds=sorted(json.load(sys.stdin)['droplets'], key=lambda d: d['name']) +for d in ds: + ip=next((n['ip_address'] for n in d['networks']['v4'] if n['type']=='public'), None) + if ip: print(ip) +" +} + +cmd_privips() { + api GET "/droplets?tag_name=$1&per_page=200" | pyq " +import json,sys +ds=sorted(json.load(sys.stdin)['droplets'], key=lambda d: d['name']) +for d in ds: + ip=next((n['ip_address'] for n in d['networks']['v4'] if n['type']=='private'), None) + if ip: print(ip) +" +} + +cmd_ssh() { + local prefix=$1 n=${2:-1}; shift 2 || true + local ip; ip=$(cmd_ips "$prefix" | sed -n "${n}p") + [ -n "$ip" ] || { log "no droplet $prefix-$n"; exit 1; } + if [ $# -gt 0 ]; then ssh "${SSH_OPTS[@]}" "root@$ip" "$@"; else ssh "${SSH_OPTS[@]}" "root@$ip"; fi +} + +# --- airgap ------------------------------------------------------------------ +# A DigitalOcean firewall with outbound rules that name only the droplets' +# own subnet. Anything else — get.k3s.io, Docker Hub, the distro mirrors — is +# dropped at DigitalOcean's edge, not by a rule on the host that the install +# script could undo. Inbound SSH from anywhere stays open, or the test could +# not drive the machines at all. +cmd_cut() { + local prefix=$1 + local ids; ids=$(api GET "/droplets?tag_name=$prefix&per_page=200" | pyq " +import json,sys +print(','.join(str(d['id']) for d in json.load(sys.stdin)['droplets'])) +") + local cidr; cidr=$(cmd_privips "$prefix" | head -1 | sed 's/\.[0-9]*$/.0\/20/') + log "blocking egress for $prefix (allowing only $cidr and ssh in)..." + api POST "/firewalls" -d "{ + \"name\": \"$prefix-airgap\", + \"droplet_ids\": [$ids], + \"inbound_rules\": [ + {\"protocol\":\"tcp\",\"ports\":\"22\",\"sources\":{\"addresses\":[\"0.0.0.0/0\"]}}, + {\"protocol\":\"tcp\",\"ports\":\"all\",\"sources\":{\"addresses\":[\"$cidr\"]}}, + {\"protocol\":\"udp\",\"ports\":\"all\",\"sources\":{\"addresses\":[\"$cidr\"]}} + ], + \"outbound_rules\": [ + {\"protocol\":\"tcp\",\"ports\":\"all\",\"destinations\":{\"addresses\":[\"$cidr\"]}}, + {\"protocol\":\"udp\",\"ports\":\"all\",\"destinations\":{\"addresses\":[\"$cidr\"]}} + ] + }" > "$STATE/$prefix.firewall.json" + pyq " +import json +d=json.load(open('$STATE/$prefix.firewall.json')) +fw=d.get('firewall') +print('firewall', fw['id'], fw['status']) if fw else print('FAILED:', d) +" +} + +cmd_mend() { + local prefix=$1 + local id; id=$(api GET "/firewalls?per_page=200" | pyq " +import json,sys +fs=json.load(sys.stdin)['firewalls'] +print(next((f['id'] for f in fs if f['name']=='$prefix-airgap'), '')) +") + [ -n "$id" ] || { log "no $prefix-airgap firewall"; return 0; } + api DELETE "/firewalls/$id" >/dev/null + log "removed $prefix-airgap" +} + +# --- teardown ---------------------------------------------------------------- +cmd_down() { + local prefix=$1 + cmd_mend "$prefix" || true + # Only ever by tag, and only tags this script applies: a typo'd prefix + # destroys nothing rather than something else. + local n; n=$(api GET "/droplets?tag_name=$prefix&per_page=200" | pyq " +import json,sys; print(len(json.load(sys.stdin)['droplets']))") + if [ "$n" = "0" ]; then log "no droplets tagged $prefix"; return 0; fi + log "destroying $n droplet(s) tagged $prefix..." + api DELETE "/droplets?tag_name=$prefix" >/dev/null + rm -f "$STATE/$prefix".*.json "$STATE/$prefix.json" + log "done" +} + +cmd_list() { + api GET "/droplets?tag_name=$TAG&per_page=200" | pyq " +import json,sys +ds=json.load(sys.stdin)['droplets'] +if not ds: print('no k3helper-test droplets'); raise SystemExit +for d in sorted(ds, key=lambda d: d['name']): + ip=next((n['ip_address'] for n in d['networks']['v4'] if n['type']=='public'), '-') + print(f\" {d['name']:24} {d['status']:8} {d['size_slug']:14} {ip}\") +print(f'{len(ds)} droplet(s) running') +" +} + +case "${1:-}" in + up) shift; cmd_up "$@" ;; + ips) shift; cmd_ips "$@" ;; + privips) shift; cmd_privips "$@" ;; + ssh) shift; cmd_ssh "$@" ;; + cut) shift; cmd_cut "$@" ;; + mend) shift; cmd_mend "$@" ;; + down) shift; cmd_down "$@" ;; + list) cmd_list ;; + key) ensure_key >/dev/null; echo "$KEY" ;; + *) sed -n '2,20p' "$0"; exit 1 ;; +esac diff --git a/test/do/screenshots/airgap.png b/test/do/screenshots/airgap.png new file mode 100644 index 0000000..4a5c621 Binary files /dev/null and b/test/do/screenshots/airgap.png differ diff --git a/test/do/screenshots/dash.png b/test/do/screenshots/dash.png new file mode 100644 index 0000000..90a810d Binary files /dev/null and b/test/do/screenshots/dash.png differ diff --git a/test/do/screenshots/dashkube.png b/test/do/screenshots/dashkube.png new file mode 100644 index 0000000..1ac8229 Binary files /dev/null and b/test/do/screenshots/dashkube.png differ diff --git a/test/do/screenshots/doctor.png b/test/do/screenshots/doctor.png new file mode 100644 index 0000000..090f36c Binary files /dev/null and b/test/do/screenshots/doctor.png differ diff --git a/test/do/screenshots/kube.png b/test/do/screenshots/kube.png new file mode 100644 index 0000000..e3f0ee3 Binary files /dev/null and b/test/do/screenshots/kube.png differ diff --git a/test/do/screenshots/pods.png b/test/do/screenshots/pods.png new file mode 100644 index 0000000..922279b Binary files /dev/null and b/test/do/screenshots/pods.png differ diff --git a/test/do/screenshots/xray.png b/test/do/screenshots/xray.png new file mode 100644 index 0000000..49dbecf Binary files /dev/null and b/test/do/screenshots/xray.png differ diff --git a/test/e2e.sh b/test/e2e.sh index ef5b2ed..9d73536 100755 --- a/test/e2e.sh +++ b/test/e2e.sh @@ -138,7 +138,11 @@ fi # ── 4. bootstrap ───────────────────────────────────────────────────────────── step "4. Bootstrap k3s on all 3 nodes over SSH" rm -f "$WORK/kubeconfig" +# The version is pinned rather than resolved from a channel: a channel is a +# lookup against update.k3s.io, and an outage there fails the install with a +# TLS error that has nothing to do with anything this suite is testing. OUT=$($K vm setup -t $TARGETS \ + --k3s-version "${K3S_VERSION:-v1.36.4+k3s1}" \ --server-extra-args "--snapshotter=native --disable=traefik" \ --agent-extra-args "--snapshotter=native" \ --kubeconfig "$WORK/kubeconfig" 2>&1)