feat: web GUI, offline kubeadm install, live-test fixes - #2
Merged
Merged
Conversation
…live testing found Three scenarios were run against real DigitalOcean VMs: a cluster built by hand and then monitored, a cluster built by k3helper, and a cluster built on two nodes with egress blocked at the provider firewall. The third needed code that did not exist; the first two found bugs that unit tests could not. Offline install. `k3helper bundle k3s --version v1.31.2+k3s1` fetches the binary, the airgap image archive and the installer on a machine that has a connection, verifying each against the release sha256 manifest. `vm setup --bundle` uploads them to every node, puts them where the installer looks, and runs it with INSTALL_K3S_SKIP_DOWNLOAD. The nodes need SSH from wherever k3helper runs and a route to each other, and nothing else. Verified with github and get.k3s.io both unreachable from the nodes. Agents joined on the wrong address. The join address came from `hostname -I`, which lists a cloud VM's public address first — the one address an air-gapped network cannot use. The agents sat retrying "failed to get CA certs" against it while the server ran perfectly well beside them. It now comes from the targets file: the operator chose that address and k3helper has just proved it works by connecting over it. `--join-address` covers the case where the two networks genuinely differ. A pinned k3s version, because the channel service went down mid-test. update.k3s.io served a Traefik default certificate from all three of its addresses — reproducible from DigitalOcean and from a laptop — so every `curl -sfL https://get.k3s.io | sh -` on the internet failed TLS verification and `vm setup` failed with it. `--k3s-version` skips that lookup and fetches straight from the GitHub release. `%!w(<nil>)` was the last thing on screen when that install failed: a command that ran and exited non-zero has no error to wrap. A healthy managed cluster scored 0% in the TUI, because every host check is skipped on a kubeconfig cluster and skips counted as "not OK". Seen in a screenshot of a live cluster. A skip is now outside both halves of the fraction, so the score reads n/a. Uploads stream rather than buffering: the image archive is 184MB per node. test/do/do.sh provisions, air-gaps and destroys the droplets through the DigitalOcean API. Everything it creates is tagged k3helper-test so teardown cannot touch anything else, and the token is read from a gitignored .env.production rather than passed on a command line where `ps` would show it.
Three scenarios against real VMs, the five things they found, and how to reproduce them. Screenshots are real captures of the TUI against the live clusters — including the one that showed a healthy managed cluster scoring 0%.
The sandbox and the E2E installed from the "stable" channel, which is a lookup against update.k3s.io. That service served a Traefik default certificate from all three of its addresses today, so every `curl -sfL https://get.k3s.io | sh -` on the internet failed TLS verification — and CI's E2E failed with it, on a branch that had nothing to do with it. A pinned version fetches straight from the GitHub release and skips the lookup. It is also simply better for a test suite: the cluster under test is now the same one every run, rather than whatever "stable" meant that morning.
… each bug Each fault the DigitalOcean run turned up is an instance of a class, so the codebase was searched for the others. Six more, none of them exercised live. kubeadm had the same join-address bug as k3s. `kubeadm init` defaults the API server's advertise address to the default route's interface — the public one on a cloud VM — and the join command handed to every agent is built from it. Same failure, different installer. The resolver is now shared rather than written twice, which is also how the k3s path came to have `installFailure` while the kubeadm path had `exitReason` doing the same job; that is consolidated too. The TUI dashboard was blank for a kubeconfig cluster. It drew one card per node in the targets file and a kubeconfig cluster has none, so the skipped host checks and the reason for them never reached the screen. That is the same bug as the 0% score one layer up: an empty dashboard reads as "all clear", which is precisely what those skipped results exist to prevent. Skipped checks are now counted in the summary line too, instead of leaving "0 ok, 0 warn, 0 fail" on a cluster where six checks were deliberately not applicable. cloud-init raced the installer, and this time it is fixed in the product rather than in the test harness. A fresh cloud image is still replacing ca-certificates when sshd answers, and every https download on the box fails TLS verification until it finishes — which reads like a firewall problem and is not one. The kubeadm path needed it more than k3s: its prerequisites run apt straight into cloud-init's dpkg lock. Guarded by a presence check so a node without cloud-init is not delayed, and bounded so a stuck one cannot hang the install forever. Three in the offline path, which had only ever run its happy path. `--bundle` was silently ignored with `--distro kubeadm`, so an operator asking for an offline install got an online one and found out on an air-gapped node. A bundle's architecture was recorded in its manifest and never compared to the node, despite a comment claiming otherwise — the first sign would have been "cannot execute binary file" after a 260MB upload. And the upload was neither verified against the hashes already in that manifest nor cleaned up afterwards, leaving a few hundred megabytes on every node's disk.
…or by tunnel
k3s installs offline from a bundle because it is one binary and one image
archive. kubeadm cannot: it needs apt packages and images from registry.k8s.io,
and neither fits in a file carried in. Two answers, both live-tested against
DigitalOcean droplets with egress blocked at the provider firewall.
--apt-mirror points a node's package manager at a mirror the operator runs.
Most sites with air-gapped Kubernetes already have one, and the useful thing to
do is point at it rather than replace it. Both source formats are rewritten —
24.04's deb822 .sources and older .list entries, missing either leaves half the
sources unreachable — with a backup of every file first, third-party
repositories left alone, and a check afterwards that the rewrite actually
matched something.
--via-proxy is for sites with no mirror either. k3helper already holds an SSH
connection to every node, so the connection carries the traffic backwards: a
listener on the node's loopback, a proxy on this machine, a host allowlist, and
both removed when the install finishes. The proxy resolves names on this side,
which matters because a genuinely cut-off node has no working DNS to resolve
them with. It is off unless asked for and announced every time it is used:
lending an isolated machine a route out is the operator's decision, and in some
environments it is not theirs to make.
Seven things live testing found that the tests could not, in order:
- The proxy hung on every CONNECT. x/crypto/ssh channels refuse deadlines,
and http.Server's hijack path interrupts its background read by setting
one; on a connection that cannot, it waits forever and never replies. The
proxy now reads the protocol itself, which for a forward proxy is less
machinery anyway. Plain http worked throughout, which is what made it look
like a network problem.
- Bytes a client pipelines after CONNECT were dropped, because the buffered
reader from Hijack was discarded. curl sends its ClientHello without
waiting for the 200.
- The allowlist was too narrow four separate times, each found only by
running it: the provider's own apt mirror, Ubuntu Pro's ESM endpoint, the
CDN that pkgs.k8s.io redirects to, and — after a cluster installed
perfectly and sat NotReady — ghcr.io, where flannel's images live.
- A refusal suggested a --proxy-allow value with the port still on it, which
would not have matched.
- The node reached its own API server through the proxy. NO_PROXY was set for
containerd and not for the commands, so kubeadm sent cluster traffic to the
operator's laptop, which refused its own node's address.
- apt wedged on the dead tunnel when a run was interrupted, holding the apt
lock for 31 minutes and failing every later run on that node with a lock
error that pointed nowhere near the cause. It has a timeout now.
- Waiting for cloud-init is not enough: apt-daily and unattended-upgrades are
on timers and take the lock later. Every apt-get now waits for it.
The tunnel is install-time only, and the README says so: when vm setup
finishes the cluster has no internet again and cannot pull a workload image.
For anything past the install, point the nodes at an internal registry — and
for a genuinely disconnected site, k3s with --bundle remains the better fit.
The third way to look at a cluster, after the CLI and the TUI, and the same one underneath: `serve` opens a connection exactly as `doctor` does and the handlers are thin wrappers over the same functions. An overview with a card per node, tables for pods, workloads, nodes and events, logs and describe in a drawer, and Doctor's ranked findings with their remediation. No Electron, no WebView2, no npm. The assets are embedded, so this is still one file to copy onto a machine and it runs on Windows, macOS, Linux and in a container with nothing else installed — about 3MB on top of the binary, against the few hundred megabytes a bundled browser runtime would cost. There is a Dockerfile too, FROM scratch, around 17MB. Read-only, deliberately. Everything lists, describes or tails; any non-GET is refused. A browser tab that can change a production cluster deserves rather more thought about who is holding it than a localhost token provides, and none of that is needed to make a cluster legible, which is what this is for. Loopback unless told otherwise. A session token, fresh per launch, is the whole of the authentication — so --host somewhere other than loopback needs --allow-remote as well, and says why. The token reaches the page in its HTML rather than staying in the URL, with no-referrer and a content security policy so nothing carries it away. Nothing from the cluster is ever interpolated into HTML: pod names and container logs are attacker-influenced in the general case, and a diagnosis tool that could be made to run script by a pod name would be an odd thing to hand an operator. --host and --port behave the way a dev server should. A default port steps past one already in use, because a previous run still shutting down is not a reason to refuse to start; an explicit one does not, because somebody who names a port has a reason. Reads are memoised for a couple of seconds and a diagnosis for fifteen. Every one of them is an SSH round trip — the live tests were talking to Singapore, where a pod list takes the best part of a second — and a browser polling four panels would otherwise keep a connection permanently busy. Polling stops entirely while the tab is hidden, rather than SSHing into production every five seconds until morning. Verified against a live two-node cluster on DigitalOcean with a crashlooping pod and a bad image planted in it: the tables, the logs of a previous container, describe, and the same four findings the CLI reports. A screenshot caught the one bug worth catching — `display: flex` beats the hidden attribute, so the drawer sat open over the page with nothing in it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
internal/web)internal/vm/offline.go,viaproxy.go)test/do/REPORT.md)Test plan
test/do/do.shlive run on DigitalOcean (seetest/do/REPORT.md+ screenshots)internal/vm/setup_test.go,internal/web/server_test.go