feat: web GUI, offline kubeadm install, live-test fixes

.gitignore

@@ -8,3 +8,11 @@ dist/
 # Fetched kubeconfigs embed cluster client certificates and keys.
 *kubeconfig*.yaml
 sandbox-kubeconfig.yaml
+
+# Credentials for live testing. The DigitalOcean token in .env.production is a
+# working key against a real account, and this repository is public — one
+# `git add -A` is all it takes.
+.env
+.env.*
+!.env.example
+test/do/.state/

CHANGELOG.md

@@ -4,6 +4,110 @@ Notable changes per release. The release workflow publishes the section
 matching the tag it is building, so this file is the source of the release
 notes on GitHub.
 
+## Unreleased
+
+### Added
+
+- **`k3helper serve` — a web interface.** The same binary, the same data layer
+  and the same diagnosis as the CLI and the TUI, shown in a browser: an
+  overview with a card per node, tables for pods, workloads, nodes and events,
+  logs and `describe` in a drawer, and Doctor's ranked findings with their
+  remediation. `--host` and `--port` work the way they do in any dev server; a
+  port left at its default steps past one already in use, an explicit one does
+  not.
+
+  It is read-only — everything lists, describes or tails, and any non-GET is
+  refused — and binds to loopback unless given `--host` *and* `--allow-remote`,
+  because a per-launch session token is the only authentication there is.
+
+  No Electron, no WebView2, no npm: the assets are embedded, so it runs on
+  Windows, macOS, Linux and in a container with nothing else installed. It
+  costs about 3MB of binary.
+- **A Dockerfile**, `FROM scratch`, around 17MB. Credentials are mounted at run
+  time, never built in.
+
+Live testing against real DigitalOcean VMs, and the four bugs it found.
+
+### Added
+
+- **`k3helper bundle k3s`** builds an offline install bundle — the k3s binary,
+  the airgap image archive and the installer — verified against the release
+  sha256 manifest. `k3helper vm setup --bundle <dir>` installs from it with
+  `INSTALL_K3S_SKIP_DOWNLOAD`, so the nodes need no internet at all. Proven on
+  two DigitalOcean droplets with egress blocked at the provider firewall:
+  `github=000, get.k3s.io=000`, cluster Ready.
+- **`--k3s-version`** pins an exact release and skips the update.k3s.io channel
+  lookup. During testing that service served a Traefik default certificate from
+  every one of its addresses, which breaks `curl -sfL https://get.k3s.io | sh -`
+  everywhere; a pinned version fetches from the GitHub release instead.
+- **`--join-address`** overrides the address other nodes dial to reach the
+  first server, for when k3helper reaches the nodes over one network and the
+  cluster talks over another.
+- **`ssh.Client.WriteFileFrom`** streams a file to a node with progress,
+  instead of holding it in memory. The airgap image archive is 184MB.
+
+- **`--apt-mirror` and `--k8s-apt-repo`** point a node's package manager at
+  mirrors you run instead of the distribution's own archive. Both source
+  formats are rewritten — 24.04's deb822 `.sources` and older `.list` entries —
+  with a backup of every file first, and third-party repositories left alone.
+- **`--via-proxy`** lends the nodes the operator's internet connection for the
+  length of an install, over the SSH connection already open to them, through a
+  proxy with a host allowlist. It is what makes an air-gapped kubeadm install
+  possible at all: unlike k3s, kubeadm needs apt packages and registry images
+  that do not fit in a bundle. Off unless asked for, announced when used, and
+  removed from the node afterwards. Proven on two DigitalOcean droplets with
+  egress blocked at the provider firewall: `✓ cluster ready`, both nodes Ready,
+  ten pods Running, `github=000`.
+
+### Fixed
+
+- **kubeadm advertised the wrong API server address**, for the same reason the
+  k3s path did: `kubeadm init` defaults it to the default route's interface,
+  which on a cloud VM is the public one, and the join command handed to every
+  agent is built from it. Both paths now share one resolver.
+- **The TUI dashboard was blank for a kubeconfig cluster.** It drew one card
+  per node in the targets file, and a kubeconfig cluster has no nodes — so the
+  skipped host checks and the reason for them never reached the screen. An
+  empty dashboard reads as "all clear", which is what those results exist to
+  prevent. Skipped checks are also now counted in the summary line instead of
+  vanishing from `0 ok, 0 warn, 0 fail`.
+- **`--bundle` was silently ignored with `--distro kubeadm`**, so an operator
+  asking for an offline install got an online one and found out on an
+  air-gapped node. It is refused now, as is `--bundle` with `--k3s-version`,
+  which contradict each other.
+- **A bundle's architecture was recorded and never checked.** Installing an
+  arm64 build on an amd64 node would have failed as "cannot execute binary
+  file" after a 260MB upload. Every node is checked before any node is uploaded
+  to.
+- **Uploaded bundles were neither verified nor cleaned up.** The manifest
+  already had the hashes; they are now checked on the node after the transfer,
+  and the staged copy is removed once the installer has run rather than left on
+  every node's disk.
+- **cloud-init raced the installer.** A fresh cloud image is still replacing
+  ca-certificates when sshd starts answering, and https downloads fail TLS
+  verification until it finishes — which looks like a firewall problem and is
+  not one. Both install paths wait first; kubeadm needed it more, since its
+  prerequisites run apt straight into cloud-init's dpkg lock.
+- **Agents joined on the wrong address.** The join address was discovered with
+  `hostname -I`, which returns a cloud VM's public address first — the one
+  address an air-gapped network cannot reach. Agents retried "failed to get CA
+  certs" indefinitely while the server ran fine beside them. It now comes from
+  the targets file: the address the operator chose, and the one k3helper has
+  just proved works by connecting over it.
+- **`%!w(<nil>)` in install failures.** A command that ran and exited non-zero
+  has no error to wrap, and the `%w` verb printed its own failure as the last
+  thing an operator saw when an install failed.
+- **A healthy managed cluster scored 0% in the TUI.** Every host check is
+  skipped on a kubeconfig cluster, and skips were counted as "not OK". A skip
+  is now left out of both halves of the fraction, so the score reads `n/a`
+  rather than putting the most alarming number on screen for a healthy cluster.
+
+### Testing
+
+- `test/do/do.sh` provisions, air-gaps and destroys DigitalOcean droplets
+  through the v2 API for live scenario testing. Everything it creates is tagged
+  `k3helper-test`, so teardown can never touch anything else.
+
 ## v0.5.0
 
 Clusters you cannot SSH into. k3helper now reaches a cluster either by its

Dockerfile

@@ -0,0 +1,42 @@
+# k3helper as a container, for `serve`.
+#
+# FROM scratch, because the binary needs nothing: the web assets are embedded
+# in it, and the cluster is reached over SSH or by a kubectl that lives on the
+# nodes. The image is the binary plus certificates, which is what makes it
+# around 12MB rather than the few hundred a base image would add.
+#
+#   docker build -t k3helper .
+#   docker run --rm -p 8787:8787 \
+#     -v "$PWD/targets.yaml:/targets.yaml:ro" \
+#     -v "$HOME/.ssh/id_ed25519:/key:ro" \
+#     k3helper serve -t /targets.yaml --host 0.0.0.0 --allow-remote
+#
+# Mount credentials at run time; never build them in. An image with a private
+# key in a layer keeps that key even after a later layer deletes it, and
+# anywhere the image goes, the key goes.
+
+FROM golang:1.26-alpine AS build
+WORKDIR /src
+
+# Dependencies first, so a change to the source does not refetch them.
+COPY go.mod go.sum ./
+RUN go mod download
+
+COPY . .
+ARG VERSION=dev
+# CGO off: the result has to run on scratch, which has no libc to link against.
+RUN CGO_ENABLED=0 go build \
+      -ldflags "-s -w -X github.com/solutionforest/k3helper/internal/cli.version=${VERSION}" \
+      -o /k3helper ./cmd/k3helper
+
+FROM scratch
+# Certificates, for the API server's TLS and for `bundle k3s` reaching GitHub.
+COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
+COPY --from=build /k3helper /k3helper
+
+# Nobody. There is nothing in this image to be root for, and a mounted key
+# should be readable by this user rather than by everyone.
+USER 65534:65534
+EXPOSE 8787
+ENTRYPOINT ["/k3helper"]
+CMD ["serve", "--host", "0.0.0.0", "--allow-remote", "--no-browser"]

Makefile

@@ -5,6 +5,12 @@ LDFLAGS    := -ldflags "-X $(VERPKG)=$(VERSION)"
 # -s -w strips the symbol table and DWARF: ~25% smaller downloads, and Go
 # panics keep their function names because the runtime carries its own tables.
 RELFLAGS   := -ldflags "-s -w -X $(VERPKG)=$(VERSION)"
+# The k3s release the sandbox and the E2E install. Pinned on purpose: a channel
+# is a lookup against update.k3s.io, and when that served a Traefik default
+# certificate from every one of its addresses it took every `curl -sfL
+# https://get.k3s.io | sh -` on the internet down with it — including CI's.
+# A pinned version fetches straight from the GitHub release.
+K3S_VERSION := v1.36.4+k3s1
 PLATFORMS  := linux/amd64 linux/arm64 darwin/arm64 darwin/amd64 windows/amd64 windows/arm64
 # Windows will not execute a downloaded file without the extension, so those
 # two assets carry .exe. Everything else stays extensionless; install.sh builds
@@ -157,6 +163,7 @@ sandbox-ssh:
 
 bootstrap:
 	go run ./cmd/k3helper vm setup -t $(TARGETS) \
+	  --k3s-version $(K3S_VERSION) \
 	  --server-extra-args "--snapshotter=native --disable=traefik" \
 	  --agent-extra-args "--snapshotter=native" \
 	  --kubeconfig sandbox-kubeconfig.yaml

README.md

@@ -15,6 +15,7 @@ k3helper is a portable k3s/kubernetes helper with a TUI.
   registry    configure the private registries the cluster pulls from
   ctx         list clusters defined in the targets file
   tui         interactive dashboard + resource browser
+  serve       the same, in a web browser (read-only)
 ```
 
 ## Why
@@ -86,6 +87,106 @@ Windows reaches clusters through a kubeconfig rather than over SSH — see
 reachable, but the host-layer commands (`vm setup`, `registry apply`) target
 Linux nodes.
 
+### Air-gapped nodes
+
+Nodes with no route to the internet cannot run the usual installer: it reaches
+update.k3s.io to resolve a channel, GitHub for the k3s binary, and a registry
+for every image a pod pulls. Build a bundle where there *is* a connection, then
+install from it:
+
+```bash
+# on a machine with internet (a laptop, or a jump host inside the network)
+k3helper bundle k3s --version v1.31.2+k3s1 --arch amd64 -o ./k3s-bundle
+#   k3s                        75MB
+#   k3s-airgap-images.tar.zst  184MB   ← imported into containerd on first start
+#   install.sh                 37KB
+#   all verified against the release sha256 manifest
+
+# from anywhere that can reach the nodes over SSH
+k3helper vm setup -t targets.yaml --bundle ./k3s-bundle
+```
+
+The nodes need nothing but SSH from wherever k3helper runs, and a route to each
+other. `--bundle` uploads the binary and the image archive to every node, puts
+them where the installer looks, and runs it with `INSTALL_K3S_SKIP_DOWNLOAD`.
+
+Two things worth knowing:
+
+- **Agents join on the address in your targets file**, not one discovered on
+  the server. `hostname -I` reports a cloud VM's public address first, which is
+  usually the one address an air-gapped network cannot use. Override it with
+  `--join-address` when k3helper reaches the nodes over one network and the
+  cluster talks over another.
+- **Workload images still have to come from somewhere.** The bundle covers the
+  cluster's own images; for yours, point the nodes at an internal registry with
+  a `registries:` block and `k3helper registry apply`.
+
+### Air-gapped kubeadm: a mirror, or your own connection
+
+k3s installs offline from a bundle because it is one binary and one image
+archive. kubeadm cannot: it needs apt packages and images from
+`registry.k8s.io`, and neither fits in a file you carry in. Two answers.
+
+**Point at your own mirror.** Most sites that run air-gapped Kubernetes already
+have one — Artifactory, Nexus, Satellite. This does not replace it, it points
+at it:
+
+```bash
+k3helper vm setup -t targets.yaml --distro kubeadm   --apt-mirror   https://nexus.corp/repository/ubuntu   --k8s-apt-repo https://nexus.corp/repository/kubernetes
+```
+
+The distribution archive is substituted in both source formats — 24.04's
+deb822 `.sources` files and older `.list` entries — and every file is backed up
+as `*.k3helper.bak` first. Third-party repositories are left alone.
+
+**Or lend the nodes your connection.** When there is no mirror either,
+`--via-proxy` opens a proxy on the machine running k3helper and reaches the
+nodes through the SSH connection already open to them:
+
+```bash
+k3helper vm setup -t targets.yaml --distro kubeadm --via-proxy
+```
+
+```
+--via-proxy: these nodes will reach the internet through this machine for the
+length of the install, and only through it.
+  allowed: 30 default hosts (distribution mirrors, pkgs.k8s.io, registry.k8s.io, docker.io)
+  the tunnel and its configuration are removed when the install finishes.
+```
+
+The proxy resolves names on your side, so the nodes need no working DNS either
+— which a genuinely cut-off machine does not have. Traffic is restricted to an
+allowlist; anything else is refused with the flag that would permit it
+(`--proxy-allow HOST`, or `--proxy-allow "*"`). Cluster-internal addresses
+never go through the tunnel.
+
+Three things to be clear about:
+
+- **This gives an isolated machine a route out.** It is temporary, proxied and
+  allowlisted, but in some environments opening one at all is a policy breach
+  regardless. It is off unless asked for, and it says what it is doing every
+  time. That call is the operator's, and sometimes not theirs to make.
+- **The tunnel is install-time only.** When `vm setup` finishes it is removed,
+  and the cluster goes back to having no internet — so it cannot pull a
+  workload image afterwards. For anything beyond the install, point the nodes
+  at an internal registry with a `registries:` block.
+- **For a genuinely disconnected site, prefer k3s.** `--bundle` installs it with
+  no network at all, which is a better fit than a cluster that needed a
+  temporary hole to be built.
+
+### If the k3s channel service is down
+
+`--k3s-version` pins an exact release and skips the channel lookup entirely:
+
+```bash
+k3helper vm setup -t targets.yaml --k3s-version v1.31.2+k3s1
+```
+
+This is not hypothetical. During live testing `update.k3s.io` served a Traefik
+default certificate from all three of its addresses, so every
+`curl -sfL https://get.k3s.io | sh -` on the internet failed TLS verification.
+Pinning a version fetches straight from the GitHub release and is unaffected.
+
 ### From source
 
 ```bash
@@ -440,6 +541,56 @@ A node it cannot reach is reported, never skipped — partial inspection must no
 1.  [85%]  Node unreachable over SSH (no evidence could be gathered)
 ```
 
+### 6b. The web interface
+
+Same data, same checks, same diagnosis — in a browser instead of a terminal:
+
+```bash
+k3helper serve                              # http://127.0.0.1:8787
+k3helper serve --port 9000                  # somewhere else
+k3helper serve --host 0.0.0.0 --allow-remote  # reachable from the network
+```
+
+![overview](docs/screenshots/web-overview.png)
+
+The Overview is the dashboard: one card per node, every host check with its
+verdict, and a banner that either says the cluster is healthy or how many
+issues are waiting in Doctor.
+
+![pods](docs/screenshots/web-pods.png)
+
+Pods, workloads, nodes and events are tables — filter by namespace in the
+header, and open logs or `describe` for any row. Doctor is the same ranked
+findings the CLI prints, each one expanding to its remediation.
+
+![doctor](docs/screenshots/web-doctor.png)
+
+Three things worth knowing:
+
+- **It is read-only.** Everything lists, describes or tails. Nothing applies,
+  deletes or restarts. A browser tab that can change a production cluster
+  deserves more thought about who is holding it than a localhost token
+  provides, and none of that is needed to make a cluster legible.
+- **It binds to loopback** unless you pass `--host` *and* `--allow-remote`. A
+  session token is the only authentication, and it is printed once, at
+  startup, fresh for every run.
+- **It is the same binary.** No Electron, no WebView2, no npm: the assets are
+  embedded, so `serve` works on Windows, macOS, Linux and in a container with
+  nothing else installed. It costs about 3MB.
+
+Running it in Docker, where `--host 0.0.0.0` is the usual answer:
+
+```bash
+docker run --rm -p 8787:8787 \
+  -v "$PWD/targets.yaml:/targets.yaml:ro" \
+  -v "$HOME/.ssh/id_ed25519:/key:ro" \
+  k3helper serve -t /targets.yaml --host 0.0.0.0 --allow-remote
+```
+
+The image is `FROM scratch` and around 17MB. Mount credentials at run time and
+never build them in: a key in a layer stays in the image even after a later
+layer deletes it.
+
 ### 7. TUI
 
 ```bash