Reach clusters through a kubeconfig, and ship a Windows .exe - #1
Merged
Merged
Conversation
…y SSH
Every layer below the cluster assumed SSH into a node and a kubectl living
there. That covers k3s and kubeadm and excludes every managed cluster, which
is what most clients run: EKS/GKE/AKS hand out a kubeconfig and keep the
machines. The cluster layer never needed the machines — only the transport
did.
A targets entry now carries either `nodes:` or `kubeconfig:`, and
`Targets.Mode()` says which. `internal/transport` is the one place that opens
connections, so a second way to reach a cluster cannot drift between the
fourteen call sites that used to dial for themselves.
The local transport parses rather than shells out. The codebase builds shell
command strings, and internal/ssh/local.go runs them through /bin/sh, which
Windows does not have. So kubectl.Local.Run asserts the command is one of its
own kubectl invocations, strips the trailing redirection, splits the arguments
honouring the single quoting kube.shellQuote produces, and execs kubectl
directly. Anything else — df, systemctl, free, the k3s probes — is refused
with exit 127. That refusal is the feature: a host probe fails where it is
written, with a reason, instead of somewhere further down the stack.
Layers are gated rather than attempted. check.HostChecks is the single list
both `check` and the TUI build from. Evidence.HostLayerUnavailable is
deliberately neither a ProbeError nor an UnreachableNode: both of those mean
"we tried and failed", which is a fault, while this means "there was never
anything to try", which is how managed clusters work. Conflating them would
report every healthy EKS cluster as degraded. `vm setup` and `registry apply`
refuse outright — a registry configured on some nodes and not others is the
exact failure the cluster-level registry block exists to prevent.
Two bugs this exposed, both fixed here:
- doctor exited 2 for every healthy managed cluster, because the scope note
counted as a fault. Findings that describe what could not be looked at now
print under "Notes on what was looked at" and do not set the exit code.
- kyaml.RemotePath hardcoded /tmp and would have built "\tmp\..." once
joined with filepath.Join from a Windows client deploying to a Linux node.
The separator now follows the target, not the host.
Proof: unit tests parse the real command strings from gather.go and deploy.go;
exec_test.go runs the whole path against a stub kubectl on PATH; the
integration suite reads the sandbox twice, once over SSH and once through its
kubeconfig, and asserts the two agree.
… three The binary cross-compiled for Windows already; nothing released it, and nothing had ever run it. Both halves are fixed here. Release: windows/amd64 and windows/arm64 join PLATFORMS. Those two assets carry .exe, because Windows will not execute a downloaded file without the extension — everything else stays extensionless, and install.sh builds the asset name the same way. The release workflow now asserts that every platform asset exists, is listed in checksums.txt, and that the Windows one is really a PE binary; a missing or misnamed asset otherwise surfaces as somebody's 404. install.sh recognises Git Bash, MSYS and Cygwin and prints the .exe URL rather than installing into a /usr/local/bin that means something different under each of them and is on none of their PATHs. CI: "single portable binary" is a claim about the runtime, not the build, so scripts/portable-check.sh runs the binary from a directory it has never seen and asserts it starts, explains a missing targets file, refuses a kubeconfig that is not there without writing anything, completes both init flows, generates and verifies YAML with no cluster and no network, and leaves nothing behind in $HOME. Linux, macOS and Windows all run that same script, so no platform is held to whatever its own job happened to test. `make portable-check` runs it locally. macOS had nothing in CI at all before this, which is awkward for the platform most operators drive a cluster from; it now gets vet, race tests and the portable check on every push. Windows gets the same, plus a second pass under PowerShell with native paths and no POSIX shell anywhere in the picture — that being the environment a client actually double-clicks into, and where a path check tends to quietly pass while being wrong.
Two asks from live use, both about reaching people the tool could not reach before: clients running managed Kubernetes, and clients on Windows. README gains a support column for managed clusters, a section on describing one by its kubeconfig, and Windows install instructions. The limitations list says plainly that a kubeconfig cluster has no host layer — that being the nature of a managed cluster rather than a gap here — and that kubeconfig mode needs kubectl on PATH. The roadmap loses its Windows entry, which this release closes.
…ssertion The Windows CI job added in the previous commit did its job immediately: it ran tests that had never run on that platform. Four failures, three kinds. A real fix: expandHome concatenated rather than joined, so "~/.kube/config" became "C:\Users\you/.kube/config". Go accepts the mixed form and kubectl would have too, but the path is printed in errors, and half-and-half reads like a bug. A test assumption: file modes are POSIX. Windows maps a mode onto the read-only attribute and reports 0666 for anything writable, so there is nothing to assert there — the manifest is still written under the user's own profile directory, which is where the protection comes from on that platform. A design boundary that had never been stated: internal/ssh's local transport runs commands through /bin/sh, which Windows does not have. Local mode means "manage the machine k3helper is on", and every command it runs is a Linux one aimed at a host running k3s — so it is Unix-only by design rather than unimplemented. It now says that, instead of failing with "exec: /bin/sh: executable file not found in %PATH%", which reads like a broken install. The tests that need the shell skip with the same explanation. Separately, `ctx` grew a REACHED column in the kubeconfig work and the E2E still matched the old two-column shape. The sandbox E2E also reported pod.crashloop undiagnosed within its 30s budget. Nothing here touches that signature or its evidence, and every other fault in the sweep was caught, so this push re-runs it before anything is changed on that account — a timing budget is not worth widening until it is clear it is timing.
CI caught it twice on this branch: doctor reported pod.not-ready where the sweep expected pod.crashloop. That is doctor being right. A crashlooping pod has to pull its image, run, exit, and be restarted enough times for kubelet to give up and back off before the status says CrashLoopBackOff; until then the pod really is Running and unready, and pod.not-ready is the honest finding. 30s was enough on a developer's machine and on a quiet runner, and not enough on a loaded one — it passed in one sweep and failed in the other within the same CI run. The oom fault is the same shape, a pod that keeps dying, and has had 45s all along. Nothing on this branch touches that signature, its evidence, or this budget; the kubeconfig work only made the flake show up twice in an afternoon.
…ring one Two CI runs disagreed about the same fault, which was the tell. When the sweep injected a crashlooping pod, doctor sometimes reported pod.crashloop and sometimes only pod.not-ready — and raising the settle budget from 30s to 45s did not fix it, because the problem was never how long we waited. A container in backoff is reported as CrashLoopBackOff only while it is waiting between attempts. The moment kubelet starts it again the pod is Running with no reason attached, and a diagnosis taken in that window saw an unready pod and nothing else. Which half of the cycle doctor lands in is chance, so the finding was chance too — on a real cluster as much as in the fault matrix. The restart count was already parsed and then dropped on the floor. It is now kept for pods that are Running and not ready, and pod.crashloop fires on either the waiting status or an unready pod that has restarted three or more times. Three, not one: a pod that fell over once and came back is not this finding, and kubelet's backoff has reached 40s by the third restart, so a pod at that count has been failing for the best part of a minute. A pod seen both ways counts once, or the confidence would double for a single crashlooping pod. The 45s budget from the previous commit stays. It was not what was wrong, but the oom fault is the same shape and has always had 45.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clients running managed Kubernetes could not use k3helper at all, and clients on Windows had no binary. Both are fixed here.
Managed clusters
Every layer below the cluster assumed SSH into a node and a kubectl living there. That covers k3s and kubeadm and excludes every managed cluster — EKS/GKE/AKS hand out a kubeconfig and keep the machines. The cluster layer never needed the machines; only the transport did.
A targets entry now carries either
nodes:orkubeconfig::doctor,deploy,verify --dry-run-server,genand the TUI (logs, describe, port-forward) all work there.internal/transportis the one place that opens connections, so a second way to reach a cluster cannot drift between the fourteen call sites that used to dial for themselves.The local transport parses rather than shells out. The codebase builds shell command strings, and
internal/ssh/local.goruns them through/bin/sh, which Windows does not have. Sokubectl.Local.Runasserts the command is one of its own kubectl invocations, strips the trailing redirection, splits arguments honouring the quotingkube.shellQuoteproduces, and execs kubectl directly. Anything else —df,systemctl, the k3s probes — is refused with exit 127. That refusal is the feature: a host probe fails where it is written, with a reason.Layers are gated, not attempted.
Evidence.HostLayerUnavailableis deliberately neither aProbeErrornor anUnreachableNode. Both of those mean "we tried and failed", which is a fault; this means "there was never anything to try", which is how managed clusters work. Conflating them would report every healthy EKS cluster as degraded.vm setupandregistry applyrefuse outright rather than doing half the job.Windows
k3helper-windows-amd64.exeand-arm64.exeare built bymake release, listed inchecksums.txt, and published with every tag. One file, no installer.install.shrecognises Git Bash/MSYS/Cygwin and prints the.exeURL instead of installing somewhere that is on nobody's PATH.CI
scripts/portable-check.shruns the binary from a directory it has never seen — it must start, explain a missing targets file, refuse a kubeconfig that is not there without writing anything, complete both init flows, generate and verify YAML with no cluster or network, and leave nothing in$HOME. Linux, macOS and Windows all run that same script. macOS had nothing in CI before this. Windows also runs a second pass under PowerShell with native paths and no POSIX shell involved.Bugs this surfaced
doctormissed crash loops it sampled between restarts. A container in backoff reads asCrashLoopBackOffonly while waiting; the moment kubelet restarts it the pod is Running with no reason attached, and the diagnosis saw nothing worse than an unready pod. Which half of the cycle doctor landed in was chance, so the finding was chance too — on real clusters as much as in the fault matrix, which is where two CI runs disagreed and gave it away. Restart counts were already parsed and dropped; they are kept now.doctorexited 2 for every healthy managed cluster, because the scope note counted as a fault.kyaml.RemotePathhardcoded/tmpand would have built\tmp\...once joined from a Windows client deploying to a Linux node.expandHomeconcatenated instead of joining, producingC:\Users\you/.kube/config.internal/sshlocal mode had an unstated boundary — it needs/bin/sh. It says so now instead of failing with "executable file not found in %PATH%".Verification
All five CI jobs green on 78ddc05; fault matrix 12/12. Integration tests read the sandbox twice, over SSH and through its kubeconfig, and assert the two agree.
No breaking changes — existing targets files keep working. Version bumped to 0.5.0; tagging
v0.5.0is what publishes the assets.