homelab gitops

Single source of truth for the home k3s cluster. Everything that runs in the cluster is declared here; ArgoCD applies it. The two exceptions — cloudflared and Tailscale — run on the Synology on purpose, and are documented in synology/.

Internet → Cloudflare (DNS + TLS) → Tunnel → cloudflared on the NAS
                                                     ↓  :30080
                              Ubuntu VM ── Traefik Gateway ── HTTPRoute → app
                                                     ↑  :30081
                                              Tailscale (admin only)

Setting this up for the first time? TODO.md is the ordered checklist of everything still to be done, including the parts that happen outside this repo (Google Console, Cloudflare, Bitwarden, the NAS).

Layout

Path	What
bootstrap/	Bare-metal → running cluster. Two scripts, run once.
charts/web/	The generic web-app chart. Every web app is 12 lines of values on top of this.
infra/	Platform components, each an umbrella around an upstream chart.
clusters/home/	ArgoCD Applications (infra, ordered by sync-wave) and the ApplicationSet that discovers apps/.
apps/	One directory per application.
templates/web/	Copy-paste skeleton for a new web app.
synology/	What is configured on the NAS, outside the cluster.

---

Add a web app

cp -r templates/web apps/myapp
$EDITOR apps/myapp/{Chart.yaml,values.yaml,appset.yaml}   # 4 values marked CHANGEME
helm dependency update apps/myapp                         # writes Chart.lock
git add apps/myapp && git commit -m "feat: myapp" && git push

Within ~3 minutes the Application appears by itself. Nothing else is touched — not this repo's ArgoCD config, not the ArgoCD UI, not Cloudflare, not the NAS. That last part works because public DNS is a wildcard and cloudflared has a single permanent catch-all rule, so hostname routing lives entirely in the HTTPRoute that charts/web renders.

Then put its secrets in Bitwarden as myapp_<KEY> (see below).

Add a third-party app (LiteLLM and friends) — same shape, but Chart.yaml depends on the upstream chart instead of charts/web, and appset.yaml sets helmPrefix to that chart's name so Image Updater knows where its image values live.

---

Change a secret

Open Bitwarden → the homelab project → edit the value → save.

Nothing else. Within ~1 minute External Secrets rewrites the Kubernetes Secret and Reloader rolls the Deployment. No commit, no kubectl rollout restart.

Keys are named <app>_<KEY> — e.g. horpakjs_MONGO_URI — and charts/web strips the prefix on the way in. One shared project keeps ClusterSecretStore untouched when apps are added, because projectID is fixed per store.

To watch it happen:

kubectl get pods -n horpakjs -w
kubectl describe externalsecret -n horpakjs horpakjs

---

First-time setup

Phase 0 — things only you can do

1. Bitwarden Secrets Manager (a separate product from the password vault — ESO cannot read an ordinary vault). Create project homelab, a machine account with read access to it, and an access token. Note the organization and project UUIDs.

2. Load the seven values the app reads in as horpakjs_<KEY>, plus platform_GIT_USERNAME and platform_GIT_TOKEN (a GitHub PAT with contents: write on this repo, used by Image Updater to commit tag bumps).

3. Cloudflare: wildcard DNS *.1721611.xyz → tunnel; one catch-all tunnel rule → http://<vm-ip>:30080.

4. Tailscale on the NAS: advertise the VM subnet, approve the route.

5. Google Cloud Console → the OAuth client → Authorized redirect URIs. Add the production hostname before testing sign-in, or Google rejects the callback — it compares redirect URIs as exact strings:

   https://app.1721611.xyz/auth/spreadsheet/callback
   https://app.1721611.xyz/api/auth/callback/google
   

   There is no staging hostname: the app is deployed under its production hostname from the start and only starts receiving traffic when the Cloudflare route is switched over.

Remaining placeholders

Domain and Bitwarden IDs are already set. What is left needs facts that only exist once the VM and tunnel are created:

grep -rn 'CHANGEME' --include='*.yaml' --include='*.yml' .

File	Set
synology/cloudflared/config.yml	tunnel UUID and the k3s VM IP
clusters/home/*.yaml, clusters/home/infra/*.yaml	repoURL, if this repo is not hongjs/gitops

bootstrap.sh refuses to run while the domain or the Bitwarden UUIDs are still placeholders.

Phase 2 — bring up the cluster

# on a fresh Ubuntu 24.04 VM, as root
export K3S_S3_ENDPOINT=... K3S_S3_BUCKET=... K3S_S3_ACCESS_KEY=... K3S_S3_SECRET_KEY=...
./bootstrap/install-k3s.sh

git clone https://github.com/hongjs/gitops.git && cd gitops
make deps && git commit -am "chore: pin chart versions" && git push
export BW_ACCESS_TOKEN='0.xxxx.xxxx:xxxx'
sudo -E ./bootstrap/bootstrap.sh

Run make deps before the first bootstrap. Chart dependencies here are declared as version floors (>= 1.16.0), not exact pins, because they could not be resolved offline when this repo was written. make deps resolves each to the current release and records it in Chart.lock, which is what actually gets deployed. Review the lock diff before committing — that is the only moment anything upstream changes version.

---

Verify

kubectl get applications -n argocd                # all Synced / Healthy
kubectl get clustersecretstore bitwarden          # Valid
kubectl get gateway,httproute -A
curl -f https://app.1721611.xyz/api/health      # {"active":true}
curl -I https://argocd.1721611.xyz                 # must NOT reach ArgoCD

Admin UIs, tailnet only (these ports are never tunnelled):

ArgoCD	http://<vm-ip>:30081 with Host: argocd.internal
Traefik dashboard	http://<vm-ip>:30900/dashboard/

---

If the NAS dies

Recovery needs exactly two things: this repository and a Bitwarden login. There is no key file to lose — the deliberate reason this uses External Secrets rather than SOPS.

./bootstrap/install-k3s.sh
git clone https://github.com/hongjs/gitops.git && cd gitops
BW_ACCESS_TOKEN=... sudo -E ./bootstrap/bootstrap.sh
# repoint cloudflared at the new VM IP

Application data is not in the cluster and never has a PVC to lose: horpakjs keeps everything in MongoDB Atlas and Firebase, litellm in Neon. Managed cloud Postgres/Mongo, not CloudNativePG, is the default for a new app's database — a stateful workload inside k8s is the exception here, not the norm, so it stays out unless a future app has a concrete reason a managed free tier can't cover.

Drill it quarterly. A backup that has never been restored is not a backup:

multipass launch 24.04 --name dr-test --cpus 4 --memory 8G --disk 60G
# run the recovery above inside it; target is app serving 200 within 30 minutes
multipass delete dr-test --purge

---

Maintenance

make lint        # helm lint every chart
make template    # render everything; catches template errors
make deps        # resolve upstream chart versions (review the Chart.lock diff)
make diff        # what ArgoCD would change, needs cluster access

After editing charts/web, run make deps. Helm renders apps from the vendored dependency tarball, not from the live source, so a change to the shared chart is invisible — and silently deploys the old values — until each app's dependency is rebuilt. Bump charts/web/Chart.yaml's version at the same time so the Chart.lock diff shows the change happened.

Certificates. k3s certs last 12 months and rotate on restart once inside 90 days of expiry — a node that never restarts expires in place and locks you out, which is how the previous k3d cluster was lost. install-k3s.sh installs a monthly restart timer, plus k3s-cert-check for an independent warning:

systemctl list-timers k3s-cert-rotate.timer
k3s-cert-check 60

Uptime. Deliberately no in-cluster monitoring stack — hours of downtime are acceptable here and Prometheus is not worth the RAM on a NAS VM. Point a free external check (UptimeRobot, Cloudflare Health Check) at the public URL and send alerts to your phone. It must be external: monitoring that dies with the node tells you nothing.