Drosera (sundew — the botanical codename line alongside lentago,
solidago, and kalmia) is the Lentago Labs observability suite. Today it
watches the on-prem estate (the machines in our own building) and renders AWS telemetry from the Lentago cloud
estate (Solidago) on demand via a live CloudWatch datasource; deeper AWS
ingestion is the next scope expansion. Renamed from homelab-observability on
2026-07-04. The runtime half of the rename is #169: the OIDC CI role, the
Terraform state key, LXC 105's checkout and all five bullpen workers' checkouts
carry the new name (the workers since their rebuilds in lentago/claytonia#65).
Authorship: The Terraform, Alloy config, dashboards, scripts, and documentation in this repo are co-written with Claude (Anthropic). I direct the work and review the output; Claude writes the code. I'm an infrastructure operator, not a software engineer — please don't read this repo as a portfolio of coding ability.
Agents / contributors: setup, architecture, and credentials are documented here. Conventions (LogQL, dashboard UIDs, node_exporter model, kiosk) are in CLAUDE.md; PR norms in CONTRIBUTING.md.
Git-driven observability for the on-prem network behind a Firewalla, powered by Grafana
Cloud (free tier) on the visualization
side. Ingestion is per-host, not a single chokepoint: each host runs its own
Grafana Alloy agent and pushes metrics
straight to Cloud (remote_write, 60s); a central Alloy on the LXC covers
what's left over — blackbox probes, the Home Assistant scrape, and one Loki
receiver (see Architecture below).
Everything is declarative:
- Dashboards live as JSON in
dashboards/. - Cloud-side resources (the
Lentagofolder, dashboards, datasources, alert rules, contact points) are managed by Terraform with thegrafana/grafanaprovider. - Central (LXC) ingestion is a single declarative
alloy/config.alloyspun up bydocker compose; per-host push agents run from the config embedded inscripts/deploy-alloy.sh.
This is a real everything-as-code observability stack: dashboards, alerts, and datasources are
Terraform, applied automatically on merge to main. Each pattern below links to the code that
proves it.
| Pattern | How it shows up here |
|---|---|
| Apply-on-merge IaC for a live SaaS surface — the merged PR is the change record | The apply job in .github/workflows/terraform.yml runs terraform apply -auto-approve on every push to main that touches terraform/ or dashboards/ — the Grafana Cloud stack is code, not clickable UI state. |
| Deadlock-safe required-check gating | The unfiltered pull_request trigger + changes filter job + gate job with if: always() in terraform.yml — a path-filtered heavy job can't hang a PR forever as an "Expected" check (rationale cites lentago/.github#27). |
| OIDC to AWS, no static credentials | terraform.yml requests id-token: write and assumes an IAM role for the S3 state backend — no long-lived AWS keys live in CI secrets. |
| Serialized apply via concurrency group | The apply job's concurrency: { group: terraform-apply, cancel-in-progress: false } in terraform.yml — two quick merges queue instead of racing on the same Terraform state. |
| Reusable CI shared across the fleet | docs-check.yml, shellcheck.yml and claude.yml are thin wrappers over uses: lentago/shared-workflows/...@v1.2.3 — one definition maintained centrally, consumed by every repo at an immutable semver tag (shared-workflows ADR-0005). |
| Branch protection as a ruleset, not ad-hoc settings | main requires the named checks gate, shellcheck / shellcheck, and docs-check / docs-check and allows squash merges only — enforced at the platform level, so merge is gated on CI, not convention. |
| Alerting-as-code with written design intent | terraform/alerts.tf provisions 66 rules across 9 groups (one from the uvularia-pipeline module); docs/adr/0001-...md records why these live in Grafana (not AWS) and why each group's no_data_state differs. |
| Cross-repo contract discipline | The Loki output contract and Solidago (AWS) contract sections name exactly which repo owns which label, field, and IAM role — explicit written boundaries instead of drift. |
These systems are real, and nothing critical rides on them. That makes this a safe place to try a change before you make the same kind of change in your own shop. Pick one:
1. Add or edit a Grafana dashboard as JSON. Drop or edit a dashboard file under
dashboards/ and reference it from a for_each map in
terraform/dashboards.tf (via locals.tf),
then open a PR. CI runs fmt/validate/plan and posts the terraform plan as a PR comment
for review; the gate check must be green to merge. On merge to main the apply job upserts
the dashboard into Grafana Cloud through the grafana/grafana provider — no manual Grafana UI
step anywhere in the loop.
Proof this works: #187 — feat(dashboard): Context Ledger section on Claytonia — Runner Fleet,
#171 — feat: add essexcrossingatmontserrat.com probe and dashboard,
#163 — Reorganize and rename dashboards along product lines.
2. Recover a live-only edit before apply-on-merge destroys it. Because merge re-applies
repo state, a dashboard hand-edited in the Grafana UI is un-codified work living on borrowed
time: the next unrelated merge's apply silently reverts it. That is exactly what happened in
2026-07 — a fleet-scoreboard revamp was pushed live via the API but never committed, and five
later bug-fix merges each ran the apply and stomped it. The fix is not to distrust apply-on-merge
but to make code the source of truth: pull the lost panel JSON out of Grafana's dashboard version
history, commit it into dashboards/ + terraform/dashboards.tf,
and merge — so future applies preserve the change instead of reverting it.
Proof this works: #119 — infra-health: restore fleet-scoreboard revamp lost to IaC drift stomp,
#120 — CLAUDE.md: add anti-drift rule for live dashboard edits.
3. Onboard alerting for a new signal. Add a rule group to
terraform/alerts.tf, choosing no_data_state deliberately per signal
type (an absent probe is an ambiguous vantage-point gap; an empty log stream is the failure —
opposite stances, both defensible), and record the rationale in an ADR under
docs/adr/. The PR triggers a plan; merge applies the rule group and, if new, the
contact-point wiring — the same apply-on-merge path as dashboards.
Proof this works: #188 — Add context-ledger alert rules,
#175 — Alert on ingest absence for the critical Loki streams,
#172 — feat: Grafana-native alerting for site probes.
Vectors 1 and 3 need no special access beyond opening a PR; humans own the policy and direct every change. Every directed pull request (one a person asked for) gets an AI review before it is armed to merge. The one exception is an opted-in scheduled routine, which merges when a fixed rule check on its author, title and changed paths passes; no routine is opted in here today (ADR-0014). Editing
the live Grafana stack directly (vector 2's failure mode) requires a Grafana Cloud account on the
lentago stack.
Node metrics push from each host (host-local Alloy → Mimir, 60s); the central Alloy on the LXC handles blackbox probes, the Home Assistant scrape, and a Loki receiver used only by the device-inventory feed (see below). Zeek/ACL logs bypass the central Alloy entirely — the Firewalla's Fluent Bit shipper (owned by lentago/betula) pushes those directly to Grafana Cloud Loki. Full rendered diagram, paths, and label conventions: docs/metrics-flow.md.
HOSTS ×6 (neptune, pve, pve2, pve3, pve4, pve5)
each: node_exporter:9100 → local Alloy ── remote_write 60s ─┐
├─▶ Grafana Cloud
CENTRAL ALLOY (LXC 105) │ (lentago)
blackbox ICMP/HTTP ───────────────── remote_write ─────────┤ ├─ Mimir (metrics)
Home Assistant /api/prometheus → HA scrape → remote_write ──┤ ├─ Loki (logs)
claytonia + cost-export → Loki receiver :3100 ──────────┘ └─ Grafana
│
Firewalla Fluent Bit (Zeek/ACL) ── direct Loki push, no relay ─────────┤
│
AWS / SOLIDAGO CloudWatch ◀── query-on-demand (assume-role) ── Grafana ─┤
│
public share ──▶ Office Display
Architecture decisions: the reasoning behind this layout — the Cloud + Alloy
- Terraform migration, host-local push, apply-on-merge, series-budget engineering, privacy-shaped egress, and the flat dashboard folder — is recorded as decision records in docs/adr/.
Firewalla's Fluent Bit shipper pushes Zeek/ACL logs directly to Grafana
Cloud Loki over HTTPS — no LAN relay, and the central Alloy on LXC 105 is not
in this path. It attaches the cluster="lentago-lab" label itself (a static
Labels directive in its Fluent Bit config), unlike the metrics side, where
the central Alloy's external_labels injects that label.
The eight active log streams, keyed by log_source (the four zeek_http/
zeek_files/zeek_notice/zeek_weird streams shipped in betula#58,
2026-07-02):
log_source |
Contents |
|---|---|
zeek_dns |
DNS query/response records from Zeek — domain, query type, client IP, answer. |
zeek_conn |
TCP/UDP connection summaries — src/dst IP and port, bytes, duration, state. |
zeek_ssl |
TLS handshake records — SNI, certificate subject, cipher, validation status. |
zeek_http |
Plain-HTTP request records — host, URI, method, status, user agent. |
zeek_files |
File-transfer analysis records — MIME type, source, hashes where computed. |
zeek_notice |
Zeek notices — the engine's own alerts/anomalies. Sparse and bursty (~150–200 lines/day measured 2026-08-09), so it carries no ingest-absence alert; see terraform/alerts.tf. |
zeek_weird |
Protocol-anomaly records ("weirds") — steady low-volume background (~10k lines/day). |
firewalla_acl |
Firewalla ACL alarm events — blocked/allowed flows, rule name, severity. |
Beyond the Firewalla streams, the change pipeline (ADR-0010)
reads and writes these, all with cluster="lentago":
log_source |
Contents |
|---|---|
github_actions_run / github_actions_job |
betula's GitHub Actions feed (clients/github, betula#113): one line per completed run attempt and per job. |
drosera_live, betula_live, sites_live |
live events (pipeline="change", stage="live", contract in ADR-0010): the SHA each surface runs, every reconcile tick, from alloy-host/gitops-pull.sh, betula's gitops-sync.sh (betula#132) and the sites probe (#268). The "By repo" row of dashboards/change-pipeline.json is the query that keys on those two labels. |
change_pipeline_state |
The pipeline producer's write-back: one line per repo and stage every 30 s, one stream per repo. It carries no pipeline/stage labels, so the live selector never picks it up. |
Adding a new log_source stream — checklist:
- Measure its 24h volume (e.g.
sum(count_over_time({log_source="X"}[24h]))). - Add a row to the table above, the stream to
EXPECTEDinscripts/check-loki-labels.sh, and the value to theCLAUDE.md§ Loki labels list. - Add a
local.loki_ingest_streamsentry interraform/alerts.tfwith a window sized from the measurement — or record why it has none, aszeek_noticedoes (sparse and bursty, so a quiet day is a real zero).
Change coordination: the Firewalla side of this pipeline — the Fluent Bit
config, its direct-to-Loki output block, and the log_source label scheme —
lives in lentago/betula (renamed from
firewalla-axiom-pipeline 2026-07-04; betula#82, merged 2026-07-09, also
dropped its former parallel Axiom output, so Grafana Cloud Loki is now its
sole destination). Any change to the log_source values or the Loki output
must be coordinated with that repo (see its issue #42, which shipped the
current label scheme) so both sides stay in sync.
Triage pointer: if Zeek/ACL dashboards go dark, look on the Firewalla /
betula side first (the Fluent Bit container, its Loki output config, box
network egress) — the central Alloy on LXC 105 no longer relays this traffic,
so it's not a suspect for these four streams. It remains the right place to
look for device_inventory gaps (below) and for the metrics pipeline.
Terraform provisions the stack's alerting end to end —
terraform/alerts.tf and the uvularia-pipeline module: 66 rules across nine
rule groups (eight in alerts.tf) in the Lentago folder, plus the stack's first contact point. See
docs/adr/0001-grafana-native-alerting-for-site-probes.md
for why this lives in Grafana instead of AWS, and
docs/adr/0008-site-availability-slos-and-burn-rate-alerts.md
for the SLO targets and burn-rate design.
Site probe alerts(8 rules — "Site down" onprobe_successand "TLS cert expiring" onprobe_ssl_earliest_cert_expiryunder a 21-day threshold, for each of the 4 site dashboards). These metrics come from the on-prem Alloy's blackbox exporter and exist nowhere in CloudWatch, so Solidago's AWS-native alerting (CloudWatch alarms → SNS, solidago ADR-0001) can't see them.no_data_state = "NoData"deliberately: the probes run from one on-prem vantage point, so an on-prem/WAN outage looks identical to a real site outage, and alerting on NoData would misfire on every on-prem hiccup instead of a real one.Loki ingest absence(9 rules — one each forzeek_dns,zeek_conn,zeek_ssl,zeek_http,zeek_files,zeek_weird,firewalla_acl,device_inventory, andgithub_branch_head, the liveness signal of betula's GitHub collector, which the Change pipeline and CI runners groups read withno_data_state = "OK"; #292): fires when a stream goes quiet longer than its expected cadence (30m–2h depending on measured volume; see the per-stream comments interraform/alerts.tf).no_data_state = "Alerting"here — the opposite of the probe rules — because an empty Loki result is the failure this group exists to catch, not an ambiguous vantage-point gap.zeek_noticecarries no absence alert (sparse, bursty by nature — a quiet day is a real zero).Site SLO burn rate(6 rules, issue #195 — a fast/slow pair for each of the three public sites: lentago.dev, icecreamtofightwith.com, pondviewlane.com). These are objective alerts, not symptom alerts: they fire on how fast the 99.9% / rolling-30d availability error budget (43.2 min/30d) is being consumed. Each is multi-window (Google SRE Workbook) — a long window sets sensitivity, a short window forces a fast reset. Fast burn pages at ≥14.4× (1h ∧ 5m; budget gone in ~50h); slow burn tickets at ≥3× (24h ∧ 2h). Burn rate is computed inline withavg_over_time— no recording rules, zero new Mimir series.no_data_state = "NoData"as with the probe group (same single-vantage reason). Operator view: the Sites — SLOs & Error Budget dashboard (uidsites-slo-error-budget).Bullpen liveness(3 rules, issue #207 — runner-fleet liveness + retry, the game-day #1 gap). Built on the sameevent="job_running"heartbeat the Claytonia — Runner Fleet dashboard renders: workers below full strength tickets when fewer than 5 distinct workers heartbeat for 5m, a.retryrequeue tickets on any reaper-driven requeue (a worker died mid-job and recovery self-healed silently), and a fully-dark fleet pages when zero workers heartbeat for 10m (no_data_state = "Alerting"— an empty result is the failure, as with the ingest-absence group). Loki-sourced, zero new Mimir series. Caveat: job_running marks busy workers, not idle-but-alive ones, so the two headcount rules can false-fire on a legitimate idle stretch until claytonia ships an always-onworkers/<host>.aliveheartbeat to Loki (boundary: drosera owns the rules, claytonia owns the telemetry). See ADR-0009.Uvularia pipeline — lentago(4 rules, issue #218 — from theuvularia-pipelinemodule, the same one clients apply into their own stacks): served digest behind published (the Ask function reported an older digest after the latest publish, for 30m; #237), Ask function silent (no served/asked event in 12h), obligation went amber or red on a publish, and the Ask daily cap nearly spent (under 20% left, or a turn refused for the cap). Loki-sourced overlog_source=uvularia_*,cluster="lentago". Each query returns a series only when something is wrong, sono_data_state = "OK". Contact point and routing are theuvularia_contact_point/uvularia_repeat_interval/uvularia_group_byvariables; the contact point defaults toSite probe email. See docs/clients/uvularia.md.Change pipeline(2 rules, issue #270, ADR-0010): change stuck when a repo's newestmaincommit (github_branch_head) has noliveevent carrying its SHA after the surface's propagation budget (ADR-0010 defines both: theliveevent is what each runtime emits with the SHA it runs, and the budget is how long a merge may take to get there): 10 min for gitops and terraform repos, 20 min for sites. One alert instance perrepo/sha. A terraform repo only counts when a terraform push run exists for the head, because every apply workflow is path-filtered and a docs merge applies nothing.no_data_state = "OK". The repo lists are locals interraform/alerts.tf.CI runners(1 rule, issue #292): self-hosted job queued when a self-hosted Actions job has waited more than 15 min for a runner, which means the runner is offline or wedged. It reads betula'sgithub_actions_queuedsample (one line per queued job per 5-min tick). One alert instance per job, labelled withrepo,workflow,job_nameandhtml_url.no_data_state = "OK": the query returns a series only for a job over budget. The query lives interraform/logql/and is tested against a real Loki by thelogql-testworkflow.- Contact point: one email contact point (
Site probe email), reused by every group. The recipient isTF_VAR_alert_email, a sensitive Terraform variable with no default, supplied via CI/.envrcand never committed (this is a public repo). Routing is scoped per-rule, so it doesn't touch the stack's root notification policy.
Workflows and serverless functions can't run Alloy, so drosera ships two tiny
emitters that push one structured event straight to a Grafana Cloud Loki
push endpoint: the loki-event
composite action (lentago/drosera/.github/actions/loki-event@main) and
clients/loki_push.py for functions. Both send the same
labels — log_source=<source>_<stage>, cluster=<org>, plus source,
pipeline, stage, repo — with the payload as a one-line JSON log. First
consumer is uvularia, pushing into each client's own Grafana Cloud stack.
Token setup and label rules: clients/README.md.
The uvularia pipeline pane reads those events: the
Uvularia — Records pipeline dashboard
(dashboards/uvularia-pipeline.json,
uid uvularia-pipeline) plus three alert rules, packaged as one Terraform
module, terraform/modules/uvularia-pipeline/.
This stack runs it for the demonstration client (cluster = "lentago", wired
in terraform/alerts.tf); a client runs the same module
from their own repo into their own free-tier stack, and nothing of theirs is
hosted here. How a client applies it, and the payload fields each stage must
send: docs/clients/uvularia.md.
The Change — Pipeline dashboard already draws a per-repo stage strip (the
collapsed "By repo" row, Graphviz panel). The same pipeline is also available
as data: pipeline-producer/ is a stdlib Python
service on LXC 105 (drosera-pipeline.service, #266). Every 30 s it reads the
Actions feed and the live events for the last 24 h, computes the six
ADR-0010 stages for each repo once, and does two things with the result:
- Serves
pipeline.json(schema 1, pinned in #266) on:8611. pub's Caddy proxieshttp://pub.lan/viewport/pipeline.jsonto it (that route lives in kalmia), and the brasenia wall pane reads it there. LXC 105 doesn't mount the web share, so the producer serves the document instead of dropping a file, and nothing on pub needs a Loki token. - Writes
change_pipeline_stateback to Loki: one line per repo and stage. The stuck alert can then be a one-line query, and the "By repo" strip can stop re-deriving state in LogQL (a follow-up, once the stream has a day of data).
It needs its own Loki read token. The collectors' token in
/opt/drosera/.env is logs:write only, and a read with it
gets 401. Mint a logs:read access-policy token on the lentago stack and
pass it to deploy.sh as DROSERA_LOKI_TOKEN.
The script refuses to start the service without it. Setup, state rules and
the document schema: pipeline-producer/README.md.
Solidago platform metrics render in this stack via a query-on-demand
CloudWatch datasource — nothing is streamed or imported into Mimir, so it
consumes zero free-tier active series. Queries bill as CloudWatch
GetMetricData at render time; the dashboard refresh floor is 1m.
- This repo owns: the
solidago-cloudwatchdatasource (terraform/datasources.tf, "Grafana Assume Role" auth) and the Solidago dashboards (dashboards/solidago-platform-health.json, titledSolidago — Platform Health). - Per-site dashboards sit in the same flat
Lentagofolder, titledSites — <domain>(dashboards/site-*.json, uid = site repo name): outside-in blackbox probes (Mimir) on top, per-TargetGroup / per-ECS-service CloudWatch below. ALB panels use SEARCH expressions on stable name fragments — TargetGroup and LoadBalancer dimension hashes rotate on the nightly DR rebuild, so pinned dimension values break every night. Site dashboards refresh at 5m (not the 1m floor) since they're viewed on demand, not kiosked. Onboarding a new site: one probe target inalloy/config.alloy, one JSON, onesites_dashboardsentry — existing sites untouched. - lentago/solidago owns: the IAM
role
solidago-dev-grafana-cloudwatch(modules/grafana-cloud), its trust and permission policies, and the External ID plumbing. - Coordination rule: renaming the role, changing auth, or widening its
policy is a cross-repo change — same discipline as the
log_sourcelabel contract above. - Overnight gaps are the DR drill, not an outage: solidago tears down and rebuilds nightly; role ARNs are deterministic, so the datasource never needs re-pointing, and "No data" while the platform is down is correct behavior.
- Alerting stays AWS-native (CloudWatch alarms → SNS; solidago ADR-0001) — Grafana is visualization only.
- Known health-check quirk: the datasource's Save & test / health endpoint
probes CloudWatch Logs as well as metrics, so it reports
AccessDeniedException … logs:DescribeLogGroupsalongside "Successfully queried the CloudWatch metrics API". That is expected, not a fault — the role is metrics-only by design (logs belong to Axiom/betula, per ADR-0001's boundary). Metrics green = healthy. Verified end-to-end 2026-07-04 (assume-role query returned live ALB series for all three target groups).
.envrc.example # template for direnv — copy to .envrc, fill in secrets
docker-compose.yml # spins up Alloy on the LXC (one service)
alloy/
config.alloy # Alloy collector config (scrape + push + receive)
blackbox.yml # blackbox prober module definitions
ha_token.example # template; real ha_token is gitignored
alloy-host/ # gitops deploy for the central Alloy on the LXC
gitops-pull.sh # 5-min pull + validate + reload-on-drift
alloy-gitops.{service,timer} # systemd units (bootstrap-only)
README.md # bootstrap + ops
dashboards/ # source of truth for Grafana dashboard JSON
network-overview.json
dns-security.json
traffic-devices.json
infra-health.json
office-display.json
neptune-nas.json # Neptune NAS real-time activity (CPU/disk/net/RAID/temps)
solidago-platform-health.json # Solidago (AWS) via the CloudWatch datasource
site-*.json # per-site health (Mimir probes + per-TG/service CloudWatch)
uvularia-pipeline.json # Uvularia — Records pipeline (Loki events, cluster variable)
change-pipeline.json # Change — Pipeline: merge-to-live per SHA (Actions feed + live events, ADR-0010)
# + collapsed "By repo" row: one Graphviz stage strip per repo (grafana-graphviz-panel)
github-actions.json # GitHub — Actions: fleet-wide run history, failures, durations, queue time (Actions feed, #234)
terraform/ # manages Cloud-side resources
*.tf # incl. datasources.tf (solidago-cloudwatch) and
# alerts.tf (site probe + Loki ingest-absence +
# bullpen + on-prem availability alert rules)
modules/uvularia-pipeline/ # uvularia dashboard + rules; estate and clients both use it
pipeline-producer/ # change-pipeline producer on LXC 105: pipeline.json on :8611 + change_pipeline_state (#266)
pipeline_producer/ # stdlib Python 3.10 package (read Loki → compute stages → serve + write back)
systemd/ deploy.sh # drosera-pipeline.service + idempotent installer
tests/ # unit tests against a fake Loki
clients/ # source-neutral client emitters (#131)
loki_push.py # stdlib-only Loki push for serverless functions
README.md # token setup + label discipline
docs/adr/ # architecture decision records (e.g. native alerting for site probes)
docs/clients/uvularia.md # client export: the uvularia pane into a client's own stack
scripts/
inventory-cloud.sh # snapshot current state of lentago.grafana.net
deploy-node-exporter.sh # install node_exporter on a host
deploy-alloy.sh # install a host-local Alloy push agent (60s remote_write)
.github/actions/
loki-event/ # composite action: one event from a workflow to Loki
.github/workflows/
terraform.yml # fmt/validate/plan on PR
loki-event-test.yml # loki-event + loki_push.py against a mock Loki
pipeline-producer-tests.yml # pipeline-producer unit suite on Python 3.10
- Stack service account (drives Terraform + future CLI tooling). In
https://lentago.grafana.net→ Administration → Users and access → Service accounts, create a service account namedterraform-iacwith role Admin, then Add token and copy the value. This becomesGRAFANA_AUTH. - Access policy tokens (drive Alloy remote_write / log push). At
https://grafana.com→ My Account → Access Policies, create one policy with scopesmetrics:writeandlogs:writefor thelentagostack. Generate a token; copy the username (a numeric stack ID per signal type) and token and the push URLs from the stack details page.
Drop all of the above into .envrc:
cp .envrc.example .envrc
$EDITOR .envrc # fill in real values
direnv allowcd terraform
terraform init
terraform plan # should show: import the Lentago folder + 5 dashboards, then update them
terraform applyState lives in S3 (solidago's solidago-tfstate-365184644049 bucket, versioned +
encrypted) — no local-state backup step needed. Day to day you don't run apply
by hand at all: merging dashboard/terraform changes to main auto-applies via
the terraform workflow. See terraform/README.md § State and
§ CI.
The first apply rewrites datasource UIDs in each dashboard from loki /
prometheus (the old self-hosted UIDs) to grafanacloud-lentago-logs /
grafanacloud-lentago-prom (the lentago stack's auto-provisioned UIDs).
The original JSON files in dashboards/ are not modified — the rewrite happens
in-memory at apply time via replace() in terraform/locals.tf.
The Cloud migration wizard created loki and prometheus datasources pointing
at the self-hosted LXC, which Grafana Cloud cannot reach. After terraform apply repoints dashboards to grafanacloud-*, delete the orphans:
for uid in loki prometheus; do
curl -sS -X DELETE \
-H "Authorization: Bearer $GRAFANA_AUTH" \
"$GRAFANA_URL/api/datasources/uid/$uid"
done# On the LXC (or any host with docker), in this repo:
cp alloy/ha_token.example alloy/ha_token
$EDITOR alloy/ha_token # paste your HA long-lived access token (no trailing newline — use printf '%s' "<token>" > alloy/ha_token to be safe)
docker compose up -d
docker compose logs -f alloy # confirm "remote_write succeeded" etc.
open http://<lxc-ip>:12345 # Alloy debug UIAfter Alloy is up:
- Confirm logs arrive in Cloud: Explore → grafanacloud-lentago-logs →
{cluster="lentago-lab"}. - Confirm metrics arrive: Explore → grafanacloud-lentago-prom →
up{cluster="lentago-lab"}should return 1 for each scrape target. - Visit a dashboard (e.g. Lentago / On-prem — Network Overview) and confirm panels render data.
Grafana Cloud doesn't allow anonymous viewers. Use public dashboard sharing:
- Open
firewalla-office-displayin the Cloud UI. - Share → Public dashboard → Enable.
- Copy the public URL and point the kiosk Chromium at it.
The public URL bypasses auth for that one dashboard only; nothing else in the stack is exposed.
Add repository secrets (Settings → Secrets and variables → Actions,
not Dependabot) so the plan and apply jobs can authenticate to Grafana:
GRAFANA_URL— full stack URL, e.g.https://lentago.grafana.net(no trailing slash).GRAFANA_AUTH— the same Grafana Cloud service account token you use in.envrcasGRAFANA_AUTH/GRAFANA_SA_TOKEN.
Both must be non-empty. If either is missing, the job fails immediately with a clear log message instead of a misleading Terraform provider error.
AWS (for the S3 state backend) is reached via GitHub OIDC — no AWS secrets are
stored; the jobs assume drosera-github-actions-terraform in account
365184644049. The workflow runs fmt -check + validate + plan (posted as a PR
comment) on every PR touching terraform/** or dashboards/**, and apply -auto-approve on push to main — so merges deploy automatically. See
terraform/README.md § CI.
- Edit the JSON file in
dashboards/(or edit in the Cloud UI and copy the exported JSON back into the file). - Open a PR — CI posts the
terraform planas a comment. Merging tomainauto-applies. (Out-of-band you can stillcd terraform && terraform apply.)
- Drop a new JSON file in
dashboards/. Title it<Group> — <What>(On-prem — …,Claytonia — …,Solidago — …,Sites — <domain>): every dashboard lives in one flatLentagofolder, so the title prefix is the only thing that groups the list. - Add an entry to the matching per-group local (
lab_dashboards,claytonia_dashboards,solidago_dashboards, orsites_dashboards) interraform/locals.tf. These maps differ in how the JSON is pre-processed, not in where the dashboard lands. - Open a PR; merging to
mainapplies it (orterraform applylocally).
Edit alloy/config.alloy, open a PR, merge. The central
Alloy on the grafana-stack LXC is gitops-managed: alloy-gitops.timer
pulls origin/main every 5 minutes, validates with alloy fmt, and reloads the
collector on drift (SIGHUP for alloy/ changes, docker compose up -d for
compose changes). No hand-editing on the LXC — see
alloy-host/README.md. To deploy immediately instead of
waiting for the timer: systemctl start alloy-gitops.service on the LXC.
Bare-metal node_exporter on each host is the metric source. Run
scripts/deploy-node-exporter.sh against any new host you add. How those
metrics reach Grafana Cloud depends on the collection model below.
On hosts with lvm2 (the five Proxmox nodes) the script also installs
lvm-thin-textfile, a one-minute root timer that writes lvm_thin_pool_*
and lvm_vg_free_bytes into node_exporter's textfile directory. The thin pool
and root filesystem rules in the "On-prem availability" group read them. Re-run
the script on each node after changing it, and before merging rule changes
that depend on new series, or they start in NoData.
There are two ways a host's node_exporter metrics get to Mimir, and the repo
supports both side by side with identical labels (job="node",
instance="<host>"), so dashboards never care which is in use:
-
Central pull (default/legacy): the Alloy on LXC 105 scrapes the host's
:9100over the LAN. Add the host to theprometheus.scrape "node"target list inalloy/config.alloy. -
Host-local push (standardized): the host runs its own Alloy that scrapes
localhost:9100andremote_writes to Mimir at 60s (#239 — faster cadences bill as extra series). Buffers across network blips, and each host owns its own shipping. Deploy withscripts/deploy-alloy.sh:source .envrc # exports GRAFANA_CLOUD_METRICS_* # on the target host (root, or via sudo): ./deploy-alloy.sh <instance-label> # e.g. neptune, pve, pve3
The script installs Alloy from the Grafana apt repo, writes
/etc/alloy/config.alloy(canonical config embedded in the script) and a0600 /etc/default/alloyholding the push token, then enables the service.
Don't run both for the same host — that double-counts series. When you move
a host to push, delete it from the central prometheus.scrape "node" block.
The rollout target is neptune + all five Proxmox nodes on push; HAOS stays on
the central HA /api/prometheus scrape (it can't run a system Alloy).
The Claytonia — Runner Fleet dashboard has a live "Fleet stream of
consciousness" panel that shows each running bullpen job's reasoning in
near-real-time (each assistant turn's narration + the tools it fired). The
source is a worker-local Alloy agent on each runner that tails Claude
Code's session transcript as it is written and ships it to Grafana Cloud Loki
as job="claude_transcript" (labels: project, worker, session_id).
Gemini CLI and Codex CLI jobs ship on the same stream with runtime and
runid labels added. The agent tails them through the labelled links that
claytonia writes under /home/claude/.claytonia/live/<runtime>/<project>/<runid>/,
and only for checkouts under /home/claude/work/
(#305).
scripts/tests/transcript/run.sh runs the config against a throwaway Loki
(CI: transcript-test).
Deploy it per worker with
scripts/deploy-runner-transcript-alloy.sh
(canonical config: alloy/runner-transcript.alloy):
source .envrc # exports GRAFANA_CLOUD_LOGS_*
# on the worker (sudo-capable), from a repo checkout:
./scripts/deploy-runner-transcript-alloy.sh # worker label is auto (from hostname)It runs as a dedicated alloy-transcript.service under the claude user (the
session files are 0600 claude:claude). Egress is deliberately scrubbed
(see #71): only
assistant text + tool names are shipped — thinking blocks, tool inputs,
and user/tool-result lines (raw repo contents) never leave the LAN. A runid
label is a possible future bullpen-side fast-follow.
Dashboards that show raw LAN source IPs (id_orig_h) resolve them to device
names by joining against a device-inventory log stream,
log_source="device_inventory". Grafana Cloud runs queries server-side and
cannot reach the LAN, and LAN topology must not be published to GitHub — so the
name↔IP mapping goes only to Grafana Cloud Loki, the trust domain that already
holds every LAN IP via the Zeek streams (see
#113, ADR-0006). Since #243 it
is pushed directly over HTTPS, like the Zeek/ACL streams; until then it
relayed through the central Alloy's :3100 receiver. That receiver
(loki.source.api "firewalla" in alloy/config.alloy)
is not dead config: it still serves the claytonia workers' cr-loki.sh
events (job="claude_runner", which the Bullpen liveness alerts read) and the
workstation claude-cost-export timer (job="claude_local", see
claude-cost-export/).
The producer is betula's Firewalla device-inventory collector
(lentago/betula scripts/device_inventory_publish.sh,
moved from this repo in 2026-10 — betula#115, #151, #243: capture is betula's
side of the boundary). It runs on the Firewalla box (pi user, hourly
from betula's cron/user_crontab, straight from betula's gitops clone). It reads the box's
own device inventory from local redis (host:mac:* hashes) and pushes one
record per (device, IP) pair straight to Grafana Cloud Loki, with the same
endpoint and GRAFANA_CLOUD_LOGS_* credentials Fluent Bit uses. It sets
cluster="lentago-lab" itself.
Stream schema — one Loki stream per (device, IP):
| Field | Value |
|---|---|
label log_source |
device_inventory |
label dev |
<name>|<ip> — load-bearing (see below) |
| line body (JSON) | {"name":"…","ip":"…","mac":"…","family":"4"|"6","source":"firewalla-redis"} |
Display name is the redis name field, else bname, else the MAC. Both the
IPv4 (ipv4Addr) and every IPv6 (ipv6Addr array) address get their own row.
The dev label contract: the <name>|<ip> shape lets a dashboard build a
template variable with label_values({log_source="device_inventory"}, dev) and
regex /(?<text>[^|]+)\|(?<value>.+)/ — the dropdown shows device names while
the variable value stays the raw IP that id_orig_h=~"$device_ip" needs. Any
| in a device name is stripped before composing the label so the split stays
unambiguous.
The schema above is the interface drosera consumes — the dashboards' IP→name
joins, the dev template variable, and Ingest absence — device_inventory
depend on it. Changing it takes a matching change on both sides.
Deployment is betula's: a merge to betula's main reaches the box through its
gitops loop, with no deploy step here. Smoke-test on the box without pushing:
ssh pi@firewalla.local 'DRY_RUN=1 ~/.firewalla/firewalla-axiom-pipeline/scripts/device_inventory_publish.sh | head'Volume is negligible (~190 devices, hourly; logs not metrics, so it does not touch the active-series cap).
Grafana Cloud free tier caps active series at 15,000. Home Assistant's
/api/prometheus export is the single biggest consumer — it emits a series
for nearly every entity plus per-entity change-counters, "last updated"
timestamps, availability flags, and *_created markers. None of that is used
by any dashboard, so the prometheus.relabel "ha_trim" component in
alloy/config.alloy drops those families before
remote_write, reclaiming ~5.8k series. The real numeric HA metrics
(temperatures, fan RPM, battery, climate, brightness, sensor states) are kept.
The same component also drops the HA UGreen integration's sensor.ugreen_nas_*
series (about 60, #330): Neptune's own node_exporter already covers the NAS. The
three that feed the Neptune NAS dashboard are kept: sensor.ugreen_nas_device_fan
and sensor.ugreen_nas_pool_1_disk_{1,2}_temperature. To chart another NAS
sensor, add it to the keep regex in ha_trim.
ha_trim also drops every series for the phone notification sensors
(sensor.*_last_notification, sensor.*_last_removed_notification,
sensor.*_active_notification_count) and for sensor.sms_inbox. Those carry
the text of incoming messages, which must not leave the house (ADR-0006,
privacy-shaped egress). The names come from lentago/epigaea's
packages/sms_inbox.yaml, which also filters them out of HA's Prometheus
export, so this rule is the second line. If epigaea renames those entities,
change the drop regex here in the same change.
Check current usage with the grafanacloud-usage datasource:
grafanacloud_instance_active_series. If you add a host (each node_exporter is
~1.3k series), watch the headroom — trim more (e.g. node_exporter discard/flush
families) or the rollout will hit err-mimir-max-active-series.
scripts/check-loki-labels.sh queries Loki for active log_source values over
the last 24h and diffs them against the expected set. That set is the eight
Firewalla streams (zeek_dns, zeek_conn, zeek_ssl, zeek_http,
zeek_files, zeek_notice, zeek_weird, firewalla_acl) plus the change
pipeline's drosera_live, betula_live, sites_live, github_actions_run,
github_actions_job and change_pipeline_state. Run it manually as a sanity
check, or wire it to a cron / GitHub Actions schedule to alert on silent log
streams:
source .envrc
./scripts/check-loki-labels.shExits 0 when all expected values are present; exits 1 and prints the missing names to stderr otherwise.
The original incarnation of this repo ran Loki, Prometheus, Grafana, and
blackbox-exporter all on a single LXC via docker compose, with file-based
provisioning. That worked but had three pain points:
- Storage on the LXC — Prometheus TSDB + Loki chunks meant disk pressure and another thing to monitor.
- Dashboards drifted between repo and UI — anything edited in the UI was lost on the next provisioner reload.
- No upgrade story for Grafana itself — each
docker compose pullwas a gamble.
Migrating to Grafana Cloud free tier + Alloy fixes all three: storage moves to the Cloud's free 50GB logs / 10K active series allotment; Terraform plus checked-in dashboard JSON makes the repo the source of truth (UI edits get overwritten on next apply); and Cloud handles Grafana upgrades.
MIT.
See CONTRIBUTING.md.
🌱 Lentago Labs is a pro-bono operations practice for organizations that run on volunteers, donations, and one overworked tech person. Everything here is free to take, and we practice what we publish: our own estate runs this way, in the open. Start at the org profile.