Skip to content
lentagoPublic

About

Drosera — the Lentago Labs observability suite. Git-driven, fully declarative telemetry into Grafana Cloud (Mimir + Loki, free tier): per-host Grafana Alloy pushes metrics + logs with no central chokepoint, and AWS CloudWatch feeds in by assume-role. Terraform-provisioned dashboards, alert rules, and contact points; node_exporter, LogQL.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

drosera — Alerts and status · git-driven, no chokepoint

main License

Grafana Terraform Alloy Prometheus

drosera

Drosera (sundew — the botanical codename line alongside lentago, solidago, and kalmia) is the Lentago Labs observability suite. Today it watches the on-prem estate (the machines in our own building) and renders AWS telemetry from the Lentago cloud estate (Solidago) on demand via a live CloudWatch datasource; deeper AWS ingestion is the next scope expansion. Renamed from homelab-observability on 2026-07-04. The runtime half of the rename is #169: the OIDC CI role, the Terraform state key, LXC 105's checkout and all five bullpen workers' checkouts carry the new name (the workers since their rebuilds in lentago/claytonia#65).

Authorship: The Terraform, Alloy config, dashboards, scripts, and documentation in this repo are co-written with Claude (Anthropic). I direct the work and review the output; Claude writes the code. I'm an infrastructure operator, not a software engineer — please don't read this repo as a portfolio of coding ability.

Agents / contributors: setup, architecture, and credentials are documented here. Conventions (LogQL, dashboard UIDs, node_exporter model, kiosk) are in CLAUDE.md; PR norms in CONTRIBUTING.md.

Git-driven observability for the on-prem network behind a Firewalla, powered by Grafana Cloud (free tier) on the visualization side. Ingestion is per-host, not a single chokepoint: each host runs its own Grafana Alloy agent and pushes metrics straight to Cloud (remote_write, 60s); a central Alloy on the LXC covers what's left over — blackbox probes, the Home Assistant scrape, and one Loki receiver (see Architecture below).

Everything is declarative:

🧭 What this repo demonstrates

This is a real everything-as-code observability stack: dashboards, alerts, and datasources are Terraform, applied automatically on merge to main. Each pattern below links to the code that proves it.

Pattern How it shows up here
Apply-on-merge IaC for a live SaaS surface — the merged PR is the change record The apply job in .github/workflows/terraform.yml runs terraform apply -auto-approve on every push to main that touches terraform/ or dashboards/ — the Grafana Cloud stack is code, not clickable UI state.
Deadlock-safe required-check gating The unfiltered pull_request trigger + changes filter job + gate job with if: always() in terraform.yml — a path-filtered heavy job can't hang a PR forever as an "Expected" check (rationale cites lentago/.github#27).
OIDC to AWS, no static credentials terraform.yml requests id-token: write and assumes an IAM role for the S3 state backend — no long-lived AWS keys live in CI secrets.
Serialized apply via concurrency group The apply job's concurrency: { group: terraform-apply, cancel-in-progress: false } in terraform.yml — two quick merges queue instead of racing on the same Terraform state.
Reusable CI shared across the fleet docs-check.yml, shellcheck.yml and claude.yml are thin wrappers over uses: lentago/shared-workflows/...@v1.2.3 — one definition maintained centrally, consumed by every repo at an immutable semver tag (shared-workflows ADR-0005).
Branch protection as a ruleset, not ad-hoc settings main requires the named checks gate, shellcheck / shellcheck, and docs-check / docs-check and allows squash merges only — enforced at the platform level, so merge is gated on CI, not convention.
Alerting-as-code with written design intent terraform/alerts.tf provisions 66 rules across 9 groups (one from the uvularia-pipeline module); docs/adr/0001-...md records why these live in Grafana (not AWS) and why each group's no_data_state differs.
Cross-repo contract discipline The Loki output contract and Solidago (AWS) contract sections name exactly which repo owns which label, field, and IAM role — explicit written boundaries instead of drift.

🛠️ Make a change yourself

These systems are real, and nothing critical rides on them. That makes this a safe place to try a change before you make the same kind of change in your own shop. Pick one:

1. Add or edit a Grafana dashboard as JSON. Drop or edit a dashboard file under dashboards/ and reference it from a for_each map in terraform/dashboards.tf (via locals.tf), then open a PR. CI runs fmt/validate/plan and posts the terraform plan as a PR comment for review; the gate check must be green to merge. On merge to main the apply job upserts the dashboard into Grafana Cloud through the grafana/grafana provider — no manual Grafana UI step anywhere in the loop. Proof this works: #187 — feat(dashboard): Context Ledger section on Claytonia — Runner Fleet, #171 — feat: add essexcrossingatmontserrat.com probe and dashboard, #163 — Reorganize and rename dashboards along product lines.

2. Recover a live-only edit before apply-on-merge destroys it. Because merge re-applies repo state, a dashboard hand-edited in the Grafana UI is un-codified work living on borrowed time: the next unrelated merge's apply silently reverts it. That is exactly what happened in 2026-07 — a fleet-scoreboard revamp was pushed live via the API but never committed, and five later bug-fix merges each ran the apply and stomped it. The fix is not to distrust apply-on-merge but to make code the source of truth: pull the lost panel JSON out of Grafana's dashboard version history, commit it into dashboards/ + terraform/dashboards.tf, and merge — so future applies preserve the change instead of reverting it. Proof this works: #119 — infra-health: restore fleet-scoreboard revamp lost to IaC drift stomp, #120 — CLAUDE.md: add anti-drift rule for live dashboard edits.

3. Onboard alerting for a new signal. Add a rule group to terraform/alerts.tf, choosing no_data_state deliberately per signal type (an absent probe is an ambiguous vantage-point gap; an empty log stream is the failure — opposite stances, both defensible), and record the rationale in an ADR under docs/adr/. The PR triggers a plan; merge applies the rule group and, if new, the contact-point wiring — the same apply-on-merge path as dashboards. Proof this works: #188 — Add context-ledger alert rules, #175 — Alert on ingest absence for the critical Loki streams, #172 — feat: Grafana-native alerting for site probes.

Vectors 1 and 3 need no special access beyond opening a PR; humans own the policy and direct every change. Every directed pull request (one a person asked for) gets an AI review before it is armed to merge. The one exception is an opted-in scheduled routine, which merges when a fixed rule check on its author, title and changed paths passes; no routine is opted in here today (ADR-0014). Editing the live Grafana stack directly (vector 2's failure mode) requires a Grafana Cloud account on the lentago stack.

Architecture

Node metrics push from each host (host-local Alloy → Mimir, 60s); the central Alloy on the LXC handles blackbox probes, the Home Assistant scrape, and a Loki receiver used only by the device-inventory feed (see below). Zeek/ACL logs bypass the central Alloy entirely — the Firewalla's Fluent Bit shipper (owned by lentago/betula) pushes those directly to Grafana Cloud Loki. Full rendered diagram, paths, and label conventions: docs/metrics-flow.md.

  HOSTS ×6 (neptune, pve, pve2, pve3, pve4, pve5)
    each: node_exporter:9100 → local Alloy ── remote_write 60s ─┐
                                                                ├─▶ Grafana Cloud
  CENTRAL ALLOY (LXC 105)                                       │    (lentago)
    blackbox ICMP/HTTP ───────────────── remote_write ─────────┤    ├─ Mimir (metrics)
    Home Assistant /api/prometheus → HA scrape → remote_write ──┤    ├─ Loki  (logs)
    claytonia + cost-export    → Loki receiver :3100 ──────────┘    └─ Grafana
                                                                          │
  Firewalla Fluent Bit (Zeek/ACL) ── direct Loki push, no relay ─────────┤
                                                                          │
  AWS / SOLIDAGO CloudWatch ◀── query-on-demand (assume-role) ── Grafana ─┤
                                                                          │
                                                  public share ──▶ Office Display

Architecture decisions: the reasoning behind this layout — the Cloud + Alloy

  • Terraform migration, host-local push, apply-on-merge, series-budget engineering, privacy-shaped egress, and the flat dashboard folder — is recorded as decision records in docs/adr/.

Loki output contract

Firewalla's Fluent Bit shipper pushes Zeek/ACL logs directly to Grafana Cloud Loki over HTTPS — no LAN relay, and the central Alloy on LXC 105 is not in this path. It attaches the cluster="lentago-lab" label itself (a static Labels directive in its Fluent Bit config), unlike the metrics side, where the central Alloy's external_labels injects that label.

The eight active log streams, keyed by log_source (the four zeek_http/ zeek_files/zeek_notice/zeek_weird streams shipped in betula#58, 2026-07-02):

log_source Contents
zeek_dns DNS query/response records from Zeek — domain, query type, client IP, answer.
zeek_conn TCP/UDP connection summaries — src/dst IP and port, bytes, duration, state.
zeek_ssl TLS handshake records — SNI, certificate subject, cipher, validation status.
zeek_http Plain-HTTP request records — host, URI, method, status, user agent.
zeek_files File-transfer analysis records — MIME type, source, hashes where computed.
zeek_notice Zeek notices — the engine's own alerts/anomalies. Sparse and bursty (~150–200 lines/day measured 2026-08-09), so it carries no ingest-absence alert; see terraform/alerts.tf.
zeek_weird Protocol-anomaly records ("weirds") — steady low-volume background (~10k lines/day).
firewalla_acl Firewalla ACL alarm events — blocked/allowed flows, rule name, severity.

Beyond the Firewalla streams, the change pipeline (ADR-0010) reads and writes these, all with cluster="lentago":

log_source Contents
github_actions_run / github_actions_job betula's GitHub Actions feed (clients/github, betula#113): one line per completed run attempt and per job.
drosera_live, betula_live, sites_live live events (pipeline="change", stage="live", contract in ADR-0010): the SHA each surface runs, every reconcile tick, from alloy-host/gitops-pull.sh, betula's gitops-sync.sh (betula#132) and the sites probe (#268). The "By repo" row of dashboards/change-pipeline.json is the query that keys on those two labels.
change_pipeline_state The pipeline producer's write-back: one line per repo and stage every 30 s, one stream per repo. It carries no pipeline/stage labels, so the live selector never picks it up.

Adding a new log_source stream — checklist:

  1. Measure its 24h volume (e.g. sum(count_over_time({log_source="X"}[24h]))).
  2. Add a row to the table above, the stream to EXPECTED in scripts/check-loki-labels.sh, and the value to the CLAUDE.md § Loki labels list.
  3. Add a local.loki_ingest_streams entry in terraform/alerts.tf with a window sized from the measurement — or record why it has none, as zeek_notice does (sparse and bursty, so a quiet day is a real zero).

Change coordination: the Firewalla side of this pipeline — the Fluent Bit config, its direct-to-Loki output block, and the log_source label scheme — lives in lentago/betula (renamed from firewalla-axiom-pipeline 2026-07-04; betula#82, merged 2026-07-09, also dropped its former parallel Axiom output, so Grafana Cloud Loki is now its sole destination). Any change to the log_source values or the Loki output must be coordinated with that repo (see its issue #42, which shipped the current label scheme) so both sides stay in sync.

Triage pointer: if Zeek/ACL dashboards go dark, look on the Firewalla / betula side first (the Fluent Bit container, its Loki output config, box network egress) — the central Alloy on LXC 105 no longer relays this traffic, so it's not a suspect for these four streams. It remains the right place to look for device_inventory gaps (below) and for the metrics pipeline.

Alerting

Terraform provisions the stack's alerting end to end — terraform/alerts.tf and the uvularia-pipeline module: 66 rules across nine rule groups (eight in alerts.tf) in the Lentago folder, plus the stack's first contact point. See docs/adr/0001-grafana-native-alerting-for-site-probes.md for why this lives in Grafana instead of AWS, and docs/adr/0008-site-availability-slos-and-burn-rate-alerts.md for the SLO targets and burn-rate design.

  • Site probe alerts (8 rules — "Site down" on probe_success and "TLS cert expiring" on probe_ssl_earliest_cert_expiry under a 21-day threshold, for each of the 4 site dashboards). These metrics come from the on-prem Alloy's blackbox exporter and exist nowhere in CloudWatch, so Solidago's AWS-native alerting (CloudWatch alarms → SNS, solidago ADR-0001) can't see them. no_data_state = "NoData" deliberately: the probes run from one on-prem vantage point, so an on-prem/WAN outage looks identical to a real site outage, and alerting on NoData would misfire on every on-prem hiccup instead of a real one.
  • Loki ingest absence (9 rules — one each for zeek_dns, zeek_conn, zeek_ssl, zeek_http, zeek_files, zeek_weird, firewalla_acl, device_inventory, and github_branch_head, the liveness signal of betula's GitHub collector, which the Change pipeline and CI runners groups read with no_data_state = "OK"; #292): fires when a stream goes quiet longer than its expected cadence (30m–2h depending on measured volume; see the per-stream comments in terraform/alerts.tf). no_data_state = "Alerting" here — the opposite of the probe rules — because an empty Loki result is the failure this group exists to catch, not an ambiguous vantage-point gap. zeek_notice carries no absence alert (sparse, bursty by nature — a quiet day is a real zero).
  • Site SLO burn rate (6 rules, issue #195 — a fast/slow pair for each of the three public sites: lentago.dev, icecreamtofightwith.com, pondviewlane.com). These are objective alerts, not symptom alerts: they fire on how fast the 99.9% / rolling-30d availability error budget (43.2 min/30d) is being consumed. Each is multi-window (Google SRE Workbook) — a long window sets sensitivity, a short window forces a fast reset. Fast burn pages at ≥14.4× (1h ∧ 5m; budget gone in ~50h); slow burn tickets at ≥3× (24h ∧ 2h). Burn rate is computed inline with avg_over_time — no recording rules, zero new Mimir series. no_data_state = "NoData" as with the probe group (same single-vantage reason). Operator view: the Sites — SLOs & Error Budget dashboard (uid sites-slo-error-budget).
  • Bullpen liveness (3 rules, issue #207 — runner-fleet liveness + retry, the game-day #1 gap). Built on the same event="job_running" heartbeat the Claytonia — Runner Fleet dashboard renders: workers below full strength tickets when fewer than 5 distinct workers heartbeat for 5m, a .retry requeue tickets on any reaper-driven requeue (a worker died mid-job and recovery self-healed silently), and a fully-dark fleet pages when zero workers heartbeat for 10m (no_data_state = "Alerting" — an empty result is the failure, as with the ingest-absence group). Loki-sourced, zero new Mimir series. Caveat: job_running marks busy workers, not idle-but-alive ones, so the two headcount rules can false-fire on a legitimate idle stretch until claytonia ships an always-on workers/<host>.alive heartbeat to Loki (boundary: drosera owns the rules, claytonia owns the telemetry). See ADR-0009.
  • Uvularia pipeline — lentago (4 rules, issue #218 — from the uvularia-pipeline module, the same one clients apply into their own stacks): served digest behind published (the Ask function reported an older digest after the latest publish, for 30m; #237), Ask function silent (no served/asked event in 12h), obligation went amber or red on a publish, and the Ask daily cap nearly spent (under 20% left, or a turn refused for the cap). Loki-sourced over log_source=uvularia_*, cluster="lentago". Each query returns a series only when something is wrong, so no_data_state = "OK". Contact point and routing are the uvularia_contact_point / uvularia_repeat_interval / uvularia_group_by variables; the contact point defaults to Site probe email. See docs/clients/uvularia.md.
  • Change pipeline (2 rules, issue #270, ADR-0010): change stuck when a repo's newest main commit (github_branch_head) has no live event carrying its SHA after the surface's propagation budget (ADR-0010 defines both: the live event is what each runtime emits with the SHA it runs, and the budget is how long a merge may take to get there): 10 min for gitops and terraform repos, 20 min for sites. One alert instance per repo/sha. A terraform repo only counts when a terraform push run exists for the head, because every apply workflow is path-filtered and a docs merge applies nothing. no_data_state = "OK". The repo lists are locals in terraform/alerts.tf.
  • CI runners (1 rule, issue #292): self-hosted job queued when a self-hosted Actions job has waited more than 15 min for a runner, which means the runner is offline or wedged. It reads betula's github_actions_queued sample (one line per queued job per 5-min tick). One alert instance per job, labelled with repo, workflow, job_name and html_url. no_data_state = "OK": the query returns a series only for a job over budget. The query lives in terraform/logql/ and is tested against a real Loki by the logql-test workflow.
  • Contact point: one email contact point (Site probe email), reused by every group. The recipient is TF_VAR_alert_email, a sensitive Terraform variable with no default, supplied via CI/.envrc and never committed (this is a public repo). Routing is scoped per-rule, so it doesn't touch the stack's root notification policy.

Client event push (Actions + serverless)

Workflows and serverless functions can't run Alloy, so drosera ships two tiny emitters that push one structured event straight to a Grafana Cloud Loki push endpoint: the loki-event composite action (lentago/drosera/.github/actions/loki-event@main) and clients/loki_push.py for functions. Both send the same labels — log_source=<source>_<stage>, cluster=<org>, plus source, pipeline, stage, repo — with the payload as a one-line JSON log. First consumer is uvularia, pushing into each client's own Grafana Cloud stack. Token setup and label rules: clients/README.md.

The uvularia pipeline pane reads those events: the Uvularia — Records pipeline dashboard (dashboards/uvularia-pipeline.json, uid uvularia-pipeline) plus three alert rules, packaged as one Terraform module, terraform/modules/uvularia-pipeline/. This stack runs it for the demonstration client (cluster = "lentago", wired in terraform/alerts.tf); a client runs the same module from their own repo into their own free-tier stack, and nothing of theirs is hosted here. How a client applies it, and the payload fields each stage must send: docs/clients/uvularia.md.

Change pipeline producer

The Change — Pipeline dashboard already draws a per-repo stage strip (the collapsed "By repo" row, Graphviz panel). The same pipeline is also available as data: pipeline-producer/ is a stdlib Python service on LXC 105 (drosera-pipeline.service, #266). Every 30 s it reads the Actions feed and the live events for the last 24 h, computes the six ADR-0010 stages for each repo once, and does two things with the result:

  • Serves pipeline.json (schema 1, pinned in #266) on :8611. pub's Caddy proxies http://pub.lan/viewport/pipeline.json to it (that route lives in kalmia), and the brasenia wall pane reads it there. LXC 105 doesn't mount the web share, so the producer serves the document instead of dropping a file, and nothing on pub needs a Loki token.
  • Writes change_pipeline_state back to Loki: one line per repo and stage. The stuck alert can then be a one-line query, and the "By repo" strip can stop re-deriving state in LogQL (a follow-up, once the stream has a day of data).

It needs its own Loki read token. The collectors' token in /opt/drosera/.env is logs:write only, and a read with it gets 401. Mint a logs:read access-policy token on the lentago stack and pass it to deploy.sh as DROSERA_LOKI_TOKEN. The script refuses to start the service without it. Setup, state rules and the document schema: pipeline-producer/README.md.

Solidago (AWS) contract

Solidago platform metrics render in this stack via a query-on-demand CloudWatch datasource — nothing is streamed or imported into Mimir, so it consumes zero free-tier active series. Queries bill as CloudWatch GetMetricData at render time; the dashboard refresh floor is 1m.

  • This repo owns: the solidago-cloudwatch datasource (terraform/datasources.tf, "Grafana Assume Role" auth) and the Solidago dashboards (dashboards/solidago-platform-health.json, titled Solidago — Platform Health).
  • Per-site dashboards sit in the same flat Lentago folder, titled Sites — <domain> (dashboards/site-*.json, uid = site repo name): outside-in blackbox probes (Mimir) on top, per-TargetGroup / per-ECS-service CloudWatch below. ALB panels use SEARCH expressions on stable name fragments — TargetGroup and LoadBalancer dimension hashes rotate on the nightly DR rebuild, so pinned dimension values break every night. Site dashboards refresh at 5m (not the 1m floor) since they're viewed on demand, not kiosked. Onboarding a new site: one probe target in alloy/config.alloy, one JSON, one sites_dashboards entry — existing sites untouched.
  • lentago/solidago owns: the IAM role solidago-dev-grafana-cloudwatch (modules/grafana-cloud), its trust and permission policies, and the External ID plumbing.
  • Coordination rule: renaming the role, changing auth, or widening its policy is a cross-repo change — same discipline as the log_source label contract above.
  • Overnight gaps are the DR drill, not an outage: solidago tears down and rebuilds nightly; role ARNs are deterministic, so the datasource never needs re-pointing, and "No data" while the platform is down is correct behavior.
  • Alerting stays AWS-native (CloudWatch alarms → SNS; solidago ADR-0001) — Grafana is visualization only.
  • Known health-check quirk: the datasource's Save & test / health endpoint probes CloudWatch Logs as well as metrics, so it reports AccessDeniedException … logs:DescribeLogGroups alongside "Successfully queried the CloudWatch metrics API". That is expected, not a fault — the role is metrics-only by design (logs belong to Axiom/betula, per ADR-0001's boundary). Metrics green = healthy. Verified end-to-end 2026-07-04 (assume-role query returned live ALB series for all three target groups).

Repo layout

.envrc.example                 # template for direnv — copy to .envrc, fill in secrets
docker-compose.yml             # spins up Alloy on the LXC (one service)
alloy/
  config.alloy                 # Alloy collector config (scrape + push + receive)
  blackbox.yml                 # blackbox prober module definitions
  ha_token.example             # template; real ha_token is gitignored
alloy-host/                    # gitops deploy for the central Alloy on the LXC
  gitops-pull.sh               # 5-min pull + validate + reload-on-drift
  alloy-gitops.{service,timer} # systemd units (bootstrap-only)
  README.md                    # bootstrap + ops
dashboards/                    # source of truth for Grafana dashboard JSON
  network-overview.json
  dns-security.json
  traffic-devices.json
  infra-health.json
  office-display.json
  neptune-nas.json             # Neptune NAS real-time activity (CPU/disk/net/RAID/temps)
  solidago-platform-health.json # Solidago (AWS) via the CloudWatch datasource
  site-*.json                  # per-site health (Mimir probes + per-TG/service CloudWatch)
  uvularia-pipeline.json       # Uvularia — Records pipeline (Loki events, cluster variable)
  change-pipeline.json         # Change — Pipeline: merge-to-live per SHA (Actions feed + live events, ADR-0010)
                               #   + collapsed "By repo" row: one Graphviz stage strip per repo (grafana-graphviz-panel)
  github-actions.json          # GitHub — Actions: fleet-wide run history, failures, durations, queue time (Actions feed, #234)
terraform/                     # manages Cloud-side resources
  *.tf                         # incl. datasources.tf (solidago-cloudwatch) and
                                # alerts.tf (site probe + Loki ingest-absence +
                                # bullpen + on-prem availability alert rules)
  modules/uvularia-pipeline/   # uvularia dashboard + rules; estate and clients both use it
pipeline-producer/             # change-pipeline producer on LXC 105: pipeline.json on :8611 + change_pipeline_state (#266)
  pipeline_producer/           # stdlib Python 3.10 package (read Loki → compute stages → serve + write back)
  systemd/ deploy.sh           # drosera-pipeline.service + idempotent installer
  tests/                       # unit tests against a fake Loki
clients/                       # source-neutral client emitters (#131)
  loki_push.py                 # stdlib-only Loki push for serverless functions
  README.md                    # token setup + label discipline
docs/adr/                      # architecture decision records (e.g. native alerting for site probes)
docs/clients/uvularia.md       # client export: the uvularia pane into a client's own stack
scripts/
  inventory-cloud.sh           # snapshot current state of lentago.grafana.net
  deploy-node-exporter.sh      # install node_exporter on a host
  deploy-alloy.sh              # install a host-local Alloy push agent (60s remote_write)
.github/actions/
  loki-event/                  # composite action: one event from a workflow to Loki
.github/workflows/
  terraform.yml                # fmt/validate/plan on PR
  loki-event-test.yml          # loki-event + loki_push.py against a mock Loki
  pipeline-producer-tests.yml  # pipeline-producer unit suite on Python 3.10

First-time setup

1. Grafana Cloud credentials

  • Stack service account (drives Terraform + future CLI tooling). In https://lentago.grafana.net → Administration → Users and access → Service accounts, create a service account named terraform-iac with role Admin, then Add token and copy the value. This becomes GRAFANA_AUTH.
  • Access policy tokens (drive Alloy remote_write / log push). At https://grafana.com → My Account → Access Policies, create one policy with scopes metrics:write and logs:write for the lentago stack. Generate a token; copy the username (a numeric stack ID per signal type) and token and the push URLs from the stack details page.

Drop all of the above into .envrc:

cp .envrc.example .envrc
$EDITOR .envrc           # fill in real values
direnv allow

2. Terraform — adopt Cloud-side resources

cd terraform
terraform init
terraform plan           # should show: import the Lentago folder + 5 dashboards, then update them
terraform apply

State lives in S3 (solidago's solidago-tfstate-365184644049 bucket, versioned + encrypted) — no local-state backup step needed. Day to day you don't run apply by hand at all: merging dashboard/terraform changes to main auto-applies via the terraform workflow. See terraform/README.md § State and § CI.

The first apply rewrites datasource UIDs in each dashboard from loki / prometheus (the old self-hosted UIDs) to grafanacloud-lentago-logs / grafanacloud-lentago-prom (the lentago stack's auto-provisioned UIDs). The original JSON files in dashboards/ are not modified — the rewrite happens in-memory at apply time via replace() in terraform/locals.tf.

3. Delete orphan datasources in Cloud

The Cloud migration wizard created loki and prometheus datasources pointing at the self-hosted LXC, which Grafana Cloud cannot reach. After terraform apply repoints dashboards to grafanacloud-*, delete the orphans:

for uid in loki prometheus; do
  curl -sS -X DELETE \
    -H "Authorization: Bearer $GRAFANA_AUTH" \
    "$GRAFANA_URL/api/datasources/uid/$uid"
done

4. Alloy on the Proxmox LXC

# On the LXC (or any host with docker), in this repo:
cp alloy/ha_token.example alloy/ha_token
$EDITOR alloy/ha_token             # paste your HA long-lived access token (no trailing newline — use printf '%s' "<token>" > alloy/ha_token to be safe)
docker compose up -d
docker compose logs -f alloy       # confirm "remote_write succeeded" etc.
open http://<lxc-ip>:12345         # Alloy debug UI

After Alloy is up:

  • Confirm logs arrive in Cloud: Explore → grafanacloud-lentago-logs → {cluster="lentago-lab"}.
  • Confirm metrics arrive: Explore → grafanacloud-lentago-prom → up{cluster="lentago-lab"} should return 1 for each scrape target.
  • Visit a dashboard (e.g. Lentago / On-prem — Network Overview) and confirm panels render data.

5. Office display (kiosk)

Grafana Cloud doesn't allow anonymous viewers. Use public dashboard sharing:

  1. Open firewalla-office-display in the Cloud UI.
  2. Share → Public dashboard → Enable.
  3. Copy the public URL and point the kiosk Chromium at it.

The public URL bypasses auth for that one dashboard only; nothing else in the stack is exposed.

6. CI

Add repository secrets (Settings → Secrets and variables → Actions, not Dependabot) so the plan and apply jobs can authenticate to Grafana:

  • GRAFANA_URL — full stack URL, e.g. https://lentago.grafana.net (no trailing slash).
  • GRAFANA_AUTH — the same Grafana Cloud service account token you use in .envrc as GRAFANA_AUTH / GRAFANA_SA_TOKEN.

Both must be non-empty. If either is missing, the job fails immediately with a clear log message instead of a misleading Terraform provider error.

AWS (for the S3 state backend) is reached via GitHub OIDC — no AWS secrets are stored; the jobs assume drosera-github-actions-terraform in account 365184644049. The workflow runs fmt -check + validate + plan (posted as a PR comment) on every PR touching terraform/** or dashboards/**, and apply -auto-approve on push to main — so merges deploy automatically. See terraform/README.md § CI.

Day-to-day

Edit a dashboard

  1. Edit the JSON file in dashboards/ (or edit in the Cloud UI and copy the exported JSON back into the file).
  2. Open a PR — CI posts the terraform plan as a comment. Merging to main auto-applies. (Out-of-band you can still cd terraform && terraform apply.)

Add a new dashboard

  1. Drop a new JSON file in dashboards/. Title it <Group> — <What> (On-prem — …, Claytonia — …, Solidago — …, Sites — <domain>): every dashboard lives in one flat Lentago folder, so the title prefix is the only thing that groups the list.
  2. Add an entry to the matching per-group local (lab_dashboards, claytonia_dashboards, solidago_dashboards, or sites_dashboards) in terraform/locals.tf. These maps differ in how the JSON is pre-processed, not in where the dashboard lands.
  3. Open a PR; merging to main applies it (or terraform apply locally).

Add new scrape targets / new log sources

Edit alloy/config.alloy, open a PR, merge. The central Alloy on the grafana-stack LXC is gitops-managed: alloy-gitops.timer pulls origin/main every 5 minutes, validates with alloy fmt, and reloads the collector on drift (SIGHUP for alloy/ changes, docker compose up -d for compose changes). No hand-editing on the LXC — see alloy-host/README.md. To deploy immediately instead of waiting for the timer: systemctl start alloy-gitops.service on the LXC.

node_exporter

Bare-metal node_exporter on each host is the metric source. Run scripts/deploy-node-exporter.sh against any new host you add. How those metrics reach Grafana Cloud depends on the collection model below.

On hosts with lvm2 (the five Proxmox nodes) the script also installs lvm-thin-textfile, a one-minute root timer that writes lvm_thin_pool_* and lvm_vg_free_bytes into node_exporter's textfile directory. The thin pool and root filesystem rules in the "On-prem availability" group read them. Re-run the script on each node after changing it, and before merging rule changes that depend on new series, or they start in NoData.

Collection models: central pull vs. host-local push

There are two ways a host's node_exporter metrics get to Mimir, and the repo supports both side by side with identical labels (job="node", instance="<host>"), so dashboards never care which is in use:

  • Central pull (default/legacy): the Alloy on LXC 105 scrapes the host's :9100 over the LAN. Add the host to the prometheus.scrape "node" target list in alloy/config.alloy.

  • Host-local push (standardized): the host runs its own Alloy that scrapes localhost:9100 and remote_writes to Mimir at 60s (#239 — faster cadences bill as extra series). Buffers across network blips, and each host owns its own shipping. Deploy with scripts/deploy-alloy.sh:

    source .envrc                     # exports GRAFANA_CLOUD_METRICS_*
    # on the target host (root, or via sudo):
    ./deploy-alloy.sh <instance-label>   # e.g. neptune, pve, pve3

    The script installs Alloy from the Grafana apt repo, writes /etc/alloy/config.alloy (canonical config embedded in the script) and a 0600 /etc/default/alloy holding the push token, then enables the service.

Don't run both for the same host — that double-counts series. When you move a host to push, delete it from the central prometheus.scrape "node" block. The rollout target is neptune + all five Proxmox nodes on push; HAOS stays on the central HA /api/prometheus scrape (it can't run a system Alloy).

Fleet reasoning stream (worker transcripts)

The Claytonia — Runner Fleet dashboard has a live "Fleet stream of consciousness" panel that shows each running bullpen job's reasoning in near-real-time (each assistant turn's narration + the tools it fired). The source is a worker-local Alloy agent on each runner that tails Claude Code's session transcript as it is written and ships it to Grafana Cloud Loki as job="claude_transcript" (labels: project, worker, session_id). Gemini CLI and Codex CLI jobs ship on the same stream with runtime and runid labels added. The agent tails them through the labelled links that claytonia writes under /home/claude/.claytonia/live/<runtime>/<project>/<runid>/, and only for checkouts under /home/claude/work/ (#305). scripts/tests/transcript/run.sh runs the config against a throwaway Loki (CI: transcript-test).

Deploy it per worker with scripts/deploy-runner-transcript-alloy.sh (canonical config: alloy/runner-transcript.alloy):

source .envrc   # exports GRAFANA_CLOUD_LOGS_*
# on the worker (sudo-capable), from a repo checkout:
./scripts/deploy-runner-transcript-alloy.sh   # worker label is auto (from hostname)

It runs as a dedicated alloy-transcript.service under the claude user (the session files are 0600 claude:claude). Egress is deliberately scrubbed (see #71): only assistant text + tool names are shipped — thinking blocks, tool inputs, and user/tool-result lines (raw repo contents) never leave the LAN. A runid label is a possible future bullpen-side fast-follow.

Device inventory feed (name ↔ IP resolution)

Dashboards that show raw LAN source IPs (id_orig_h) resolve them to device names by joining against a device-inventory log stream, log_source="device_inventory". Grafana Cloud runs queries server-side and cannot reach the LAN, and LAN topology must not be published to GitHub — so the name↔IP mapping goes only to Grafana Cloud Loki, the trust domain that already holds every LAN IP via the Zeek streams (see #113, ADR-0006). Since #243 it is pushed directly over HTTPS, like the Zeek/ACL streams; until then it relayed through the central Alloy's :3100 receiver. That receiver (loki.source.api "firewalla" in alloy/config.alloy) is not dead config: it still serves the claytonia workers' cr-loki.sh events (job="claude_runner", which the Bullpen liveness alerts read) and the workstation claude-cost-export timer (job="claude_local", see claude-cost-export/).

The producer is betula's Firewalla device-inventory collector (lentago/betula scripts/device_inventory_publish.sh, moved from this repo in 2026-10 — betula#115, #151, #243: capture is betula's side of the boundary). It runs on the Firewalla box (pi user, hourly from betula's cron/user_crontab, straight from betula's gitops clone). It reads the box's own device inventory from local redis (host:mac:* hashes) and pushes one record per (device, IP) pair straight to Grafana Cloud Loki, with the same endpoint and GRAFANA_CLOUD_LOGS_* credentials Fluent Bit uses. It sets cluster="lentago-lab" itself.

Stream schema — one Loki stream per (device, IP):

Field Value
label log_source device_inventory
label dev <name>|<ip> — load-bearing (see below)
line body (JSON) {"name":"…","ip":"…","mac":"…","family":"4"|"6","source":"firewalla-redis"}

Display name is the redis name field, else bname, else the MAC. Both the IPv4 (ipv4Addr) and every IPv6 (ipv6Addr array) address get their own row.

The dev label contract: the <name>|<ip> shape lets a dashboard build a template variable with label_values({log_source="device_inventory"}, dev) and regex /(?<text>[^|]+)\|(?<value>.+)/ — the dropdown shows device names while the variable value stays the raw IP that id_orig_h=~"$device_ip" needs. Any | in a device name is stripped before composing the label so the split stays unambiguous.

The schema above is the interface drosera consumes — the dashboards' IP→name joins, the dev template variable, and Ingest absence — device_inventory depend on it. Changing it takes a matching change on both sides.

Deployment is betula's: a merge to betula's main reaches the box through its gitops loop, with no deploy step here. Smoke-test on the box without pushing:

ssh pi@firewalla.local 'DRY_RUN=1 ~/.firewalla/firewalla-axiom-pipeline/scripts/device_inventory_publish.sh | head'

Volume is negligible (~190 devices, hourly; logs not metrics, so it does not touch the active-series cap).

Series budget / HA export trim

Grafana Cloud free tier caps active series at 15,000. Home Assistant's /api/prometheus export is the single biggest consumer — it emits a series for nearly every entity plus per-entity change-counters, "last updated" timestamps, availability flags, and *_created markers. None of that is used by any dashboard, so the prometheus.relabel "ha_trim" component in alloy/config.alloy drops those families before remote_write, reclaiming ~5.8k series. The real numeric HA metrics (temperatures, fan RPM, battery, climate, brightness, sensor states) are kept.

The same component also drops the HA UGreen integration's sensor.ugreen_nas_* series (about 60, #330): Neptune's own node_exporter already covers the NAS. The three that feed the Neptune NAS dashboard are kept: sensor.ugreen_nas_device_fan and sensor.ugreen_nas_pool_1_disk_{1,2}_temperature. To chart another NAS sensor, add it to the keep regex in ha_trim.

ha_trim also drops every series for the phone notification sensors (sensor.*_last_notification, sensor.*_last_removed_notification, sensor.*_active_notification_count) and for sensor.sms_inbox. Those carry the text of incoming messages, which must not leave the house (ADR-0006, privacy-shaped egress). The names come from lentago/epigaea's packages/sms_inbox.yaml, which also filters them out of HA's Prometheus export, so this rule is the second line. If epigaea renames those entities, change the drop regex here in the same change.

Check current usage with the grafanacloud-usage datasource: grafanacloud_instance_active_series. If you add a host (each node_exporter is ~1.3k series), watch the headroom — trim more (e.g. node_exporter discard/flush families) or the rollout will hit err-mimir-max-active-series.

Check Loki label health

scripts/check-loki-labels.sh queries Loki for active log_source values over the last 24h and diffs them against the expected set. That set is the eight Firewalla streams (zeek_dns, zeek_conn, zeek_ssl, zeek_http, zeek_files, zeek_notice, zeek_weird, firewalla_acl) plus the change pipeline's drosera_live, betula_live, sites_live, github_actions_run, github_actions_job and change_pipeline_state. Run it manually as a sanity check, or wire it to a cron / GitHub Actions schedule to alert on silent log streams:

source .envrc
./scripts/check-loki-labels.sh

Exits 0 when all expected values are present; exits 1 and prints the missing names to stderr otherwise.

Why this layout

The original incarnation of this repo ran Loki, Prometheus, Grafana, and blackbox-exporter all on a single LXC via docker compose, with file-based provisioning. That worked but had three pain points:

  1. Storage on the LXC — Prometheus TSDB + Loki chunks meant disk pressure and another thing to monitor.
  2. Dashboards drifted between repo and UI — anything edited in the UI was lost on the next provisioner reload.
  3. No upgrade story for Grafana itself — each docker compose pull was a gamble.

Migrating to Grafana Cloud free tier + Alloy fixes all three: storage moves to the Cloud's free 50GB logs / 10K active series allotment; Terraform plus checked-in dashboard JSON makes the repo the source of truth (UI edits get overwritten on next apply); and Cloud handles Grafana upgrades.

License

MIT.

Contributing

See CONTRIBUTING.md.


🌱 Lentago Labs is a pro-bono operations practice for organizations that run on volunteers, donations, and one overworked tech person. Everything here is free to take, and we practice what we publish: our own estate runs this way, in the open. Start at the org profile.

About

Drosera — the Lentago Labs observability suite. Git-driven, fully declarative telemetry into Grafana Cloud (Mimir + Loki, free tier): per-host Grafana Alloy pushes metrics + logs with no central chokepoint, and AWS CloudWatch feeds in by assume-role. Terraform-provisioned dashboards, alert rules, and contact points; node_exporter, LogQL.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages