From 168aad104f750dabd261e76876aadc76c27b5887 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 18:33:29 -0400 Subject: [PATCH 01/47] docs(design): draft product metrics (opt-out adoption telemetry) spec Design a separate, opt-out-by-default OTLP pipeline reporting bounded adoption/usage counters to Stacklok's public metrics collector, fully isolated from the existing operator-facing telemetry pipeline. Co-Authored-By: Claude Sonnet 5 --- .../2026-09-08-product-metrics-otel-design.md | 236 ++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md diff --git a/docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md b/docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md new file mode 100644 index 0000000000..1904945dca --- /dev/null +++ b/docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md @@ -0,0 +1,236 @@ +# Product (adoption) metrics over OTLP — design + +- Status: Draft +- Date: 2026-09-08 +- Scope: new `internal/adapter/productmetrics`, `internal/cliconfig`, `internal/adapter/permconfig` (new `telemetry:` operator section), `cmd/mecated`, `cmd/mecatui`, `cmd/mecatequi`, `cmd/mecak8s` + +## Context + +mecatl has no visibility into community adoption today: no install counts, no +feature-adoption signal, no aggregate usage depth. Stacklok's infra team has +stood up a dedicated, internet-facing OTLP/HTTP metrics ingest at +`https://metrics.stacklok.com/v1/metrics` specifically for mecatl binaries +running on infrastructure Stacklok does not control (`stacklok/infra#5604`): +API-key-gated at the edge (`x-mecatl-metrics-key` header, stripped before the +collector), and server-side filtered to accept only metric names matching +`^mecatl\..*`. + +This is a **new, separate concern** from mecatl's existing operator-facing +observability. `internal/adapter/telemetry` already ships a full OTel pipeline +(ADR 0018/0045/0098): a `MeterProvider` with an always-on Prometheus reader and +an optional OTLP push reader an *operator* points at *their own* collector, plus +an OTLP trace exporter. That pipeline exists so an operator can observe their +own deployment. It must never become the transport for community-adoption data +— an operator's own `--otlp-endpoint` configuration must have zero effect on +what does or doesn't reach Stacklok, and enabling product metrics must have +zero effect on what an operator's own collector receives. + +`toolhive-core` (already an mecatl dependency, `v0.0.43`) ships +`telemetry/providers`: a small, already-reviewed OTel SDK-wiring layer +(`providers.NewCompositeProvider`) that builds a `metric.MeterProvider` from an +options struct (endpoint, headers, service name/version, custom resource +attributes) without ever installing it as the process-global provider. This is +the natural building block for the new pipeline's OTLP/HTTP exporter — it +already speaks the exact shape the new collector expects (custom headers, +OTLP/HTTP, a resource with service name/version), so mecatl does not need to +hand-roll a third OTLP wiring implementation next to the two it already has +(`internal/adapter/telemetry/otlp.go`'s own inline construction, and the +toolhive-core one). + +## Decision + +### 1. A fully independent adapter, `internal/adapter/productmetrics` + +Zero import relationship with `internal/adapter/telemetry`. It owns: + +- Its own `metric.MeterProvider`, built via `toolhive-core/telemetry/providers` + with a **hardcoded** endpoint (`https://metrics.stacklok.com/v1/metrics`) and + a **hardcoded** header key baked into the binary at build time (matching the + infra PR's `x-mecatl-metrics-key` contract) — neither is operator-configurable. + There is exactly one place this data can go. +- Its own small `Recorder` type implementing `port.EventSink` + + `port.ToolCallRecorder` (the same two seams `internal/adapter/telemetry` + taps), but extracting *only* the bounded counts in the catalog below — the + type has no field or parameter through which a tool name, session id, model + id, or free text could ever flow. +- Its own heartbeat ticker (fires once at start, then every ~24h for + long-running processes; a single fire + flush-before-exit for the short-lived + `mecatequi`, mirroring the existing OTLP-push-with-flush precedent in ADR + 0098). +- Its own install-identity file. + +Composition combines the two independent sinks with a trivial fan-out helper +in `internal/cliconfig` (the existing Rule-of-Three home for cross-binary +telemetry wiring, per ADR 0098) — `internal/app` stays import-free of +`productmetrics`, exactly as it is of `telemetry` today. Each `cmd/*/main.go` +builds its existing operator telemetry pipeline unchanged, and — only when +product metrics are enabled — separately constructs a `productmetrics.Recorder` +and tees it in alongside. + +This means: an operator who disables their own OTLP export still has product +metrics flow (if enabled) to Stacklok, and an operator who fully disables +product metrics has zero effect on their own OTLP/Prometheus pipeline. The two +literally cannot leak into each other because they share no struct, provider, +registry, or destination — only the same two read-only observation points in +the engine (`port.EventSink`, `port.ToolCallRecorder`), which every consumer +of those ports already receives independently per composition's existing +fan-out discipline. + +### 2. Metric catalog + +All instrument names are namespaced under `mecatl.adoption.*` — passes the +collector's `^mecatl\..*` filter, and is visually/query-wise distinct from the +operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family, so nobody looking at +either series family can mistake one for the other. + +**Resource attributes** (set once per process, not per-metric labels): +- `service.name` = `mecatl`, `service.version` +- `os.type`, `host.arch` (standard OTel semconv, generic platform facts) +- `mecatl.install.id` — a random v4 UUID (see §4) +- `mecatl.binary` — one of `mecated`/`mecatui`/`mecatequi`/`mecak8s` (closed set) + +**Heartbeat** (on start, then every ~24h for long-running processes; single +fire for `mecatequi`): +- `mecatl.adoption.heartbeat` (counter, +1 per fire) — the install/liveness signal. +- `mecatl.adoption.feature_enabled{feature=...}` (counter, +1 per enabled + feature per heartbeat) — `feature` is one of a closed set: `memory`, + `learning`, `guardrails`, `mcp`, `teams`, `subagents`, `scheduling`. +- `mecatl.adoption.provider_configured{family=...}` (counter) — `family` is one + of `anthropic`/`openai`/`openrouter`/`other` (never a model id/alias). +- `mecatl.adoption.deployment_mode{mode=...}` (counter) — `mode` is one of + `interactive`/`headless`/`k8s`. + +**Coarse usage** (derived from the event/tool-call tap, exported on the +provider's normal periodic-reader cadence — no manual batching needed since +these are cumulative counters): +- `mecatl.adoption.sessions_started` (counter) — on `EvSessionInit`. +- `mecatl.adoption.runs_completed{stop=...}` (counter) — on `EvResult`; `stop` + reuses the existing bounded `session.StopReason` enum. +- `mecatl.adoption.tool_calls` (counter, **no tool/MCP-server name label at + all**) — on every `ToolCallRecorder.ToolCall`. +- `mecatl.adoption.tokens{kind=...}` (counter) — `kind` reuses the existing + bounded token-kind enum (`input`/`output`/`cache_read`/`cache_write`/ + `reasoning`). +- `mecatl.adoption.subagent_used` / `mecatl.adoption.team_used` (counter, + bumped at most once per run when that delegation family appears at all — no + def/member/model name ever surfaces). + +Nothing here is free text, a session/run/model identifier, a tool or MCP +server name, a file path, a prompt, or an output. Every label value is drawn +from a closed enum that already exists internally or is defined fresh in this +package as a small closed set. + +### 3. Config & opt-out + +**Enabled by default** (opt-out), but the toggle is **operator-tier only** — +same trust boundary as `guardrails:`/`openrouter:` (AGENTS.md's existing +operator-tier-only precedent): a project-tier `.mecatl/settings.yaml` can +neither enable nor disable it for a user; a project silently overriding a +user's own telemetry choice in either direction would itself be a trust +violation, so it is parsed with the same WARN-and-ignore discipline as the +other operator-only subtrees. + +`~/.config/mecatl/settings.yaml`: +```yaml +telemetry: + productMetrics: + enabled: true # default; set false to opt out +``` + +Additional disable signals, in precedence order (highest first): +1. CLI flag `--product-metrics=false` (or `--no-product-metrics`) on all four binaries. +2. `DO_NOT_TRACK` environment variable (any non-empty value) — the + cross-ecosystem convention (consoledonottrack.com), so a single env var + already used to opt CI fleets and dev machines out of *other* tools' telemetry + also covers mecatl, with no mecatl-specific config needed. +3. `telemetry.productMetrics.enabled: false` in the operator settings file. +4. Default: enabled. + +A dedicated `MECATL_PRODUCT_METRICS=0` env var is deliberately **not** added on +top of `DO_NOT_TRACK` — one standard signal is preferable to two overlapping +ones with subtly different names. + +**First-run disclosure.** The first time a binary is about to actually send +product metrics in a given run (i.e., telemetry is enabled and this is the +first invocation since the install-id file didn't yet exist), it prints one +non-blocking line to stderr: what is collected (link to the user-docs page), +and the exact flag/setting/env var to disable it. This is not a prompt — it +never blocks — but it is a hard requirement for an opt-out default to be +defensible to the community; the same disclosure text is what the mecatui +zero-state and mecated startup banner both use. + +**Dry-run / audit flag.** `--product-metrics-dry-run` prints every metric this +process would have sent to stderr instead of exporting it, so a skeptical +operator can verify the "no PII" claim directly rather than trusting the docs. + +### 4. Install identity + +A random v4 UUID, generated on first use and persisted at +`$XDG_CONFIG_HOME/mecatl/telemetry-id` (or the platform equivalent via the +existing `internal/adapter/xdgconfig` helper). It contains no machine or user +information and is trivially reset by deleting the file (equivalent, from +Stacklok's side, to seeing what looks like a new install). It exists purely so +"ten heartbeats from one install" isn't miscounted as "ten installs" — no +other purpose. + +### 5. Privacy safeguards + +- A reflect-based guard test (mirroring the existing `attrRole`/`attrStop` + bounded-label discipline, and the `engine/port/diagnostics_imports_test.go` + import-tripwire pattern) asserts the `Recorder`'s entire public API accepts + no bare `string`/free-text parameter — only bounded enum types (Go types + with a small closed value set) and counts. This makes "no PII can flow + through this type" a property a future PR's CI run checks, not just a + code-review norm. +- Exporter failures are silent to the app (lazy-dial exporter matching the + existing OTLP exporter pattern; a dead `metrics.stacklok.com` never blocks + or slows down a session). `Shutdown` is bounded (~3s) so a hung network path + can never delay process exit. +- No content, error message text, file path, or identifier of any kind is + ever an attribute value — every label is a value from a fixed Go-level enum + reviewed in this document. + +### 6. Lifecycle across binaries + +All four binaries (`mecated`, `mecatui`, `mecatequi`, `mecak8s`) wire this +identically through the shared `internal/cliconfig` helper (extending the +existing `HeadlessTelemetry`-style Rule-of-Three home). `mecatequi` (short-lived, +often sub-second) fires one heartbeat and flushes before exit, exactly as its +existing OTLP metrics push already does (ADR 0098). The other three run the +24h ticker for the process lifetime and flush on graceful shutdown. + +### 7. Documentation + +- A new ADR (next available number) records this decision, the exact catalog, + and the "operator-tier only, separate pipeline, no PII" invariants — the + same discipline as ADR 0098/0020, and the natural home for the "why opt-out + is acceptable here" rationale (disclosure + easy universal disable + a + published, reviewable catalog). +- A short `user-docs/` page explains what's collected, links the catalog in + this doc, and gives the exact disable instructions (flag, env var, setting). + +## Testing + +- Unit tests for the `Recorder`'s bounded-enum bumps (one per metric, table- + driven against the catalog above). +- The reflect-based "no free-text parameter" guard test (privacy invariant, + §5). +- A conformance-style test that composing the operator sink + product-metrics + sink and disabling one leaves the other's counters unaffected (isolation + invariant). +- `internal/cliconfig` wiring test verifying the opt-out precedence order + (flag > `DO_NOT_TRACK` > settings.yaml > default-enabled). +- Offline only — no live network call to `metrics.stacklok.com` in tests; the + OTLP exporter construction is exercised against a local httptest server, the + same pattern `internal/app/openrouter_route_e2e_test.go` already uses for a + comparable "real adapter against an httptest stand-in" check. + +## Out of scope (deferred) + +- A per-run cost/dollar metric (mirrors the existing deferred #192 cost-metric + gap in the operator pipeline — no `Cost` field exists anywhere yet). +- Any richer per-feature usage counters beyond the catalog above (e.g. per-MCP + transport type, per-model-family latency) — start narrow, revisit only if a + concrete adoption question the catalog can't answer comes up. +- A first-run interactive consent prompt (explicitly declined in favor of + opt-out + disclosure, per the design decision above). From 42f765488f4e139dcdeba75aaedef6ff86a42bc6 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 18:48:08 -0400 Subject: [PATCH 02/47] docs(plan): implementation plan for opt-out product/adoption metrics 15-task TDD plan implementing the approved design: an independent internal/adapter/productmetrics adapter (toolhive-core-backed OTLP provider, bounded EventSink/ToolCallRecorder, heartbeat, install id, dry-run audit mode), operator-tier-only settings.yaml gate, opt-out precedence via cliconfig, and wiring into all four binaries. Co-Authored-By: Claude Sonnet 5 --- .../plans/2026-09-08-product-metrics-otel.md | 2389 +++++++++++++++++ 1 file changed, 2389 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-08-product-metrics-otel.md diff --git a/docs/superpowers/plans/2026-09-08-product-metrics-otel.md b/docs/superpowers/plans/2026-09-08-product-metrics-otel.md new file mode 100644 index 0000000000..a29f4713b4 --- /dev/null +++ b/docs/superpowers/plans/2026-09-08-product-metrics-otel.md @@ -0,0 +1,2389 @@ +# Product Metrics (Opt-Out Adoption Telemetry) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship a fully independent, opt-out-by-default OTLP pipeline that reports bounded adoption/usage counters (install heartbeat, feature flags, coarse session/run/tool-call/token counts) to Stacklok's public metrics collector, with zero coupling to mecatl's existing operator-facing telemetry. + +**Architecture:** A new `internal/adapter/productmetrics` package owns its own OTel `MeterProvider` (built via the already-vendored `toolhive-core/telemetry/providers`, pointed at a hardcoded `https://metrics.stacklok.com/v1/metrics` endpoint with a build-time-baked header key) and its own `port.EventSink`/`port.ToolCallRecorder` implementation that extracts only bounded, closed-enum counts — no tool/session/model names, no free text. Composition (`internal/cliconfig`, then each of the four `cmd/*` mains) tees this second sink alongside the existing operator telemetry sink; the two share no struct, registry, or destination. + +**Tech Stack:** Go 1.26, `go.opentelemetry.io/otel/{metric,sdk/metric}`, `github.com/stacklok/toolhive-core/telemetry/providers` (already a dependency), `github.com/google/uuid` (already an indirect dependency, promoted to direct). + +**Spec:** `docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md` — read it alongside this plan; the plan implements it with one deliberate scope refinement (see Global Constraints). + +## Global Constraints + +- **Opt-out, enabled by default.** Precedence, highest first: CLI flag `--product-metrics` (explicit) > `DO_NOT_TRACK` env var (any non-empty value disables) > operator `settings.yaml` `telemetry.productMetrics.enabled` > default `true`. +- **Operator-tier only.** A project-tier `.mecatl/settings.yaml` `telemetry:` block is parsed but IGNORED with a WARN — mirrors the existing `guardrails:`/`openrouter:`/`posture:` discipline exactly. +- **Zero import relationship** between `internal/adapter/productmetrics` and `internal/adapter/telemetry`. Combined only via a fan-out at the composition edge. +- **No free text ever becomes a metric attribute.** Every label value is a bounded Go closed-set type defined in this package, or an already-existing bounded enum (`session.StopReason`, the token-kind strings). +- **Metric names are namespaced `mecatl.adoption.*`** (passes the collector's `^mecatl\..*` server-side filter; visually distinct from the operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family). +- **Scope refinement vs the spec:** the spec's illustrative catalog listed `teams`/`subagents`/`learning` as heartbeat *feature flags*. This plan drops them from the heartbeat (no reliable, already-verified boolean signal exists at CLI-flag level for "learning enabled", and "teams"/"subagents" are core engine capabilities, not togglable features) and instead captures Subagent/Team **adoption** via the event tap (`mecatl.adoption.subagent_used`/`team_used`, already in the spec's coarse-usage catalog) — the more honest signal. The heartbeat's `feature_enabled` set for this plan is `memory`, `guardrails`, `mcp`, `scheduling` — every one backed by a real, already-verified `cmd/mecated` config field (see Task 11). +- **Exporter failures must never affect the app.** Lazy-dial exporter (matches the existing OTLP exporters in `internal/adapter/telemetry/otlp.go`); bounded `Shutdown` (~3s). +- **`task lint && task test`** must stay green after every task. + +--- + +### Task 1: `productmetrics` package skeleton — closed enums, `FeatureSnapshot`, `Config` + +**Files:** +- Create: `internal/adapter/productmetrics/config.go` +- Test: `internal/adapter/productmetrics/config_test.go` + +**Interfaces:** +- Produces: `type Feature string` + consts `FeatureMemory`, `FeatureGuardrails`, `FeatureMCP`, `FeatureScheduling`; `type ProviderFamily string` + consts `ProviderAnthropic`, `ProviderOpenAI`, `ProviderOpenRouter`, `ProviderOther`; `type DeploymentMode string` + consts `ModeInteractive`, `ModeHeadless`, `ModeK8s`; `type Binary string` + consts `BinaryMecated`, `BinaryMecatui`, `BinaryMecatequi`, `BinaryMecak8s`; `type FeatureSnapshot struct{ Memory, Guardrails, MCP, Scheduling bool; Provider ProviderFamily; Mode DeploymentMode }` with method `func (s FeatureSnapshot) enabled() map[Feature]bool`; `type Config struct{ Binary Binary; Version string; InstallID string }`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import "testing" + +func TestFeatureSnapshotEnabledIsClosedAndBounded(t *testing.T) { + snap := FeatureSnapshot{Memory: true, MCP: true, Provider: ProviderAnthropic, Mode: ModeInteractive} + got := snap.enabled() + + want := map[Feature]bool{ + FeatureMemory: true, + FeatureGuardrails: false, + FeatureMCP: true, + FeatureScheduling: false, + } + if len(got) != len(want) { + t.Fatalf("enabled() returned %d entries, want %d (%v)", len(got), len(want), got) + } + for f, v := range want { + if got[f] != v { + t.Errorf("enabled()[%q] = %v, want %v", f, got[f], v) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestFeatureSnapshotEnabledIsClosedAndBounded -v` +Expected: FAIL — package doesn't exist yet / `enabled` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package productmetrics is a fully independent, opt-out-by-default OTel +// metrics adapter reporting bounded adoption/usage counters to Stacklok's +// public metrics collector. It shares no import, struct, MeterProvider, or +// destination with internal/adapter/telemetry (mecatl's operator-facing +// observability pipeline) — the two are combined only at the composition +// edge (internal/cliconfig), by fanning both into the engine's +// port.EventSink/port.ToolCallRecorder seams. +// +// Every exported type in this package that can become a metric attribute is +// a closed Go string-alias enum. Nothing here carries a session id, model +// id/alias, tool or MCP-server name, file path, or free text. +package productmetrics + +// Feature is the closed set of major toggleable features reported at +// heartbeat time. Never a def/model/tool name — only these four values. +type Feature string + +const ( + FeatureMemory Feature = "memory" + FeatureGuardrails Feature = "guardrails" + FeatureMCP Feature = "mcp" + FeatureScheduling Feature = "scheduling" +) + +// ProviderFamily is the closed set of configured LLM provider families. +// Never a model id or alias. +type ProviderFamily string + +const ( + ProviderAnthropic ProviderFamily = "anthropic" + ProviderOpenAI ProviderFamily = "openai" + ProviderOpenRouter ProviderFamily = "openrouter" + ProviderOther ProviderFamily = "other" +) + +// DeploymentMode is the closed set of process shapes. +type DeploymentMode string + +const ( + ModeInteractive DeploymentMode = "interactive" + ModeHeadless DeploymentMode = "headless" + ModeK8s DeploymentMode = "k8s" +) + +// Binary is the closed set of the four mecatl entry points. +type Binary string + +const ( + BinaryMecated Binary = "mecated" + BinaryMecatui Binary = "mecatui" + BinaryMecatequi Binary = "mecatequi" + BinaryMecak8s Binary = "mecak8s" +) + +// FeatureSnapshot is a closed-shape, read-only snapshot of which major +// features are enabled and which provider family / deployment mode this +// process runs as. It carries no free text and no model id/alias. +type FeatureSnapshot struct { + Memory bool + Guardrails bool + MCP bool + Scheduling bool + Provider ProviderFamily + Mode DeploymentMode +} + +// enabled returns every Feature mapped to whether this snapshot reports it +// enabled. It is the single place Heartbeat iterates, so adding a Feature +// const without adding it here is caught by the exhaustiveness this map +// documents (and by TestFeatureSnapshotEnabledIsClosedAndBounded above). +func (s FeatureSnapshot) enabled() map[Feature]bool { + return map[Feature]bool{ + FeatureMemory: s.Memory, + FeatureGuardrails: s.Guardrails, + FeatureMCP: s.MCP, + FeatureScheduling: s.Scheduling, + } +} + +// Config configures a Provider/Recorder pair for one process. +type Config struct { + // Binary identifies which of the four entry points this process is. + Binary Binary + // Version is the mecatl build version (resource attribute service.version). + Version string + // InstallID is this process's persisted anonymous install identifier. + InstallID string +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestFeatureSnapshotEnabledIsClosedAndBounded -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add internal/adapter/productmetrics/config.go internal/adapter/productmetrics/config_test.go +git commit -m "feat(productmetrics): add closed enums, FeatureSnapshot, Config" +``` + +--- + +### Task 2: Install identity persistence + +**Files:** +- Create: `internal/adapter/productmetrics/installid.go` +- Test: `internal/adapter/productmetrics/installid_test.go` + +**Interfaces:** +- Consumes: `xdgconfig.ResolveEnv`, `xdgconfig.UserStateDir(env)` (`internal/adapter/xdgconfig`, already read: `func UserStateDir(env ResolveEnv) string`). +- Produces: `func LoadOrCreateInstallID(env xdgconfig.ResolveEnv, readFile func(string) ([]byte, error), writeFile func(string, []byte, os.FileMode) error, mkdirAll func(string, os.FileMode) error) (id string, firstRun bool, err error)` and a default-wiring convenience `func LoadOrCreateInstallIDDefault() (id string, firstRun bool, err error)`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "errors" + "os" + "testing" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func TestLoadOrCreateInstallIDCreatesOnFirstRun(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + written := map[string][]byte{} + readFile := func(path string) ([]byte, error) { + data, ok := written[path] + if !ok { + return nil, os.ErrNotExist + } + return data, nil + } + writeFile := func(path string, data []byte, _ os.FileMode) error { + written[path] = data + return nil + } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on an empty store, want true") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + + // Second call reads back the SAME id and reports firstRun=false. + id2, firstRun2, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateInstallID: %v", err) + } + if firstRun2 { + t.Error("firstRun = true on second call, want false") + } + if id2 != id { + t.Errorf("second call returned id %q, want %q (unchanged)", id2, id) + } +} + +func TestLoadOrCreateInstallIDRegeneratesOnCorruptFile(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + readFile := func(string) ([]byte, error) { return []byte("not-a-uuid"), nil } + var gotWrite []byte + writeFile := func(_ string, data []byte, _ os.FileMode) error { gotWrite = data; return nil } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on a corrupt file, want true (treated as absent)") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + if string(gotWrite) != id { + t.Errorf("written content %q != returned id %q", gotWrite, id) + } +} + +func TestLoadOrCreateInstallIDFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + _, _, err := LoadOrCreateInstallID(env, nil, nil, nil) + if err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestLoadOrCreateInstallID -v` +Expected: FAIL — `LoadOrCreateInstallID` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// installIDRelPath is the state-dir-relative path to the persisted anonymous +// install identifier — machine-written runtime state, not human config, so +// it lives under XDG_STATE_HOME (mirroring mecatui's +// $XDG_STATE_HOME/mecatl/mecatui.log precedent), not XDG_CONFIG_HOME. +const installIDRelPath = "mecatl/telemetry-id" + +// LoadOrCreateInstallID reads the persisted install UUID, creating one if +// absent or unparseable. The id is a bare random v4 UUID: it carries no +// machine or user information, and is trivially reset by deleting the file +// (the next opt-in mints a new one). firstRun is true whenever a new id was +// just minted — the caller uses it to decide whether to print the one-time +// disclosure notice. readFile/writeFile/mkdirAll are injected for testing; +// LoadOrCreateInstallIDDefault binds the real filesystem. +func LoadOrCreateInstallID( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (id string, firstRun bool, err error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return "", false, fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + path := filepath.Join(base, installIDRelPath) + + if readFile != nil { + if data, rerr := readFile(path); rerr == nil { + if existing := strings.TrimSpace(string(data)); existing != "" { + if _, perr := uuid.Parse(existing); perr == nil { + return existing, false, nil + } + // Corrupt file: fall through and regenerate. + } + } + } + + fresh := uuid.NewString() + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return "", false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte(fresh), 0o600); werr != nil { + return "", false, fmt.Errorf("productmetrics: write install id: %w", werr) + } + } + return fresh, true, nil +} + +// LoadOrCreateInstallIDDefault binds LoadOrCreateInstallID to the real +// process environment and filesystem. +func LoadOrCreateInstallIDDefault() (id string, firstRun bool, err error) { + return LoadOrCreateInstallID(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestLoadOrCreateInstallID -v` +Expected: PASS (all three tests) + +- [ ] **Step 5: Commit** + +```bash +git add internal/adapter/productmetrics/installid.go internal/adapter/productmetrics/installid_test.go +git commit -m "feat(productmetrics): persist an anonymous random install id" +``` + +--- + +### Task 3: OTLP provider construction via toolhive-core + +**Files:** +- Create: `internal/adapter/productmetrics/provider.go` +- Test: `internal/adapter/productmetrics/provider_test.go` +- Modify: `Taskfile.yml:46` (extend `BUILD_LDFLAGS` with the baked-key `-X` flag) + +**Interfaces:** +- Consumes: `providers.NewCompositeProvider(ctx, ...ProviderOption) (*providers.CompositeProvider, error)`, `providers.WithServiceName`, `WithServiceVersion`, `WithOTLPEndpoint`, `WithMetricsEnabled`, `WithHeaders`, `WithCustomAttributes` (`github.com/stacklok/toolhive-core/telemetry/providers`, already verified); `Config` from Task 1. +- Produces: `type Provider struct{...}`, `func NewProvider(ctx context.Context, cfg Config) (*Provider, error)`, `func (p *Provider) Meter() metric.MeterProvider`, `func (p *Provider) Shutdown(ctx context.Context) error`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "context" + "net/http" + "net/http/httptest" + "testing" +) + +func TestNewProviderFailsClosedWithNoBakedKey(t *testing.T) { + orig := bakedKey + bakedKey = "" + defer func() { bakedKey = orig }() + + _, err := NewProvider(context.Background(), Config{Binary: BinaryMecated, Version: "test"}) + if err == nil { + t.Fatal("expected an error when no ingest key is baked into the build, got nil") + } +} + +func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { + origKey := bakedKey + bakedKey = "test-key" + defer func() { bakedKey = origKey }() + + var gotHeader string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotHeader = r.Header.Get(headerKeyName) + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + origEndpoint := endpoint + endpoint = srv.URL + defer func() { endpoint = origEndpoint }() + + p, err := NewProvider(context.Background(), Config{ + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", + }) + if err != nil { + t.Fatalf("NewProvider: %v", err) + } + defer p.Shutdown(context.Background()) + + meter := p.Meter().Meter("test") + counter, cerr := meter.Int64Counter("mecatl.adoption.test") + if cerr != nil { + t.Fatalf("Int64Counter: %v", cerr) + } + counter.Add(context.Background(), 1) + if err := p.Shutdown(context.Background()); err != nil { + t.Fatalf("Shutdown: %v", err) + } + if gotHeader != "test-key" { + t.Errorf("collector received %s=%q, want %q", headerKeyName, gotHeader, "test-key") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestNewProvider -v` +Expected: FAIL — `NewProvider`/`bakedKey`/`endpoint`/`headerKeyName` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/toolhive-core/telemetry/providers" +) + +// endpoint and headerKeyName are the ONE destination this pipeline can ever +// send to (stacklok/infra#5604): a dedicated, internet-facing OTLP/HTTP +// ingest at metrics.stacklok.com, gated by a single shared key baked into +// the binary. Neither is operator-configurable — an operator's own +// --otlp-endpoint has zero effect on this path, and this path has zero +// effect on the operator's own OTLP/Prometheus pipeline (a completely +// separate MeterProvider, never installed as global). endpoint is a var +// (not a const) so tests can point it at an httptest server. +var ( + endpoint = "https://metrics.stacklok.com/v1/metrics" + headerKeyName = "x-mecatl-metrics-key" +) + +// bakedKey is the shared ingest key baked into the binary at build time via +// `-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey=…` +// (see Taskfile.yml's BUILD_LDFLAGS). An empty key — every local/dev/CI-test +// build that does not set the ldflag — disables the pipeline entirely: +// NewProvider refuses to construct, so a non-release build can never +// accidentally phone home with an invalid or absent key. +var bakedKey = "" + +// Provider wraps the toolhive-core OTLP metrics provider. Its MeterProvider +// is NEVER installed as the process-global provider (mirrors +// internal/adapter/telemetry's own discipline in otlp.go), so it cannot +// collide with an operator's own OTel setup. +type Provider struct { + composite *providers.CompositeProvider +} + +// NewProvider builds the product-metrics MeterProvider for one process. A +// network-unreachable endpoint is NOT an error here — the OTLP/HTTP +// exporter dials lazily on first export, matching the existing exporters in +// internal/adapter/telemetry/otlp.go. +func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { + if bakedKey == "" { + return nil, fmt.Errorf("productmetrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") + } + composite, err := providers.NewCompositeProvider(ctx, + providers.WithServiceName("mecatl"), + providers.WithServiceVersion(cfg.Version), + providers.WithOTLPEndpoint(endpoint), + providers.WithMetricsEnabled(true), + providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), + providers.WithCustomAttributes(map[string]string{ + "mecatl.install.id": cfg.InstallID, + "mecatl.binary": string(cfg.Binary), + }), + ) + if err != nil { + return nil, fmt.Errorf("productmetrics: build provider: %w", err) + } + return &Provider{composite: composite}, nil +} + +// Meter returns the underlying metric.MeterProvider for instrument construction. +func (p *Provider) Meter() metric.MeterProvider { return p.composite.MeterProvider() } + +// Shutdown flushes and stops the provider, bounded by the caller's ctx. +func (p *Provider) Shutdown(ctx context.Context) error { return p.composite.Shutdown(ctx) } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestNewProvider -v` +Expected: PASS (both tests) + +- [ ] **Step 5: Extend the release build's ldflags** + +In `Taskfile.yml`, change line 46 from: + +```yaml + BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}}' +``` + +to: + +```yaml + BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}} -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.MECATL_METRICS_KEY}}' +``` + +`MECATL_METRICS_KEY` is an env-driven Taskfile variable (empty for every local/dev build — the byte-identical "disabled" posture from Step 3 — set only by the release CI job from a repo secret). Add near the top of `Taskfile.yml` alongside the existing `vars:` block: + +```yaml + MECATL_METRICS_KEY: '{{.MECATL_METRICS_KEY | default ""}}' +``` + +- [ ] **Step 6: Run the full build to confirm it still compiles with an empty key** + +Run: `task build` +Expected: succeeds; `bin/mecated` etc. are built with `bakedKey=""` (unchanged local-dev behavior). + +- [ ] **Step 7: Commit** + +```bash +git add internal/adapter/productmetrics/provider.go internal/adapter/productmetrics/provider_test.go Taskfile.yml +git commit -m "feat(productmetrics): build the OTLP provider via toolhive-core, baked-key gated" +``` + +--- + +### Task 4: `Recorder` — `port.EventSink` (sessions/runs/tokens) + +**Files:** +- Create: `internal/adapter/productmetrics/metrics.go` +- Test: `internal/adapter/productmetrics/metrics_test.go` + +**Interfaces:** +- Consumes: `metric.MeterProvider` (Task 3's `Provider.Meter()`), `session.Event`/`session.EventType`/`session.ResultPayload`/`session.StopReason`/`session.Usage` (`engine/session`, already verified), `port.EventSink` (`engine/port`). +- Produces: `type Recorder struct{...}`, `func NewRecorder(mp metric.MeterProvider) (*Recorder, error)`, `func (r *Recorder) Emit(ctx context.Context, ev session.Event)` (satisfies `port.EventSink`). + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "context" + "testing" + + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +func newTestRecorder(t *testing.T) (*Recorder, *sdkmetric.ManualReader) { + t.Helper() + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + r, err := NewRecorder(mp) + if err != nil { + t.Fatalf("NewRecorder: %v", err) + } + return r, reader +} + +func collect(t *testing.T, reader *sdkmetric.ManualReader) map[string]metricdata.Aggregation { + t.Helper() + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + out := make(map[string]metricdata.Aggregation) + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + out[md.Name] = md.Data + } + } + return out +} + +func sumValue(t *testing.T, agg metricdata.Aggregation) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + var total int64 + for _, dp := range sum.DataPoints { + total += dp.Value + } + return total +} + +func sumPoint(t *testing.T, agg metricdata.Aggregation, key, value string) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + for _, dp := range sum.DataPoints { + if v, present := dp.Attributes.Value(attribute.Key(key)); present && v.AsString() == value { + return dp.Value + } + } + t.Fatalf("no data point with %s=%q", key, value) + return 0 +} + +func TestRecorderEmitSessionsStarted(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + + agg, ok := collect(t, reader)["mecatl.adoption.sessions_started"] + if !ok { + t.Fatal("mecatl.adoption.sessions_started missing") + } + if got := sumValue(t, agg); got != 2 { + t.Errorf("sessions_started = %d, want 2", got) + } +} + +func TestRecorderEmitRunsCompletedByStopReason(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.adoption.runs_completed"] + if got := sumPoint(t, agg, "stop", "end_turn"); got != 1 { + t.Errorf("runs_completed{stop=end_turn} = %d, want 1", got) + } + if got := sumPoint(t, agg, "stop", "error"); got != 1 { + t.Errorf("runs_completed{stop=error} = %d, want 1", got) + } +} + +func TestRecorderEmitTokensByKind(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopEndTurn, + Usage: session.Usage{ + InputTokens: 100, + OutputTokens: 50, + CacheReadTokens: 20, + CacheWriteTokens: 5, + ReasoningTokens: 10, + }, + }, + }) + + agg := collect(t, reader)["mecatl.adoption.tokens"] + cases := map[string]int64{"input": 100, "output": 50, "cache_read": 20, "cache_write": 5, "reasoning": 10} + for kind, want := range cases { + if got := sumPoint(t, agg, "kind", kind); got != want { + t.Errorf("tokens{kind=%s} = %d, want %d", kind, got, want) + } + } +} + +func TestRecorderEmitSubagentAndTeamUsed(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.adoption.subagent_used"]); got != 1 { + t.Errorf("subagent_used = %d, want 1", got) + } + if got := sumValue(t, collected["mecatl.adoption.team_used"]); got != 1 { + t.Errorf("team_used = %d, want 1", got) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderEmit -v` +Expected: FAIL — `Recorder`/`NewRecorder` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// meterName is the instrumentation scope name for this package's meter. +const meterName = "github.com/stacklok/mecatl/internal/adapter/productmetrics" + +// Attribute keys. Every value ever attached under these keys is drawn from a +// bounded closed set (session.StopReason, the fixed token-kind strings, or +// this package's own Feature/ProviderFamily/DeploymentMode enums) — never a +// session id, model id, tool name, or free text. +const ( + attrStop = "stop" + attrKind = "kind" + attrFeature = "feature" + attrProvider = "family" + attrMode = "mode" +) + +// Recorder is the product-metrics adapter: it implements port.EventSink +// (this file) and port.ToolCallRecorder (toolcall.go), deriving ONLY the +// bounded counts in the design's catalog. It never reads a tool name, +// session id, model id, or any free-text field. +type Recorder struct { + heartbeat metric.Int64Counter + featureEnabled metric.Int64Counter + providerConfig metric.Int64Counter + deploymentMode metric.Int64Counter + sessionsStarted metric.Int64Counter + runsCompleted metric.Int64Counter + toolCalls metric.Int64Counter + tokens metric.Int64Counter + subagentUsed metric.Int64Counter + teamUsed metric.Int64Counter +} + +// Compile-time interface checks. +var ( + _ port.EventSink = (*Recorder)(nil) + _ port.ToolCallRecorder = (*Recorder)(nil) +) + +// NewRecorder constructs every instrument from the given MeterProvider. It +// returns an error if any instrument fails to construct — the OTel meter API +// is fallible. +func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { + meter := mp.Meter(meterName) + r := &Recorder{} + var err error + + if r.heartbeat, err = meter.Int64Counter("mecatl.adoption.heartbeat", + metric.WithDescription("Process liveness heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: heartbeat counter: %w", err) + } + if r.featureEnabled, err = meter.Int64Counter("mecatl.adoption.feature_enabled", + metric.WithDescription("Major feature enabled, by closed feature name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: feature_enabled counter: %w", err) + } + if r.providerConfig, err = meter.Int64Counter("mecatl.adoption.provider_configured", + metric.WithDescription("Configured LLM provider family, by closed family name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: provider_configured counter: %w", err) + } + if r.deploymentMode, err = meter.Int64Counter("mecatl.adoption.deployment_mode", + metric.WithDescription("Process deployment mode, by closed mode name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: deployment_mode counter: %w", err) + } + if r.sessionsStarted, err = meter.Int64Counter("mecatl.adoption.sessions_started", + metric.WithDescription("Total sessions started.")); err != nil { + return nil, fmt.Errorf("productmetrics: sessions_started counter: %w", err) + } + if r.runsCompleted, err = meter.Int64Counter("mecatl.adoption.runs_completed", + metric.WithDescription("Total runs completed, by bounded stop reason.")); err != nil { + return nil, fmt.Errorf("productmetrics: runs_completed counter: %w", err) + } + if r.toolCalls, err = meter.Int64Counter("mecatl.adoption.tool_calls", + metric.WithDescription("Total tool calls executed (no tool identity attached).")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls counter: %w", err) + } + if r.tokens, err = meter.Int64Counter("mecatl.adoption.tokens", + metric.WithDescription("Total tokens accounted, by bounded kind."), + metric.WithUnit("{token}")); err != nil { + return nil, fmt.Errorf("productmetrics: tokens counter: %w", err) + } + if r.subagentUsed, err = meter.Int64Counter("mecatl.adoption.subagent_used", + metric.WithDescription("Runs that used the Subagent delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: subagent_used counter: %w", err) + } + if r.teamUsed, err = meter.Int64Counter("mecatl.adoption.team_used", + metric.WithDescription("Runs that used the Team delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: team_used counter: %w", err) + } + return r, nil +} + +// Emit derives coarse, bounded counts from a single domain Event. It reads +// ONLY ev.Type, ev.Result.Stop, and ev.Result.Usage — never a session id, +// model id/alias, tool name, or any free-text field (ev.Result.Text/Error +// are never touched). +func (r *Recorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + r.sessionsStarted.Add(ctx, 1) + case session.EvResult: + r.recordResult(ctx, ev.Result) + case session.EvSubagentStart: + r.subagentUsed.Add(ctx, 1) + case session.EvTeamStart: + r.teamUsed.Add(ctx, 1) + } +} + +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload) { + if res == nil { + r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(session.StopNone)))) + return + } + r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(res.Stop)))) + u := res.Usage + r.tokens.Add(ctx, int64(u.InputTokens), metric.WithAttributes(attribute.String(attrKind, "input"))) + r.tokens.Add(ctx, int64(u.OutputTokens), metric.WithAttributes(attribute.String(attrKind, "output"))) + r.tokens.Add(ctx, int64(u.CacheReadTokens), metric.WithAttributes(attribute.String(attrKind, "cache_read"))) + r.tokens.Add(ctx, int64(u.CacheWriteTokens), metric.WithAttributes(attribute.String(attrKind, "cache_write"))) + r.tokens.Add(ctx, int64(u.ReasoningTokens), metric.WithAttributes(attribute.String(attrKind, "reasoning"))) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderEmit -v` +Expected: PASS (all four tests) + +- [ ] **Step 5: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/metrics_test.go +git commit -m "feat(productmetrics): Recorder.Emit derives bounded session/run/token counts" +``` + +--- + +### Task 5: `Recorder` — `port.ToolCallRecorder` and heartbeat + +**Files:** +- Create: `internal/adapter/productmetrics/toolcall.go` +- Create: `internal/adapter/productmetrics/heartbeat.go` +- Test: `internal/adapter/productmetrics/toolcall_test.go` +- Test: `internal/adapter/productmetrics/heartbeat_test.go` + +**Interfaces:** +- Consumes: `port.ToolCallRecorder.ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration)` (verified signature); `Recorder` from Task 4; `FeatureSnapshot` from Task 1. +- Produces: `func (r *Recorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration)`; `func (r *Recorder) Heartbeat(snap FeatureSnapshot)`; `const DefaultHeartbeatInterval = 24 * time.Hour`; `func RunHeartbeat(ctx context.Context, r *Recorder, interval time.Duration, snap FeatureSnapshot)`. + +- [ ] **Step 1: Write the failing test (ToolCall)** + +```go +package productmetrics + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +func TestRecorderToolCallCountsWithoutIdentity(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCall( + session.SessionID("sensitive-session-id"), + session.ToolCall{Name: "read_secret_file"}, + session.ToolResult{Content: "super secret content", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.ToolCall(session.SessionID("other"), session.ToolCall{Name: "another_tool"}, session.ToolResult{}, 0, 0) + + agg := collect(t, reader)["mecatl.adoption.tool_calls"] + if got := sumValue(t, agg); got != 2 { + t.Errorf("tool_calls = %d, want 2", got) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderToolCallCountsWithoutIdentity -v` +Expected: FAIL — `Recorder.ToolCall` undefined. + +- [ ] **Step 3: Write minimal implementation (toolcall.go)** + +```go +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +// ToolCall records ONLY that a tool call happened — no tool name, no +// session id, no result content, no duration. It satisfies +// port.ToolCallRecorder. The three typed parameters it ignores (id, call, +// result) are accepted only because the port's signature requires them; not +// one of their fields is ever read. +func (r *Recorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + r.toolCalls.Add(context.Background(), 1) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderToolCallCountsWithoutIdentity -v` +Expected: PASS + +- [ ] **Step 5: Write the failing test (Heartbeat)** + +```go +package productmetrics + +import ( + "context" + "testing" + "time" +) + +func TestRecorderHeartbeatRecordsClosedLabelsOnly(t *testing.T) { + r, reader := newTestRecorder(t) + r.Heartbeat(FeatureSnapshot{ + Memory: true, MCP: true, + Provider: ProviderAnthropic, Mode: ModeInteractive, + }) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.adoption.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want 1", got) + } + featureAgg := collected["mecatl.adoption.feature_enabled"] + if got := sumPoint(t, featureAgg, "feature", "memory"); got != 1 { + t.Errorf("feature_enabled{feature=memory} = %d, want 1", got) + } + if got := sumPoint(t, featureAgg, "feature", "mcp"); got != 1 { + t.Errorf("feature_enabled{feature=mcp} = %d, want 1", got) + } + // guardrails/scheduling were false in the snapshot: TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent + // (Task 6) is the exhaustive "no other data point exists" check; this test + // only asserts the enabled ones are present with the right value. + if got := sumPoint(t, collected["mecatl.adoption.provider_configured"], "family", "anthropic"); got != 1 { + t.Errorf("provider_configured{family=anthropic} = %d, want 1", got) + } + if got := sumPoint(t, collected["mecatl.adoption.deployment_mode"], "mode", "interactive"); got != 1 { + t.Errorf("deployment_mode{mode=interactive} = %d, want 1", got) + } +} + +func TestRunHeartbeatFiresImmediatelyThenStopsOnCtxDone(t *testing.T) { + r, reader := newTestRecorder(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancelled BEFORE RunHeartbeat: only the immediate fire happens. + + RunHeartbeat(ctx, r, time.Hour, FeatureSnapshot{Mode: ModeHeadless}) + + if got := sumValue(t, collect(t, reader)["mecatl.adoption.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want exactly 1 (immediate fire only)", got) + } +} +``` + +- [ ] **Step 6: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderHeartbeat -v` and `-run TestRunHeartbeat` +Expected: FAIL — `Heartbeat`/`RunHeartbeat` undefined. + +- [ ] **Step 7: Write minimal implementation (heartbeat.go)** + +```go +package productmetrics + +import ( + "context" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// DefaultHeartbeatInterval is the steady-state heartbeat cadence for +// long-running processes (mecated, mecatui, mecak8s). mecatequi (short-lived) +// passes 0 — a single immediate fire only, no ticker. +const DefaultHeartbeatInterval = 24 * time.Hour + +// Heartbeat records the periodic liveness + feature/provider/mode signal. +// Every attribute value comes from the closed Feature/ProviderFamily/ +// DeploymentMode enums — never a def/model/tool name. +func (r *Recorder) Heartbeat(snap FeatureSnapshot) { + ctx := context.Background() + r.heartbeat.Add(ctx, 1) + for feature, on := range snap.enabled() { + if on { + r.featureEnabled.Add(ctx, 1, metric.WithAttributes(attribute.String(attrFeature, string(feature)))) + } + } + r.providerConfig.Add(ctx, 1, metric.WithAttributes(attribute.String(attrProvider, string(snap.Provider)))) + r.deploymentMode.Add(ctx, 1, metric.WithAttributes(attribute.String(attrMode, string(snap.Mode)))) +} + +// RunHeartbeat fires one heartbeat immediately, then one every interval, +// until ctx is done. interval<=0 disables the ticker (a single fire only — +// mecatequi's shape). Meant to run in its own goroutine, owned by the +// caller (composition), which cancels ctx on shutdown. +func RunHeartbeat(ctx context.Context, r *Recorder, interval time.Duration, snap FeatureSnapshot) { + r.Heartbeat(snap) + if interval <= 0 { + return + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + r.Heartbeat(snap) + } + } +} +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd internal/adapter/productmetrics && go test ./... -v` +Expected: PASS (every test in the package so far) + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/toolcall.go internal/adapter/productmetrics/heartbeat.go \ + internal/adapter/productmetrics/toolcall_test.go internal/adapter/productmetrics/heartbeat_test.go +git commit -m "feat(productmetrics): ToolCall counting and the heartbeat ticker" +``` + +--- + +### Task 6: Privacy guard test — exhaustive bounded-attribute check + +**Files:** +- Create: `internal/adapter/productmetrics/bounded_test.go` + +**Interfaces:** +- Consumes: everything from Tasks 1, 4, 5 (`Recorder`, `FeatureSnapshot`, closed enums). +- Produces: no new production code — a test-only safety net that fails CI the moment a future change attaches an unbounded attribute or an unlisted key. + +This test is the automated version of the design doc's privacy invariant: it drives the Recorder with values chosen to look like they'd leak something sensitive if the code were wrong (a "secret"-looking session id, a suspicious tool name, free-text tool output, an odd stop reason), then asserts across *every* collected data point of *every* instrument that (a) every attribute key is in a fixed allowlist, and (b) no attribute value or metric name contains any of the injected "sensitive" substrings. + +- [ ] **Step 1: Write the test** + +```go +package productmetrics + +import ( + "context" + "strings" + "testing" + "time" + + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +// allowedAttributeKeys is the COMPLETE set of attribute keys any instrument +// in this package may ever carry. A future change that attaches a new label +// must add it here explicitly — the same "closed set is a reviewed +// decision" discipline as internal/adapter/telemetry's attrRole. +var allowedAttributeKeys = map[string]bool{ + attrStop: true, + attrKind: true, + attrFeature: true, + attrProvider: true, + attrMode: true, +} + +// sensitiveMarkers are strings injected into every field the Recorder must +// NEVER read. If any of these ever shows up in a collected metric name or +// attribute value, something started reading a field it shouldn't. +var sensitiveMarkers = []string{ + "sensitive-session-id-marker", + "secret-tool-name-marker", + "secret-tool-content-marker", + "secret-error-text-marker", +} + +func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T) { + r, reader := newTestRecorder(t) + + // Drive every observation path with deliberately sensitive-looking data. + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopError, + Text: "sensitive-session-id-marker should never be read", + Error: "secret-error-text-marker: connection to 10.0.0.5 failed", + Usage: session.Usage{InputTokens: 1, OutputTokens: 1}, + }, + }) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + r.ToolCall( + session.SessionID("sensitive-session-id-marker"), + session.ToolCall{Name: "secret-tool-name-marker"}, + session.ToolResult{Content: "secret-tool-content-marker", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.Heartbeat(FeatureSnapshot{ + Memory: true, Guardrails: true, MCP: true, Scheduling: true, + Provider: ProviderOther, Mode: ModeK8s, + }) + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + assertNoSensitiveSubstring(t, md.Name) + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("metric %s: aggregation is %T, want Sum[int64]", md.Name, md.Data) + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + key := string(kv.Key) + if !allowedAttributeKeys[key] { + t.Errorf("metric %s carries attribute key %q, not in allowedAttributeKeys", md.Name, key) + } + assertNoSensitiveSubstring(t, kv.Value.AsString()) + } + } + // mecatl.adoption.tool_calls carries NO attributes at all — the + // strongest form of "no tool identity ever attaches." + if md.Name == "mecatl.adoption.tool_calls" { + for _, dp := range sum.DataPoints { + if dp.Attributes.Len() != 0 { + t.Errorf("mecatl.adoption.tool_calls data point carries %d attributes, want 0: %v", + dp.Attributes.Len(), dp.Attributes) + } + } + } + } + } +} + +func assertNoSensitiveSubstring(t *testing.T, s string) { + t.Helper() + for _, marker := range sensitiveMarkers { + if strings.Contains(s, marker) { + t.Errorf("value %q contains sensitive marker %q", s, marker) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails or passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent -v` +Expected: PASS immediately (Tasks 4-5's implementation already satisfies it) — this task adds no production code, only the safety net. If it fails, the failure output names exactly which instrument/attribute leaked; fix `metrics.go`/`toolcall.go`/`heartbeat.go` until it passes, never loosen this test. + +- [ ] **Step 3: Commit** + +```bash +git add internal/adapter/productmetrics/bounded_test.go +git commit -m "test(productmetrics): exhaustive guard against unbounded/sensitive attributes" +``` + +--- + +### Task 7: `permconfig` schema — `telemetry:` operator subtree + +**Files:** +- Modify: `internal/adapter/permconfig/schema.go` (add `Telemetry *TelemetrySection` field to the top-level `Config` struct near line 154's `OpenRouter` field, plus the new `TelemetrySection`/`ProductMetricsSection` types near line 1202's `OpenRouterSection`) +- Test: `internal/adapter/permconfig/telemetry_schema_test.go` + +**Interfaces:** +- Produces: `type TelemetrySection struct{ ProductMetrics *ProductMetricsSection }`, `type ProductMetricsSection struct{ Enabled *bool }`, both with strict `UnmarshalYAML` (mirroring `OpenRouterSection`, already verified). + +- [ ] **Step 1: Write the failing test** + +```go +package permconfig + +import "testing" + +func TestParseYAMLTelemetryProductMetricsEnabled(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enabled: false\n") + cfg, err := parseYAML(data) + if err != nil { + t.Fatalf("parseYAML: %v", err) + } + if cfg.Telemetry == nil || cfg.Telemetry.ProductMetrics == nil { + t.Fatal("Telemetry.ProductMetrics is nil") + } + if cfg.Telemetry.ProductMetrics.Enabled == nil || *cfg.Telemetry.ProductMetrics.Enabled != false { + t.Errorf("Enabled = %v, want explicit false", cfg.Telemetry.ProductMetrics.Enabled) + } +} + +func TestParseYAMLTelemetryUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productmetric:\n enabled: false\n") // typo: productmetric + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown telemetry.productmetric key, got nil") + } +} + +func TestParseYAMLTelemetryProductMetricsUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enable: false\n") // typo: enable + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown enable key, got nil") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestParseYAMLTelemetry -v` +Expected: FAIL — `Config.Telemetry` undefined. + +- [ ] **Step 3: Add the field to the top-level `Config` struct** + +In `internal/adapter/permconfig/schema.go`, immediately after the `OpenRouter *OpenRouterSection` field (the one ending around line 154), add: + +```go + // Telemetry holds the OPERATOR-TIER `telemetry:` subtree (opt-out product/ + // adoption metrics). Like OpenRouter/Guardrails/Posture it is honoured ONLY + // from the user-global + CLI tiers; a project-tier file's telemetry: block + // is IGNORED with a WARN (a project repo cannot flip a user's own telemetry + // choice in either direction). Parsed STRICTLY (unknown keys error). A nil + // Telemetry means the key was absent — composition then falls through the + // DO_NOT_TRACK env var and finally defaults to enabled. + Telemetry *TelemetrySection `yaml:"telemetry"` +``` + +- [ ] **Step 4: Add the new section types** + +In `internal/adapter/permconfig/schema.go`, immediately after the `OpenRouterSection`/`OpenRouterModelRoute` block (after the code ending around line 1249), add: + +```go +// TelemetrySection is the `telemetry:` operator-tier YAML subtree: the opt-out +// switch for community/adoption product metrics. Parsed STRICTLY (unknown +// keys error), mirroring OpenRouterSection/GuardrailsSection. +type TelemetrySection struct { + // ProductMetrics is the opt-out product/adoption metrics config. + ProductMetrics *ProductMetricsSection `yaml:"productMetrics"` +} + +func (s *TelemetrySection) strictFields() map[string]any { + return map[string]any{ + "productMetrics": &s.ProductMetrics, + } +} + +// UnmarshalYAML decodes the telemetry: mapping STRICTLY: an unknown key +// (e.g. a typo'd product-metrics:) is a parse error, same discipline as +// openrouter:/guardrails:. +func (s *TelemetrySection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry", s.strictFields()) +} + +// ProductMetricsSection is the `telemetry.productMetrics:` subtree. +type ProductMetricsSection struct { + // Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit + // false: nil = absent (composition falls through to DO_NOT_TRACK then the + // enabled-by-default posture); a non-nil value is honoured exactly. + Enabled *bool `yaml:"enabled"` +} + +func (s *ProductMetricsSection) strictFields() map[string]any { + return map[string]any{ + "enabled": newPermconfigNodePointer(&s.Enabled), + } +} + +// UnmarshalYAML decodes the productMetrics: mapping STRICTLY. +func (s *ProductMetricsSection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry.productMetrics", s.strictFields()) +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestParseYAMLTelemetry -v` +Expected: PASS (all three tests) + +- [ ] **Step 6: Commit** + +```bash +git add internal/adapter/permconfig/schema.go internal/adapter/permconfig/telemetry_schema_test.go +git commit -m "feat(permconfig): add the strict telemetry.productMetrics: operator schema" +``` + +--- + +### Task 8: `permconfig` resolver — operator-tier-only capture + project-tier WARN-ignore + +**Files:** +- Modify: `internal/adapter/permconfig/resolve.go` (Resolver struct field near `operatorOpenRouter` (~line 197), capture method near `captureOpenRouter` (~line 1082), accessor near `OperatorOpenRouter` (~line 388), call sites at both `loadUserRules` locations (~line 907 and ~line 946), and the project-tier WARN block inside `loadProjectRules` (~line 672)) +- Test: `internal/adapter/permconfig/telemetry_resolve_test.go` + +**Interfaces:** +- Produces: `func (r *Resolver) OperatorProductMetricsEnabled() *bool` (nil = absent — the SOLE accessor composition uses). + +- [ ] **Step 1: Write the failing test** + +```go +package permconfig + +import ( + "context" + "testing" + + "github.com/stacklok/mecatl/engine/adapter/memfs" + "github.com/stacklok/mecatl/engine/tool" +) + +func TestOperatorProductMetricsEnabledFromUserGlobal(t *testing.T) { + env := fakeEnv(t, map[string]string{ + "mecatl/settings.yaml": "telemetry:\n productMetrics:\n enabled: false\n", + }) + r := NewResolver(Options{Conventional: true}, env, nil) + got := r.OperatorProductMetricsEnabled() + if got == nil || *got != false { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want explicit false", got) + } +} + +func TestOperatorProductMetricsEnabledAbsentIsNil(t *testing.T) { + env := fakeEnv(t, map[string]string{}) + r := NewResolver(Options{Conventional: true}, env, nil) + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want nil (absent)", got) + } +} + +func TestProjectTierTelemetryBlockIsIgnoredWithWarn(t *testing.T) { + ws := memfs.NewWorkspace(t.TempDir()) + if err := ws.Write(context.Background(), ".mecatl/settings.yaml", + []byte("telemetry:\n productMetrics:\n enabled: false\n")); err != nil { + t.Fatalf("seed project file: %v", err) + } + r := NewResolver(Options{Conventional: true}, fakeEnv(t, nil), nil) + if _, _, err := r.Rules(context.Background(), ws.(tool.WorkspaceReader)); err != nil { + t.Fatalf("Rules: %v", err) + } + // A project-tier telemetry: block must NEVER be captured as operator config. + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Errorf("OperatorProductMetricsEnabled() = %v after a PROJECT-tier block, want nil (project-tier is ignored)", got) + } +} +``` + +Note: `fakeEnv`/`NewResolver`/`r.Rules` signatures above must match this package's existing test helpers exactly — before writing this test, read one existing resolver test (e.g. the file containing `TestOperatorOpenRouterFromUserGlobal`-shaped tests, likely `internal/adapter/permconfig/resolve_test.go` or an `openrouter_test.go` sibling) and copy its exact `fakeEnv`/construction idiom rather than inventing a new one. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestOperatorProductMetricsEnabled -v` and `-run TestProjectTierTelemetryBlockIsIgnoredWithWarn` +Expected: FAIL — `OperatorProductMetricsEnabled` undefined. + +- [ ] **Step 3: Add the Resolver field** + +In `internal/adapter/permconfig/resolve.go`, immediately after the `operatorOpenRouter *OpenRouterSection` field, add: + +```go + // operatorTelemetry is the OPERATOR-TIER telemetry: subtree, read ONCE at + // construction from the user-global + CLI tiers ONLY (the SOLE capture path + // is captureTelemetry from loadUserRules — mirroring captureOpenRouter). A + // project-tier file's telemetry: block is IGNORED with a WARN in + // loadProjectRules. nil when no operator-tier file carried a telemetry: + // section. CLI (explicit files) out-ranks user-global (first-non-nil keeps + // CLI). + operatorTelemetry *TelemetrySection +``` + +- [ ] **Step 4: Add the capture method** + +Immediately after `captureOpenRouter`, add: + +```go +// captureTelemetry records the FIRST operator-tier telemetry: block seen +// during loadUserRules (CLI files out-rank user-global, so first-non-nil +// keeps CLI). Mirrors captureOpenRouter. +func (r *Resolver) captureTelemetry(s *TelemetrySection) { + if s == nil || r.operatorTelemetry != nil { + return + } + r.operatorTelemetry = s +} +``` + +- [ ] **Step 5: Add the accessor** + +Immediately after `OperatorOpenRouter`, add: + +```go +// OperatorProductMetricsEnabled returns the operator-tier +// telemetry.productMetrics.enabled: value (user-global + CLI only), or nil +// when none was configured. It is the SOLE accessor composition uses to +// read the product-metrics opt-out from config — by construction it never +// returns a project-tier value (a project telemetry: block is ignored with +// a WARN in loadProjectRules). nil-safe. Mirrors OperatorOpenRouter(). +func (r *Resolver) OperatorProductMetricsEnabled() *bool { + if r == nil || r.operatorTelemetry == nil || r.operatorTelemetry.ProductMetrics == nil { + return nil + } + return r.operatorTelemetry.ProductMetrics.Enabled +} +``` + +- [ ] **Step 6: Wire the two `loadUserRules` capture call sites** + +At both locations identified in the Files list (immediately after each existing `r.captureOpenRouter(cfg.OpenRouter)` line — one in the explicit-CLI-files branch, one in the user-global-YAML branch), add: + +```go + // Operator-tier telemetry (opt-out product metrics): same + // first-non-nil-keeps-CLI discipline as openrouter. + r.captureTelemetry(cfg.Telemetry) +``` + +- [ ] **Step 7: Wire the project-tier WARN-ignore in `loadProjectRules`** + +Immediately after the existing `if cfg.OpenRouter != nil { ... }` WARN block inside `loadProjectRules`, add: + +```go + if cfg.Telemetry != nil { + r.diag.Log(context.Background(), port.LevelWarn, + "telemetry: IGNORING a project-tier telemetry: block (operator-tier only — a project repo cannot change a user's own product-metrics opt-out in either direction; set telemetry in your user-global settings.yaml)", + "file", src.path, "root", ws.Root()) + } +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestOperatorProductMetricsEnabled -v` and `-run TestProjectTierTelemetryBlockIsIgnoredWithWarn` +Expected: PASS + +- [ ] **Step 9: Run the full permconfig suite to catch any regression** + +Run: `cd internal/adapter/permconfig && go test ./...` +Expected: PASS + +- [ ] **Step 10: Commit** + +```bash +git add internal/adapter/permconfig/resolve.go internal/adapter/permconfig/telemetry_resolve_test.go +git commit -m "feat(permconfig): resolve the operator-tier telemetry.productMetrics opt-out" +``` + +--- + +### Task 9: `cliconfig` — opt-out precedence + `ToolCallRecorder` fan-out + +**Files:** +- Create: `internal/cliconfig/productmetrics_config.go` +- Test: `internal/cliconfig/productmetrics_config_test.go` + +**Interfaces:** +- Consumes: `port.EventSink`, `port.ToolCallRecorder`, `session.SessionID`/`session.ToolCall`/`session.ToolResult` (`engine/port`, `engine/session`). +- Produces: `type ProductMetricsPrecedence struct{ FlagSet, FlagValue bool; Getenv func(string) string; SettingsEnabled *bool }`, `func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool`, `func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecorder`. + +- [ ] **Step 1: Write the failing test** + +```go +package cliconfig + +import "testing" + +func boolPtr(b bool) *bool { return &b } + +func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { + getenvSet := func(string) string { return "1" } + getenvUnset := func(string) string { return "" } + + cases := []struct { + name string + p ProductMetricsPrecedence + want bool + }{ + {"flag true wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: true, Getenv: getenvSet, SettingsEnabled: boolPtr(false)}, true}, + {"flag false wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: false, Getenv: getenvUnset, SettingsEnabled: boolPtr(true)}, false}, + {"DO_NOT_TRACK disables when no flag", ProductMetricsPrecedence{Getenv: getenvSet, SettingsEnabled: boolPtr(true)}, false}, + {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: boolPtr(false)}, false}, + {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: nil}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := ResolveProductMetricsEnabled(tc.p); got != tc.want { + t.Errorf("ResolveProductMetricsEnabled(%+v) = %v, want %v", tc.p, got, tc.want) + } + }) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/cliconfig && go test ./... -run TestResolveProductMetricsEnabledPrecedence -v` +Expected: FAIL — `ProductMetricsPrecedence`/`ResolveProductMetricsEnabled` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package cliconfig + +import ( + "os" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// ProductMetricsPrecedence carries the opt-out inputs +// ResolveProductMetricsEnabled folds, highest precedence first: an explicit +// CLI flag, then the DO_NOT_TRACK env var convention (consoledonottrack.com), +// then the operator settings.yaml value, then default-enabled. +type ProductMetricsPrecedence struct { + // FlagSet/FlagValue report whether --product-metrics was explicitly + // passed on the command line and its value. + FlagSet bool + FlagValue bool + // Getenv abstracts os.Getenv for DO_NOT_TRACK / testing. Defaults to + // os.Getenv when nil. + Getenv func(string) string + // SettingsEnabled is permconfig.Resolver.OperatorProductMetricsEnabled() + // — nil when the operator set no telemetry.productMetrics.enabled value. + SettingsEnabled *bool +} + +// ResolveProductMetricsEnabled applies the opt-out precedence documented on +// ProductMetricsPrecedence. Default (nothing set anywhere) is true — product +// metrics are OPT-OUT, not opt-in. +func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { + if p.FlagSet { + return p.FlagValue + } + getenv := p.Getenv + if getenv == nil { + getenv = os.Getenv + } + if getenv("DO_NOT_TRACK") != "" { + return false + } + if p.SettingsEnabled != nil { + return *p.SettingsEnabled + } + return true +} + +// TeeToolCallRecorder combines multiple ToolCallRecorders into one — the +// ToolCallRecorder twin of internal/adapter/telemetry.NewSink's EventSink +// fan-out (no such helper existed before product metrics, because until now +// only one ToolCallRecorder ever observed a given engine). nil entries are +// skipped, so a caller can pass an always-present operator recorder +// alongside an optional product-metrics one without a conditional slice +// build. +func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecorder { + var non []port.ToolCallRecorder + for _, r := range recorders { + if r != nil { + non = append(non, r) + } + } + return multiToolCallRecorder(non) +} + +type multiToolCallRecorder []port.ToolCallRecorder + +func (m multiToolCallRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + for _, r := range m { + r.ToolCall(id, call, result, queued, took) + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/cliconfig && go test ./... -run TestResolveProductMetricsEnabledPrecedence -v` +Expected: PASS (all five subtests) + +- [ ] **Step 5: Add a small test for `TeeToolCallRecorder`** + +```go +func TestTeeToolCallRecorderCallsEveryNonNilRecorder(t *testing.T) { + var calls []string + rec := func(name string) port.ToolCallRecorder { + return recorderFunc(func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + calls = append(calls, name) + }) + } + tee := TeeToolCallRecorder(rec("a"), nil, rec("b")) + tee.ToolCall(session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if len(calls) != 2 || calls[0] != "a" || calls[1] != "b" { + t.Errorf("calls = %v, want [a b] (nil skipped, order preserved)", calls) + } +} + +// recorderFunc adapts a plain func to port.ToolCallRecorder for this test. +type recorderFunc func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) + +func (f recorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f(id, call, result, queued, took) +} +``` + +Add the necessary `"github.com/stacklok/mecatl/engine/port"`, `"github.com/stacklok/mecatl/engine/session"`, and `"time"` imports to the test file if not already present from Step 1. + +- [ ] **Step 6: Run test to verify it passes** + +Run: `cd internal/cliconfig && go test ./... -run TestTeeToolCallRecorderCallsEveryNonNilRecorder -v` +Expected: PASS + +- [ ] **Step 7: Commit** + +```bash +git add internal/cliconfig/productmetrics_config.go internal/cliconfig/productmetrics_config_test.go +git commit -m "feat(cliconfig): product-metrics opt-out precedence + ToolCallRecorder fan-out" +``` + +--- + +### Task 10: `cliconfig` — the `BuildProductMetrics` composition helper + +**Files:** +- Create: `internal/cliconfig/productmetrics.go` +- Test: `internal/cliconfig/productmetrics_test.go` + +**Interfaces:** +- Consumes: everything from `internal/adapter/productmetrics` (Tasks 1-5) and `ResolveProductMetricsEnabled`/`TeeToolCallRecorder` (Task 9). +- Produces: `type ProductMetricsHandles struct{ Sink port.EventSink; ToolCallRecorder port.ToolCallRecorder; Shutdown func(context.Context) error; FirstRun bool }`, `func BuildProductMetrics(ctx, heartbeatCtx context.Context, enabled bool, binary productmetrics.Binary, version string, heartbeatInterval time.Duration, snap productmetrics.FeatureSnapshot) (ProductMetricsHandles, error)`, `const ProductMetricsDisclosureNotice = "..."`. + +- [ ] **Step 1: Write the failing test** + +```go +package cliconfig + +import ( + "context" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=false): %v", err) + } + if h.Sink != nil || h.ToolCallRecorder != nil { + t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder: %+v", h) + } + if h.Shutdown == nil { + t.Fatal("Shutdown must be non-nil even when disabled (a no-op)") + } + if err := h.Shutdown(context.Background()); err != nil { + t.Errorf("no-op Shutdown returned an error: %v", err) + } +} + +func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { + // bakedKey is empty in every non-release build/test — enabling must + // surface the error rather than silently disabling, so a caller notices + // its release build is missing the ldflag. + _, err := BuildProductMetrics(context.Background(), context.Background(), true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + if err == nil { + t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/cliconfig && go test ./... -run TestBuildProductMetrics -v` +Expected: FAIL — `BuildProductMetrics` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package cliconfig + +import ( + "context" + "fmt" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// ProductMetricsDisclosureNotice is printed ONCE — the first run after +// product metrics were enabled and this install's telemetry id did not yet +// exist — to stderr, non-blockingly, before any pipeline is built. Opt-out +// telemetry without a visible disclosure is the pattern that burns +// community trust; this is the whole of that disclosure. +const ProductMetricsDisclosureNotice = `mecatl reports anonymous product-adoption metrics (version, OS/arch, which +major features you have enabled, and coarse session/run/tool-call counts — +never a prompt, file path, tool name, or model id) to help Stacklok understand +community adoption. This is on by default. To opt out: pass +--product-metrics=false, set DO_NOT_TRACK=1, or set +telemetry.productMetrics.enabled: false in your settings.yaml. Details: +. +` + +// ProductMetricsHandles bundles the handles a cmd main threads into its +// EventSink/ToolCallRecorder fan-out (via TeeToolCallRecorder / +// internal/adapter/telemetry.NewSink alongside the operator sink) and its +// shutdown defer. Every field is zero-valued when telemetry is disabled. +type ProductMetricsHandles struct { + Sink port.EventSink + ToolCallRecorder port.ToolCallRecorder + // Shutdown flushes + stops the provider. Always non-nil (a no-op when + // disabled), so a caller can defer it unconditionally. + Shutdown func(context.Context) error + // FirstRun is true the first time this install's telemetry id was just + // minted — the caller prints ProductMetricsDisclosureNotice when true. + FirstRun bool +} + +// BuildProductMetrics constructs the full opt-out product-metrics pipeline +// when enabled is true; when false it returns zero handles (the +// byte-identical disabled posture) and no error. heartbeatInterval is +// productmetrics.DefaultHeartbeatInterval for long-running processes, or 0 +// for a single-fire-only short-lived process (mecatequi). heartbeatCtx is +// cancelled by the caller on shutdown to stop the periodic ticker goroutine +// this starts. +func BuildProductMetrics( + ctx, heartbeatCtx context.Context, + enabled bool, + binary productmetrics.Binary, + version string, + heartbeatInterval time.Duration, + snap productmetrics.FeatureSnapshot, +) (ProductMetricsHandles, error) { + noop := func(context.Context) error { return nil } + if !enabled { + return ProductMetricsHandles{Shutdown: noop}, nil + } + + installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + InstallID: installID, + }) + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: provider: %w", err) + } + + recorder, err := productmetrics.NewRecorder(provider.Meter()) + if err != nil { + _ = provider.Shutdown(ctx) + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: recorder: %w", err) + } + + go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) + + return ProductMetricsHandles{ + Sink: recorder, + ToolCallRecorder: recorder, + Shutdown: provider.Shutdown, + FirstRun: firstRun, + }, nil +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/cliconfig && go test ./... -run TestBuildProductMetrics -v` +Expected: PASS (both tests) + +- [ ] **Step 5: Commit** + +```bash +git add internal/cliconfig/productmetrics.go internal/cliconfig/productmetrics_test.go +git commit -m "feat(cliconfig): BuildProductMetrics composition helper + disclosure notice" +``` + +--- + +### Task 11: Wire into `cmd/mecated` + +**Files:** +- Modify: `cmd/mecated/main.go` + +**Interfaces:** +- Consumes: `cliconfig.ResolveProductMetricsEnabled`, `cliconfig.BuildProductMetrics`, `cliconfig.TeeToolCallRecorder` (Tasks 9-10); `internal/adapter/permconfig` resolver's `OperatorProductMetricsEnabled()` (Task 8) — mecated already builds a `permconfig.Resolver` for its permission rules; thread its `OperatorProductMetricsEnabled()` value through to this wiring. +- Produces: a new `--product-metrics` flag, defaulting `true`; the fanned-in sink/recorder feeding the existing `sink`/`mainScoped` variables at lines ~927/949. + +- [ ] **Step 1: Add the flag** + +Near the existing `fs.StringVar(&cfg.otlpEndpoint, "otlp-endpoint", ...)` registration (line 1610), add a new `cfg` field and flag: + +```go + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") +``` + +Add `productMetrics bool` to the `config` struct near the existing `otlpEndpoint string` field. + +- [ ] **Step 2: Compute `cliExplicit["product-metrics"]` (already-established mechanism)** + +No new code needed here: `cmd/mecated/main.go`'s existing `fs.Visit(func(f *flag.Flag) { cfg.cliExplicit[f.Name] = true })` (around line 1877-1879) already marks every explicitly-passed flag by name, so `cfg.cliExplicit["product-metrics"]` is populated for free once Step 1's flag is registered. + +- [ ] **Step 3: Resolve the effective enabled value and build the handles** + +In `run()`, immediately after `obs, err := setupObservability(ctx, cfg, diag)` (line 899) and before its `defer` block, add: + +```go + productMetricsEnabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.cliExplicit["product-metrics"], + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + defer cancelHeartbeat() + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, + productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot(cfg)) + if err != nil { + slog.Warn("product metrics disabled: setup failed", "err", err) + } else { + defer func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if serr := pm.Shutdown(shutdownCtx); serr != nil { + slog.Warn("product metrics shutdown", "err", serr) + } + }() + if pm.FirstRun { + fmt.Fprint(os.Stderr, cliconfig.ProductMetricsDisclosureNotice) + } + } +``` + +Replace `permResolver` above with whatever local variable name `cmd/mecated/main.go`'s `run()` already binds its constructed `*permconfig.Resolver` to (read the surrounding ~50 lines of `run()` to find the exact name before writing this — it is used to build the engine's permission policy earlier in the same function). + +- [ ] **Step 4: Fan the new sink/recorder into the existing pipeline** + +Change the existing: + +```go + sinks := []port.EventSink{mainScoped, tracing} +``` + +to: + +```go + sinks := []port.EventSink{mainScoped, tracing} + if pm.Sink != nil { + sinks = append(sinks, pm.Sink) + } +``` + +and change the existing: + +```go + composition := appConfig(cfg, sink, mainScoped, roleScoper, obs.metrics, diag) +``` + +to: + +```go + composition := appConfig(cfg, sink, cliconfig.TeeToolCallRecorder(mainScoped, pm.ToolCallRecorder), roleScoper, obs.metrics, diag) +``` + +- [ ] **Step 5: Write `productMetricsSnapshot`** + +Add this helper (near `appConfig` or `setupObservability`): + +```go +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on cfg — never a +// model id/alias, only whether each feature is configured at all. +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + provider := productmetrics.ProviderOther + switch { + case cfg.useOpenAI: + provider = productmetrics.ProviderOpenAI + case strings.Contains(strings.ToLower(cfg.defaultProvider), "openrouter"): + provider = productmetrics.ProviderOpenRouter + case strings.Contains(strings.ToLower(cfg.defaultProvider), "openai"): + provider = productmetrics.ProviderOpenAI + case cfg.defaultProvider == "" || strings.Contains(strings.ToLower(cfg.defaultProvider), "anthropic"): + provider = productmetrics.ProviderAnthropic + } + return productmetrics.FeatureSnapshot{ + Memory: cfg.memoryDir != "", + Guardrails: cfg.guardrailsModel != "", + MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, + Scheduling: !cfg.noScheduler, + Provider: provider, + Mode: productmetrics.ModeInteractive, + } +} +``` + +Add the `"strings"` import if not already present, and `"github.com/stacklok/mecatl/internal/adapter/productmetrics"` + `"github.com/stacklok/mecatl/internal/buildinfo"` (for `buildinfo.BuildID`, already used elsewhere in this file per the `BUILD_LDFLAGS` reference) to the import block. + +- [ ] **Step 6: Build and run the existing test suite** + +Run: `task build && cd cmd/mecated && go test ./... -v` +Expected: builds and existing tests pass (this task changes wiring only, no new mecated-level tests are required beyond what already exercises flag parsing/appConfig — if `cmd/mecated` has a flag-parsing golden test, verify it still passes with the new `--product-metrics` flag appearing in its help output). + +- [ ] **Step 7: Manually verify the disabled/default posture** + +Run: `go run ./cmd/mecademo` (unaffected — mecademo doesn't wire telemetry) and `bin/mecated --help 2>&1 | grep product-metrics` to confirm the flag is registered and its help text is legible. + +- [ ] **Step 8: Commit** + +```bash +git add cmd/mecated/main.go +git commit -m "feat(mecated): wire opt-out product metrics alongside operator telemetry" +``` + +--- + +### Task 12: Wire into `cmd/mecatui` + +**Files:** +- Modify: `cmd/mecatui/embed/embed.go` (around the `wirePerfSinks` function, line ~549, and its `telemetry.Setup` call at line ~400) +- Modify: `cmd/mecatui/config.go` or `cmd/mecatui/main.go` (wherever mecatui's top-level flags are registered — read the file first to find the exact flag-registration function name, mirroring Task 11 Step 1's `--product-metrics` flag) + +**Interfaces:** +- Consumes: the same `cliconfig.BuildProductMetrics`/`ResolveProductMetricsEnabled`/`TeeToolCallRecorder` as Task 11. + +- [ ] **Step 1: Read the exact flag-registration and `wirePerfSinks` call site** + +Before writing code, read `cmd/mecatui/embed/embed.go` lines 373-560 (already excerpted above) and the file that registers mecatui's top-level CLI flags (find it via `grep -n "flag.NewFlagSet\|RegisterProviderFlags" cmd/mecatui/*.go`) to confirm the exact local variable/function names this task's diff must anchor to — mecatui's structure was not fully read during planning; it mirrors mecated's shape closely (same `sinks := []port.EventSink{mainScoped, tracing}` idiom at line 552) but names may differ slightly. + +- [ ] **Step 2: Add the `--product-metrics` flag** + +Mirror Task 11 Step 1 exactly, in whichever file registers mecatui's top-level flags. + +- [ ] **Step 3: Build and thread the handles** + +In `wirePerfSinks` (or its caller — whichever holds `cfg *app.Config` and constructs `sinks`), mirror Task 11 Steps 3-4: resolve `productMetricsEnabled`, call `cliconfig.BuildProductMetrics` with `productmetrics.BinaryMecatui` and `productmetrics.ModeInteractive`, append `pm.Sink` to `sinks`, and wrap `cfg.ToolCallRecorder` with `cliconfig.TeeToolCallRecorder`. + +- [ ] **Step 4: Print the disclosure notice** + +Wherever mecatui prints its own existing privacy warning (`docs/usage.md:282` references one — find its call site via `grep -rn "privacy warning" cmd/mecatui/`), print `cliconfig.ProductMetricsDisclosureNotice` alongside it when `pm.FirstRun` is true, so the two disclosures appear together rather than as two unrelated startup messages. + +- [ ] **Step 5: Build and run mecatui's existing tests** + +Run: `task build && cd cmd/mecatui && go test ./... -v` +Expected: builds and passes. + +- [ ] **Step 6: Commit** + +```bash +git add cmd/mecatui/ +git commit -m "feat(mecatui): wire opt-out product metrics alongside operator telemetry" +``` + +--- + +### Task 13: Wire into `cmd/mecatequi` and `cmd/mecak8s` + +**Files:** +- Modify: `cmd/mecatequi/observability.go`, `cmd/mecatequi/flags.go` +- Modify: `cmd/mecak8s/observability.go`, `cmd/mecak8s/flags.go` + +**Interfaces:** +- Consumes: `cliconfig.BuildProductMetrics`/`ResolveProductMetricsEnabled`/`TeeToolCallRecorder`. +- Produces: extends the existing `observability` struct in both files with the product-metrics handles. + +- [ ] **Step 1: Add the `--product-metrics` flag to both `flags.go` files** + +Mirror Task 11 Step 1 in each binary's flag registration (both already register `--otlp-endpoint` etc. per the `HeadlessTelemetryConfig` fields read earlier — add `--product-metrics` alongside them). + +- [ ] **Step 2: Extend `cmd/mecatequi/observability.go`** + +```go +package main + +import ( + "context" + "fmt" + "io" + "time" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// observability carries the telemetry handles realMain threads into appConfig, +// plus the flush-on-exit Shutdown the main owns. +type observability struct { + cliconfig.HeadlessTelemetryHandles + productMetrics cliconfig.ProductMetricsHandles +} + +func buildObservability(ctx context.Context, f flags, permResolver telemetryOperatorSource) (observability, error) { + h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ + ServiceName: "mecatequi", + OTLPTraceEndpoint: f.otlpEndpoint, + OTLPTraceProtocol: f.otlpProtocol, + OTLPTraceInsecure: f.otlpInsecure, + OTLPMetricsEndpoint: f.otlpMetricsEndpoint, + OTLPMetricsProtocol: f.otlpMetricsProtocol, + OTLPMetricsInsecure: f.otlpInsecure, + }) + if err != nil { + return observability{}, err + } + + enabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: f.productMetricsSet, + FlagValue: f.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, + productmetrics.BinaryMecatequi, f.version, 0, /* single fire, short-lived */ + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}) + if pmErr != nil { + // Mirror the existing telemetry-setup-failure posture: a warning, never + // a fatal error — product metrics are best-effort and must not block a + // CI run. Logged by the caller (realMain already has a diag/logger in + // scope) rather than here, to keep this function's error return + // meaningful for the OTLP half only. + pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} + } + + return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil +} + +func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { + ctx := context.Background() + if timeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, timeout) + defer cancel() + } + if obs.Shutdown != nil { + if err := obs.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry flush: %v\n", err) + } + } + if obs.productMetrics.Shutdown != nil { + if err := obs.productMetrics.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: product metrics flush: %v\n", err) + } + } + if obs.productMetrics.FirstRun { + _, _ = fmt.Fprint(stderr, cliconfig.ProductMetricsDisclosureNotice) + } +} +``` + +`telemetryOperatorSource` is a one-method interface (`OperatorProductMetricsEnabled() *bool`) — define it in this file so `observability.go` doesn't need to import `permconfig` directly; pass mecatequi's already-constructed `*permconfig.Resolver` as the argument at the call site (find it via `grep -n "buildObservability(" cmd/mecatequi/*.go` and thread the resolver variable already in scope there). + +- [ ] **Step 3: Thread the product-metrics handles into `appConfig`** + +At mecatequi's `appConfig`-equivalent assembly point (wherever `HeadlessTelemetryHandles.Sink`/`ToolCallRecorder` currently feed `app.Config.Sink`/`ToolCallRecorder`), fan in the product-metrics handles the same way as Task 11 Step 4: + +```go + sink := telemetry.NewSink(obs.Sink, obs.productMetrics.Sink) // telemetry.NewSink already skips nothing — verify it tolerates a nil element, or filter nils first + recorder := cliconfig.TeeToolCallRecorder(obs.ToolCallRecorder, obs.productMetrics.ToolCallRecorder) +``` + +`internal/adapter/telemetry.NewSink`'s `fanOut.Emit` calls every wrapped sink unconditionally — a nil `port.EventSink` element would panic on `.Emit`. Guard it: build the slice with a nil check before calling `NewSink`, exactly like Task 11 Step 4's `if pm.Sink != nil { sinks = append(...) }` pattern, rather than passing a possibly-nil element directly. + +- [ ] **Step 4: Repeat Steps 2-3 for `cmd/mecak8s/observability.go`**, using `productmetrics.BinaryMecak8s`, `productmetrics.ModeK8s`, and `productmetrics.DefaultHeartbeatInterval` (mecak8s is long-running, unlike mecatequi) with `heartbeatCtx` tied to the server's shutdown context rather than the short-lived `ctx`. + +- [ ] **Step 5: Add `productMetrics`/`productMetricsSet` fields + flag registration to both `flags.go` files** + +Mirror Task 11 Step 1 in each. + +- [ ] **Step 6: Build and run both binaries' existing tests** + +Run: `task build && cd cmd/mecatequi && go test ./... -v && cd ../mecak8s && go test ./... -v` +Expected: builds and passes. + +- [ ] **Step 7: Commit** + +```bash +git add cmd/mecatequi/ cmd/mecak8s/ +git commit -m "feat(mecatequi,mecak8s): wire opt-out product metrics via cliconfig" +``` + +--- + +### Task 14: Dry-run / audit recorder + +**Files:** +- Create: `internal/adapter/productmetrics/dryrun.go` +- Test: `internal/adapter/productmetrics/dryrun_test.go` +- Modify: `internal/cliconfig/productmetrics.go` (thread a `dryRun bool` parameter into `BuildProductMetrics`) +- Modify: all four `cmd/*` flag-registration sites from Tasks 11-13 (add `--product-metrics-dry-run`) + +**Interfaces:** +- Produces: `type DryRunRecorder struct{...}` implementing `port.EventSink` + `port.ToolCallRecorder`, logging every would-be observation via `port.Diagnostics` instead of exporting it — `func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "context" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +type capturingDiag struct { + lines []string +} + +func (c *capturingDiag) Log(_ context.Context, _ port.Level, msg string, args ...any) { + c.lines = append(c.lines, msg) + _ = args +} +func (c *capturingDiag) With(...any) port.Diagnostics { return c } + +func TestDryRunRecorderLogsInsteadOfExporting(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "sensitive-name"}, session.ToolResult{Content: "sensitive-content"}, 0, time.Millisecond) + + if len(diag.lines) != 2 { + t.Fatalf("got %d logged lines, want 2: %v", len(diag.lines), diag.lines) + } + for _, line := range diag.lines { + if contains(line, "sensitive") { + t.Errorf("dry-run log line leaked sensitive content: %q", line) + } + } +} + +func contains(s, substr string) bool { + return len(s) >= len(substr) && (func() bool { + for i := 0; i+len(substr) <= len(s); i++ { + if s[i:i+len(substr)] == substr { + return true + } + } + return false + })() +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestDryRunRecorderLogsInsteadOfExporting -v` +Expected: FAIL — `NewDryRunRecorder` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// DryRunRecorder implements the same two ports as Recorder but logs every +// would-be observation via port.Diagnostics instead of exporting it over +// OTLP — the --product-metrics-dry-run audit path, so a skeptical operator +// can see exactly what this pipeline would have sent without trusting the +// docs. It logs ONLY the same bounded fields Recorder ever reads (event +// type, stop reason, token kind/amounts) — never a tool name, session id, +// or result content, mirroring Recorder's own restraint exactly. +type DryRunRecorder struct { + diag port.Diagnostics +} + +var ( + _ port.EventSink = (*DryRunRecorder)(nil) + _ port.ToolCallRecorder = (*DryRunRecorder)(nil) +) + +// NewDryRunRecorder builds a DryRunRecorder over the given Diagnostics sink. +func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder { + return &DryRunRecorder{diag: diag} +} + +// Emit logs the bounded event type (and, for EvResult, the stop reason and +// token counts by kind) — the exact same fields Recorder.Emit reads. +func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") + case session.EvResult: + if ev.Result == nil { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed{stop=\"\"}+1") + return + } + u := ev.Result.Usage + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record run + tokens", + "stop", string(ev.Result.Stop), + "input_tokens", u.InputTokens, "output_tokens", u.OutputTokens, + "cache_read_tokens", u.CacheReadTokens, "cache_write_tokens", u.CacheWriteTokens, + "reasoning_tokens", u.ReasoningTokens) + case session.EvSubagentStart: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") + case session.EvTeamStart: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record team_used+1") + } +} + +// ToolCall logs only that a call happened — no name, no content, matching +// Recorder.ToolCall's restraint exactly. +func (d *DryRunRecorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1") +} + +// Heartbeat logs the closed-enum feature/provider/mode signal, matching +// Recorder.Heartbeat's fields exactly. +func (d *DryRunRecorder) Heartbeat(snap FeatureSnapshot) { + enabled := make([]string, 0, 4) + for f, on := range snap.enabled() { + if on { + enabled = append(enabled, string(f)) + } + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record heartbeat", + "features_enabled", enabled, "provider_family", string(snap.Provider), "deployment_mode", string(snap.Mode)) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestDryRunRecorderLogsInsteadOfExporting -v` +Expected: PASS + +- [ ] **Step 5: Thread a `dryRun` option through `BuildProductMetrics`** + +In `internal/cliconfig/productmetrics.go`, change `BuildProductMetrics`'s signature to accept a `dryRun bool` parameter (after `enabled`), and branch near the top of the function body: + +```go +func BuildProductMetrics( + ctx, heartbeatCtx context.Context, + enabled, dryRun bool, + binary productmetrics.Binary, + version string, + heartbeatInterval time.Duration, + snap productmetrics.FeatureSnapshot, + diag port.Diagnostics, +) (ProductMetricsHandles, error) { + noop := func(context.Context) error { return nil } + if !enabled { + return ProductMetricsHandles{Shutdown: noop}, nil + } + if dryRun { + rec := productmetrics.NewDryRunRecorder(diag) + go func() { + rec.Heartbeat(snap) + // A dry run never persists an install id or starts a real ticker — + // it exists to show ONE representative sample, not to simulate the + // full 24h cadence. + }() + return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil + } + // ... existing enabled, non-dry-run body unchanged below this point ... +``` + +Update the two callers from Task 10's test file and every `cmd/*` call site from Tasks 11-13 to pass `false` for `dryRun` (or the resolved flag value) and a `port.Diagnostics` (each `cmd/*/main.go` already constructs one — thread the existing `diag` variable through). + +- [ ] **Step 6: Add `--product-metrics-dry-run` to all four binaries' flags** + +Mirror Task 11 Step 1's flag pattern in each of the four flag-registration sites touched in Tasks 11-13: + +```go + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") +``` + +- [ ] **Step 7: Run the full `productmetrics` and `cliconfig` suites** + +Run: `cd internal/adapter/productmetrics && go test ./... -v && cd ../../cliconfig && go test ./... -v` +Expected: PASS + +- [ ] **Step 8: Run `task build` to confirm every binary still compiles with the new parameter threaded through** + +Run: `task build` +Expected: succeeds. + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/dryrun.go internal/adapter/productmetrics/dryrun_test.go \ + internal/cliconfig/productmetrics.go cmd/mecated/main.go cmd/mecatui/ cmd/mecatequi/ cmd/mecak8s/ +git commit -m "feat(productmetrics): --product-metrics-dry-run audit mode across all four binaries" +``` + +--- + +### Task 15: ADR + user-docs + +**Files:** +- Create: `docs/adr/0317-product-metrics.md` (confirm 0317 is still the next free number by running `ls docs/adr | grep -oE '^[0-9]+' | sort -n | tail -1` immediately before creating the file — another PR may have landed a higher number since this plan was written) +- Create: `user-docs/building/what-you-get/product-metrics.md` (confirm the exact directory naming convention by listing `user-docs/building/what-you-get/` first — follow its existing file-naming/frontmatter pattern exactly) +- Modify: `internal/cliconfig/productmetrics.go` (fill in `ProductMetricsDisclosureNotice`'s doc link placeholder with the real path) + +**Interfaces:** none — documentation only. + +- [ ] **Step 1: Confirm the next free ADR number** + +Run: `ls docs/adr | grep -oE '^[0-9]+' | sort -n | tail -1` +Use ` + 1` as the filename prefix (0317 as of this plan's writing). + +- [ ] **Step 2: Write the ADR** + +Create `docs/adr/0317-product-metrics.md` (or whatever number Step 1 resolved) following `docs/adr/template.md`'s structure (header with `- Status: Accepted` / `- Date:` / `- Scope:` / `- Supersedes: —` / `- Superseded by: —`, then Context/Decision/Consequences/See also, mirroring ADR 0098's structure read during planning). Content: summarize this plan's design — the fully independent adapter, the exact `mecatl.adoption.*` catalog (Tasks 4-5), the opt-out precedence (Task 9), the operator-tier-only settings gate (Tasks 7-8), the privacy guard test (Task 6), the dry-run audit mode (Task 14), and the "why opt-out is defensible here" rationale (disclosure notice + `DO_NOT_TRACK` + a reviewable, tested catalog). Cross-reference ADR 0098 (headless telemetry) and ADR 0020 (diagnostics/the three-channel table this is a deliberate fourth, separate channel from) as prior art it deliberately does NOT reuse. + +- [ ] **Step 3: Write the user-docs page** + +List `user-docs/building/what-you-get/` first to match its existing frontmatter/heading conventions, then add a short page: what's collected (link the ADR's catalog), the opt-out mechanisms (flag, `DO_NOT_TRACK`, settings.yaml), and how to self-verify via `--product-metrics-dry-run`. + +- [ ] **Step 4: Fill in the disclosure notice's doc link** + +In `internal/cliconfig/productmetrics.go`, replace `` in `ProductMetricsDisclosureNotice` with the real path/URL to the page written in Step 3. + +- [ ] **Step 5: Run the docs gates** + +Run: `task docs` (regenerates the configuration reference + runs the strict link gate) and `task site:build` (Docusaurus build, catches a broken link before CI does). +Expected: both succeed with no broken links/anchors. + +- [ ] **Step 6: Run `task lint && task test` one final time across the whole feature** + +Run: `task lint && task test` +Expected: green. + +- [ ] **Step 7: Commit** + +```bash +git add docs/adr/ user-docs/building/what-you-get/ internal/cliconfig/productmetrics.go +git commit -m "docs: add ADR and user-docs for opt-out product/adoption metrics" +``` From dbcfe9a8e8b7cb678c0bb49662df7dc2f4142cfb Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 22:25:54 -0400 Subject: [PATCH 03/47] feat(productmetrics): add closed enums, FeatureSnapshot, Config Task 1 of the product-metrics-otel plan: create the internal/adapter/ productmetrics package skeleton with closed-set enums (Feature, ProviderFamily, DeploymentMode, Binary), the FeatureSnapshot value object + its enabled() method, and Config. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/config.go | 88 +++++++++++++++++++ .../adapter/productmetrics/config_test.go | 23 +++++ 2 files changed, 111 insertions(+) create mode 100644 internal/adapter/productmetrics/config.go create mode 100644 internal/adapter/productmetrics/config_test.go diff --git a/internal/adapter/productmetrics/config.go b/internal/adapter/productmetrics/config.go new file mode 100644 index 0000000000..f3d425c622 --- /dev/null +++ b/internal/adapter/productmetrics/config.go @@ -0,0 +1,88 @@ +// Package productmetrics is a fully independent, opt-out-by-default OTel +// metrics adapter reporting bounded adoption/usage counters to Stacklok's +// public metrics collector. It shares no import, struct, MeterProvider, or +// destination with internal/adapter/telemetry (mecatl's operator-facing +// observability pipeline) — the two are combined only at the composition +// edge (internal/cliconfig), by fanning both into the engine's +// port.EventSink/port.ToolCallRecorder seams. +// +// Every exported type in this package that can become a metric attribute is +// a closed Go string-alias enum. Nothing here carries a session id, model +// id/alias, tool or MCP-server name, file path, or free text. +package productmetrics + +// Feature is the closed set of major toggleable features reported at +// heartbeat time. Never a def/model/tool name — only these four values. +type Feature string + +const ( + FeatureMemory Feature = "memory" + FeatureGuardrails Feature = "guardrails" + FeatureMCP Feature = "mcp" + FeatureScheduling Feature = "scheduling" +) + +// ProviderFamily is the closed set of configured LLM provider families. +// Never a model id or alias. +type ProviderFamily string + +const ( + ProviderAnthropic ProviderFamily = "anthropic" + ProviderOpenAI ProviderFamily = "openai" + ProviderOpenRouter ProviderFamily = "openrouter" + ProviderOther ProviderFamily = "other" +) + +// DeploymentMode is the closed set of process shapes. +type DeploymentMode string + +const ( + ModeInteractive DeploymentMode = "interactive" + ModeHeadless DeploymentMode = "headless" + ModeK8s DeploymentMode = "k8s" +) + +// Binary is the closed set of the four mecatl entry points. +type Binary string + +const ( + BinaryMecated Binary = "mecated" + BinaryMecatui Binary = "mecatui" + BinaryMecatequi Binary = "mecatequi" + BinaryMecak8s Binary = "mecak8s" +) + +// FeatureSnapshot is a closed-shape, read-only snapshot of which major +// features are enabled and which provider family / deployment mode this +// process runs as. It carries no free text and no model id/alias. +type FeatureSnapshot struct { + Memory bool + Guardrails bool + MCP bool + Scheduling bool + Provider ProviderFamily + Mode DeploymentMode +} + +// enabled returns every Feature mapped to whether this snapshot reports it +// enabled. It is the single place Heartbeat iterates, so adding a Feature +// const without adding it here is caught by the exhaustiveness this map +// documents (and by TestFeatureSnapshotEnabledIsClosedAndBounded above). +func (s FeatureSnapshot) enabled() map[Feature]bool { + return map[Feature]bool{ + FeatureMemory: s.Memory, + FeatureGuardrails: s.Guardrails, + FeatureMCP: s.MCP, + FeatureScheduling: s.Scheduling, + } +} + +// Config configures a Provider/Recorder pair for one process. +type Config struct { + // Binary identifies which of the four entry points this process is. + Binary Binary + // Version is the mecatl build version (resource attribute service.version). + Version string + // InstallID is this process's persisted anonymous install identifier. + InstallID string +} diff --git a/internal/adapter/productmetrics/config_test.go b/internal/adapter/productmetrics/config_test.go new file mode 100644 index 0000000000..1107c03a75 --- /dev/null +++ b/internal/adapter/productmetrics/config_test.go @@ -0,0 +1,23 @@ +package productmetrics + +import "testing" + +func TestFeatureSnapshotEnabledIsClosedAndBounded(t *testing.T) { + snap := FeatureSnapshot{Memory: true, MCP: true, Provider: ProviderAnthropic, Mode: ModeInteractive} + got := snap.enabled() + + want := map[Feature]bool{ + FeatureMemory: true, + FeatureGuardrails: false, + FeatureMCP: true, + FeatureScheduling: false, + } + if len(got) != len(want) { + t.Fatalf("enabled() returned %d entries, want %d (%v)", len(got), len(want), got) + } + for f, v := range want { + if got[f] != v { + t.Errorf("enabled()[%q] = %v, want %v", f, got[f], v) + } + } +} From fdd5d604d290db99a19f8fb39fb4e7068f896a1c Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 22:29:59 -0400 Subject: [PATCH 04/47] feat(productmetrics): persist an anonymous random install id Adds LoadOrCreateInstallID (+ LoadOrCreateInstallIDDefault), reading/ writing a bare v4 UUID under $XDG_STATE_HOME/mecatl/telemetry-id via xdgconfig.UserStateDir, regenerating on a missing/corrupt file and failing closed when no state dir can be resolved. Promotes github.com/google/uuid from indirect to direct in go.mod (task tidy). Co-Authored-By: Claude Sonnet 5 --- go.mod | 2 +- internal/adapter/productmetrics/installid.go | 68 ++++++++++++++ .../adapter/productmetrics/installid_test.go | 90 +++++++++++++++++++ 3 files changed, 159 insertions(+), 1 deletion(-) create mode 100644 internal/adapter/productmetrics/installid.go create mode 100644 internal/adapter/productmetrics/installid_test.go diff --git a/go.mod b/go.mod index e2b167d82d..34aabb13bf 100644 --- a/go.mod +++ b/go.mod @@ -193,7 +193,7 @@ require ( github.com/google/gnostic-models v0.7.1 // indirect github.com/google/go-cmp v0.7.0 // indirect github.com/google/go-containerregistry v0.22.0 // indirect - github.com/google/uuid v1.6.0 // indirect + github.com/google/uuid v1.6.0 github.com/gorilla/css v1.0.1 // indirect github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect diff --git a/internal/adapter/productmetrics/installid.go b/internal/adapter/productmetrics/installid.go new file mode 100644 index 0000000000..863be18884 --- /dev/null +++ b/internal/adapter/productmetrics/installid.go @@ -0,0 +1,68 @@ +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// installIDRelPath is the state-dir-relative path to the persisted anonymous +// install identifier — machine-written runtime state, not human config, so +// it lives under XDG_STATE_HOME (mirroring mecatui's +// $XDG_STATE_HOME/mecatl/mecatui.log precedent), not XDG_CONFIG_HOME. +const installIDRelPath = "mecatl/telemetry-id" + +// LoadOrCreateInstallID reads the persisted install UUID, creating one if +// absent or unparseable. The id is a bare random v4 UUID: it carries no +// machine or user information, and is trivially reset by deleting the file +// (the next opt-in mints a new one). firstRun is true whenever a new id was +// just minted — the caller uses it to decide whether to print the one-time +// disclosure notice. readFile/writeFile/mkdirAll are injected for testing; +// LoadOrCreateInstallIDDefault binds the real filesystem. +func LoadOrCreateInstallID( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (id string, firstRun bool, err error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return "", false, fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + path := filepath.Join(base, installIDRelPath) + + if readFile != nil { + if data, rerr := readFile(path); rerr == nil { + if existing := strings.TrimSpace(string(data)); existing != "" { + if _, perr := uuid.Parse(existing); perr == nil { + return existing, false, nil + } + // Corrupt file: fall through and regenerate. + } + } + } + + fresh := uuid.NewString() + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return "", false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte(fresh), 0o600); werr != nil { + return "", false, fmt.Errorf("productmetrics: write install id: %w", werr) + } + } + return fresh, true, nil +} + +// LoadOrCreateInstallIDDefault binds LoadOrCreateInstallID to the real +// process environment and filesystem. +func LoadOrCreateInstallIDDefault() (id string, firstRun bool, err error) { + return LoadOrCreateInstallID(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} diff --git a/internal/adapter/productmetrics/installid_test.go b/internal/adapter/productmetrics/installid_test.go new file mode 100644 index 0000000000..2adc0c94b6 --- /dev/null +++ b/internal/adapter/productmetrics/installid_test.go @@ -0,0 +1,90 @@ +package productmetrics + +import ( + "errors" + "os" + "testing" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func TestLoadOrCreateInstallIDCreatesOnFirstRun(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + written := map[string][]byte{} + readFile := func(path string) ([]byte, error) { + data, ok := written[path] + if !ok { + return nil, os.ErrNotExist + } + return data, nil + } + writeFile := func(path string, data []byte, _ os.FileMode) error { + written[path] = data + return nil + } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on an empty store, want true") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + + // Second call reads back the SAME id and reports firstRun=false. + id2, firstRun2, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateInstallID: %v", err) + } + if firstRun2 { + t.Error("firstRun = true on second call, want false") + } + if id2 != id { + t.Errorf("second call returned id %q, want %q (unchanged)", id2, id) + } +} + +func TestLoadOrCreateInstallIDRegeneratesOnCorruptFile(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + readFile := func(string) ([]byte, error) { return []byte("not-a-uuid"), nil } + var gotWrite []byte + writeFile := func(_ string, data []byte, _ os.FileMode) error { gotWrite = data; return nil } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on a corrupt file, want true (treated as absent)") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + if string(gotWrite) != id { + t.Errorf("written content %q != returned id %q", gotWrite, id) + } +} + +func TestLoadOrCreateInstallIDFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + _, _, err := LoadOrCreateInstallID(env, nil, nil, nil) + if err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } +} From 54caec5fdaa908b30a7fcb6edb0f8d13ebebaa0a Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 22:37:08 -0400 Subject: [PATCH 05/47] fix(productmetrics): add missing doc comments on exported enum blocks (revive) Adds a one-line doc comment above each of the four const blocks (Feature, ProviderFamily, DeploymentMode, Binary) in config.go to satisfy revive's exported-const-needs-comment rule. No names or values changed. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/config.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/internal/adapter/productmetrics/config.go b/internal/adapter/productmetrics/config.go index f3d425c622..40506f3af1 100644 --- a/internal/adapter/productmetrics/config.go +++ b/internal/adapter/productmetrics/config.go @@ -15,6 +15,7 @@ package productmetrics // heartbeat time. Never a def/model/tool name — only these four values. type Feature string +// The four closed Feature values. const ( FeatureMemory Feature = "memory" FeatureGuardrails Feature = "guardrails" @@ -26,6 +27,7 @@ const ( // Never a model id or alias. type ProviderFamily string +// The four closed ProviderFamily values. const ( ProviderAnthropic ProviderFamily = "anthropic" ProviderOpenAI ProviderFamily = "openai" @@ -36,6 +38,7 @@ const ( // DeploymentMode is the closed set of process shapes. type DeploymentMode string +// The three closed DeploymentMode values. const ( ModeInteractive DeploymentMode = "interactive" ModeHeadless DeploymentMode = "headless" @@ -45,6 +48,7 @@ const ( // Binary is the closed set of the four mecatl entry points. type Binary string +// The four closed Binary values. const ( BinaryMecated Binary = "mecated" BinaryMecatui Binary = "mecatui" From e216e4682b1c594620a789fa9ad750b1933e08c1 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 22:50:20 -0400 Subject: [PATCH 06/47] feat(productmetrics): build the OTLP provider via toolhive-core, baked-key gated Provider/NewProvider wraps toolhive-core's CompositeProvider to export mecatl adoption metrics to the fixed metrics.stacklok.com OTLP/HTTP ingest, gated by a build-time-baked ingest key (BUILD_LDFLAGS in Taskfile.yml). An empty baked key (every local/dev/CI build) disables the pipeline entirely, so NewProvider always errors. Also wires providers.WithInsecure off the endpoint's scheme, since toolhive-core strips the scheme and defaults to TLS regardless -- required for the package's own httptest-backed test to actually reach a plaintext listener. Co-Authored-By: Claude Sonnet 5 --- Taskfile.yml | 3 +- internal/adapter/productmetrics/provider.go | 77 +++++++++++++++++++ .../adapter/productmetrics/provider_test.go | 59 ++++++++++++++ 3 files changed, 138 insertions(+), 1 deletion(-) create mode 100644 internal/adapter/productmetrics/provider.go create mode 100644 internal/adapter/productmetrics/provider_test.go diff --git a/Taskfile.yml b/Taskfile.yml index 4629c023ea..37f09208b8 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -43,7 +43,8 @@ vars: else git describe --tags --match 'v[0-9]*' --always --dirty 2>/dev/null || printf %s dev fi - BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}}' + MECATL_METRICS_KEY: '{{.MECATL_METRICS_KEY | default ""}}' + BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}} -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.MECATL_METRICS_KEY}}' tasks: default: diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go new file mode 100644 index 0000000000..fdb8471fef --- /dev/null +++ b/internal/adapter/productmetrics/provider.go @@ -0,0 +1,77 @@ +package productmetrics + +import ( + "context" + "fmt" + "strings" + + "github.com/stacklok/toolhive-core/telemetry/providers" + "go.opentelemetry.io/otel/metric" +) + +// endpoint and headerKeyName are the ONE destination this pipeline can ever +// send to (stacklok/infra#5604): a dedicated, internet-facing OTLP/HTTP +// ingest at metrics.stacklok.com, gated by a single shared key baked into +// the binary. Neither is operator-configurable — an operator's own +// --otlp-endpoint has zero effect on this path, and this path has zero +// effect on the operator's own OTLP/Prometheus pipeline (a completely +// separate MeterProvider, never installed as global). endpoint is a var +// (not a const) so tests can point it at an httptest server. +var ( + endpoint = "https://metrics.stacklok.com/v1/metrics" + headerKeyName = "x-mecatl-metrics-key" +) + +// bakedKey is the shared ingest key baked into the binary at build time via +// `-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey=…` +// (see Taskfile.yml's BUILD_LDFLAGS). An empty key — every local/dev/CI-test +// build that does not set the ldflag — disables the pipeline entirely: +// NewProvider refuses to construct, so a non-release build can never +// accidentally phone home with an invalid or absent key. +var bakedKey = "" + +// Provider wraps the toolhive-core OTLP metrics provider. Its MeterProvider +// is NEVER installed as the process-global provider (mirrors +// internal/adapter/telemetry's own discipline in otlp.go), so it cannot +// collide with an operator's own OTel setup. +type Provider struct { + composite *providers.CompositeProvider +} + +// NewProvider builds the product-metrics MeterProvider for one process. A +// network-unreachable endpoint is NOT an error here — the OTLP/HTTP +// exporter dials lazily on first export, matching the existing exporters in +// internal/adapter/telemetry/otlp.go. +func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { + if bakedKey == "" { + return nil, fmt.Errorf("productmetrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") + } + // toolhive-core's OTLP metric exporter strips the scheme from the + // endpoint and defaults to a secure (TLS) connection regardless — so a + // plain http:// endpoint (only ever true in this package's own test, + // pointed at an httptest.Server) must explicitly opt into WithInsecure, + // or the exporter tries TLS against a plaintext listener and every + // export fails. The real production endpoint is always https://. + composite, err := providers.NewCompositeProvider(ctx, + providers.WithServiceName("mecatl"), + providers.WithServiceVersion(cfg.Version), + providers.WithOTLPEndpoint(endpoint), + providers.WithMetricsEnabled(true), + providers.WithInsecure(strings.HasPrefix(endpoint, "http://")), + providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), + providers.WithCustomAttributes(map[string]string{ + "mecatl.install.id": cfg.InstallID, + "mecatl.binary": string(cfg.Binary), + }), + ) + if err != nil { + return nil, fmt.Errorf("productmetrics: build provider: %w", err) + } + return &Provider{composite: composite}, nil +} + +// Meter returns the underlying metric.MeterProvider for instrument construction. +func (p *Provider) Meter() metric.MeterProvider { return p.composite.MeterProvider() } + +// Shutdown flushes and stops the provider, bounded by the caller's ctx. +func (p *Provider) Shutdown(ctx context.Context) error { return p.composite.Shutdown(ctx) } diff --git a/internal/adapter/productmetrics/provider_test.go b/internal/adapter/productmetrics/provider_test.go new file mode 100644 index 0000000000..71777b2abd --- /dev/null +++ b/internal/adapter/productmetrics/provider_test.go @@ -0,0 +1,59 @@ +package productmetrics + +import ( + "context" + "net/http" + "net/http/httptest" + "testing" +) + +func TestNewProviderFailsClosedWithNoBakedKey(t *testing.T) { + orig := bakedKey + bakedKey = "" + defer func() { bakedKey = orig }() + + _, err := NewProvider(context.Background(), Config{Binary: BinaryMecated, Version: "test"}) + if err == nil { + t.Fatal("expected an error when no ingest key is baked into the build, got nil") + } +} + +func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { + origKey := bakedKey + bakedKey = "test-key" + defer func() { bakedKey = origKey }() + + var gotHeader string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotHeader = r.Header.Get(headerKeyName) + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + origEndpoint := endpoint + endpoint = srv.URL + defer func() { endpoint = origEndpoint }() + + p, err := NewProvider(context.Background(), Config{ + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", + }) + if err != nil { + t.Fatalf("NewProvider: %v", err) + } + defer p.Shutdown(context.Background()) + + meter := p.Meter().Meter("test") + counter, cerr := meter.Int64Counter("mecatl.adoption.test") + if cerr != nil { + t.Fatalf("Int64Counter: %v", cerr) + } + counter.Add(context.Background(), 1) + if err := p.Shutdown(context.Background()); err != nil { + t.Fatalf("Shutdown: %v", err) + } + if gotHeader != "test-key" { + t.Errorf("collector received %s=%q, want %q", headerKeyName, gotHeader, "test-key") + } +} From 6feb5982362f37083a9efdb42c58f09632f179f6 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 22:55:50 -0400 Subject: [PATCH 07/47] feat(productmetrics): Recorder.Emit derives bounded session/run/token counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recorder implements port.EventSink, constructing all ten adoption instruments up front (heartbeat/feature/provider/mode counters are wired for a later task's Heartbeat method; tool_calls for its ToolCall method). Emit reads only ev.Type, ev.Result.Stop, and ev.Result.Usage — never ev.Result.Text or ev.Result.Error. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/metrics.go | 131 +++++++++++++++++ .../adapter/productmetrics/metrics_test.go | 139 ++++++++++++++++++ 2 files changed, 270 insertions(+) create mode 100644 internal/adapter/productmetrics/metrics.go create mode 100644 internal/adapter/productmetrics/metrics_test.go diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go new file mode 100644 index 0000000000..eb416bb1ae --- /dev/null +++ b/internal/adapter/productmetrics/metrics.go @@ -0,0 +1,131 @@ +package productmetrics + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// meterName is the instrumentation scope name for this package's meter. +const meterName = "github.com/stacklok/mecatl/internal/adapter/productmetrics" + +// Attribute keys. Every value ever attached under these keys is drawn from a +// bounded closed set (session.StopReason, the fixed token-kind strings, or +// this package's own Feature/ProviderFamily/DeploymentMode enums) — never a +// session id, model id, tool name, or free text. +const ( + attrStop = "stop" + attrKind = "kind" + attrFeature = "feature" + attrProvider = "family" + attrMode = "mode" +) + +// Recorder is the product-metrics adapter: it implements port.EventSink +// (this file) and will implement port.ToolCallRecorder (a later task's +// toolcall.go), deriving ONLY the bounded counts in the design's catalog. It +// never reads a tool name, session id, model id, or any free-text field. +type Recorder struct { + heartbeat metric.Int64Counter + featureEnabled metric.Int64Counter + providerConfig metric.Int64Counter + deploymentMode metric.Int64Counter + sessionsStarted metric.Int64Counter + runsCompleted metric.Int64Counter + toolCalls metric.Int64Counter + tokens metric.Int64Counter + subagentUsed metric.Int64Counter + teamUsed metric.Int64Counter +} + +// Compile-time interface check. port.ToolCallRecorder is satisfied once a +// later task adds Recorder.ToolCall; asserting it here would fail to build. +var _ port.EventSink = (*Recorder)(nil) + +// NewRecorder constructs every instrument from the given MeterProvider. It +// returns an error if any instrument fails to construct — the OTel meter API +// is fallible. +func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { + meter := mp.Meter(meterName) + r := &Recorder{} + var err error + + if r.heartbeat, err = meter.Int64Counter("mecatl.adoption.heartbeat", + metric.WithDescription("Process liveness heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: heartbeat counter: %w", err) + } + if r.featureEnabled, err = meter.Int64Counter("mecatl.adoption.feature_enabled", + metric.WithDescription("Major feature enabled, by closed feature name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: feature_enabled counter: %w", err) + } + if r.providerConfig, err = meter.Int64Counter("mecatl.adoption.provider_configured", + metric.WithDescription("Configured LLM provider family, by closed family name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: provider_configured counter: %w", err) + } + if r.deploymentMode, err = meter.Int64Counter("mecatl.adoption.deployment_mode", + metric.WithDescription("Process deployment mode, by closed mode name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: deployment_mode counter: %w", err) + } + if r.sessionsStarted, err = meter.Int64Counter("mecatl.adoption.sessions_started", + metric.WithDescription("Total sessions started.")); err != nil { + return nil, fmt.Errorf("productmetrics: sessions_started counter: %w", err) + } + if r.runsCompleted, err = meter.Int64Counter("mecatl.adoption.runs_completed", + metric.WithDescription("Total runs completed, by bounded stop reason.")); err != nil { + return nil, fmt.Errorf("productmetrics: runs_completed counter: %w", err) + } + if r.toolCalls, err = meter.Int64Counter("mecatl.adoption.tool_calls", + metric.WithDescription("Total tool calls executed (no tool identity attached).")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls counter: %w", err) + } + if r.tokens, err = meter.Int64Counter("mecatl.adoption.tokens", + metric.WithDescription("Total tokens accounted, by bounded kind."), + metric.WithUnit("{token}")); err != nil { + return nil, fmt.Errorf("productmetrics: tokens counter: %w", err) + } + if r.subagentUsed, err = meter.Int64Counter("mecatl.adoption.subagent_used", + metric.WithDescription("Runs that used the Subagent delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: subagent_used counter: %w", err) + } + if r.teamUsed, err = meter.Int64Counter("mecatl.adoption.team_used", + metric.WithDescription("Runs that used the Team delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: team_used counter: %w", err) + } + return r, nil +} + +// Emit derives coarse, bounded counts from a single domain Event. It reads +// ONLY ev.Type, ev.Result.Stop, and ev.Result.Usage — never a session id, +// model id/alias, tool name, or any free-text field (ev.Result.Text/Error +// are never touched). +func (r *Recorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + r.sessionsStarted.Add(ctx, 1) + case session.EvResult: + r.recordResult(ctx, ev.Result) + case session.EvSubagentStart: + r.subagentUsed.Add(ctx, 1) + case session.EvTeamStart: + r.teamUsed.Add(ctx, 1) + } +} + +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload) { + if res == nil { + r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(session.StopNone)))) + return + } + r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(res.Stop)))) + u := res.Usage + r.tokens.Add(ctx, int64(u.InputTokens), metric.WithAttributes(attribute.String(attrKind, "input"))) + r.tokens.Add(ctx, int64(u.OutputTokens), metric.WithAttributes(attribute.String(attrKind, "output"))) + r.tokens.Add(ctx, int64(u.CacheReadTokens), metric.WithAttributes(attribute.String(attrKind, "cache_read"))) + r.tokens.Add(ctx, int64(u.CacheWriteTokens), metric.WithAttributes(attribute.String(attrKind, "cache_write"))) + r.tokens.Add(ctx, int64(u.ReasoningTokens), metric.WithAttributes(attribute.String(attrKind, "reasoning"))) +} diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go new file mode 100644 index 0000000000..e32320f245 --- /dev/null +++ b/internal/adapter/productmetrics/metrics_test.go @@ -0,0 +1,139 @@ +package productmetrics + +import ( + "context" + "testing" + + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +func newTestRecorder(t *testing.T) (*Recorder, *sdkmetric.ManualReader) { + t.Helper() + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + r, err := NewRecorder(mp) + if err != nil { + t.Fatalf("NewRecorder: %v", err) + } + return r, reader +} + +func collect(t *testing.T, reader *sdkmetric.ManualReader) map[string]metricdata.Aggregation { + t.Helper() + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + out := make(map[string]metricdata.Aggregation) + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + out[md.Name] = md.Data + } + } + return out +} + +func sumValue(t *testing.T, agg metricdata.Aggregation) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + var total int64 + for _, dp := range sum.DataPoints { + total += dp.Value + } + return total +} + +func sumPoint(t *testing.T, agg metricdata.Aggregation, key, value string) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + for _, dp := range sum.DataPoints { + if v, present := dp.Attributes.Value(attribute.Key(key)); present && v.AsString() == value { + return dp.Value + } + } + t.Fatalf("no data point with %s=%q", key, value) + return 0 +} + +func TestRecorderEmitSessionsStarted(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + + agg, ok := collect(t, reader)["mecatl.adoption.sessions_started"] + if !ok { + t.Fatal("mecatl.adoption.sessions_started missing") + } + if got := sumValue(t, agg); got != 2 { + t.Errorf("sessions_started = %d, want 2", got) + } +} + +func TestRecorderEmitRunsCompletedByStopReason(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.adoption.runs_completed"] + if got := sumPoint(t, agg, "stop", "end_turn"); got != 1 { + t.Errorf("runs_completed{stop=end_turn} = %d, want 1", got) + } + if got := sumPoint(t, agg, "stop", "error"); got != 1 { + t.Errorf("runs_completed{stop=error} = %d, want 1", got) + } +} + +func TestRecorderEmitTokensByKind(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopEndTurn, + Usage: session.Usage{ + InputTokens: 100, + OutputTokens: 50, + CacheReadTokens: 20, + CacheWriteTokens: 5, + ReasoningTokens: 10, + }, + }, + }) + + agg := collect(t, reader)["mecatl.adoption.tokens"] + cases := map[string]int64{"input": 100, "output": 50, "cache_read": 20, "cache_write": 5, "reasoning": 10} + for kind, want := range cases { + if got := sumPoint(t, agg, "kind", kind); got != want { + t.Errorf("tokens{kind=%s} = %d, want %d", kind, got, want) + } + } +} + +func TestRecorderEmitSubagentAndTeamUsed(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.adoption.subagent_used"]); got != 1 { + t.Errorf("subagent_used = %d, want 1", got) + } + if got := sumValue(t, collected["mecatl.adoption.team_used"]); got != 1 { + t.Errorf("team_used = %d, want 1", got) + } +} From b6cb26c9324710b55d0846a8787594b1ac08a18b Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 22:59:54 -0400 Subject: [PATCH 08/47] feat(productmetrics): ToolCall counting and the heartbeat ticker Adds Recorder.ToolCall (port.ToolCallRecorder) and Recorder.Heartbeat + RunHeartbeat, restoring the port.ToolCallRecorder compile-time assertion in metrics.go now that ToolCall exists. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/heartbeat.go | 50 +++++++++++++++++++ .../adapter/productmetrics/heartbeat_test.go | 48 ++++++++++++++++++ internal/adapter/productmetrics/metrics.go | 14 +++--- internal/adapter/productmetrics/toolcall.go | 17 +++++++ .../adapter/productmetrics/toolcall_test.go | 24 +++++++++ 5 files changed, 147 insertions(+), 6 deletions(-) create mode 100644 internal/adapter/productmetrics/heartbeat.go create mode 100644 internal/adapter/productmetrics/heartbeat_test.go create mode 100644 internal/adapter/productmetrics/toolcall.go create mode 100644 internal/adapter/productmetrics/toolcall_test.go diff --git a/internal/adapter/productmetrics/heartbeat.go b/internal/adapter/productmetrics/heartbeat.go new file mode 100644 index 0000000000..40809d82d8 --- /dev/null +++ b/internal/adapter/productmetrics/heartbeat.go @@ -0,0 +1,50 @@ +package productmetrics + +import ( + "context" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// DefaultHeartbeatInterval is the steady-state heartbeat cadence for +// long-running processes (mecated, mecatui, mecak8s). mecatequi (short-lived) +// passes 0 — a single immediate fire only, no ticker. +const DefaultHeartbeatInterval = 24 * time.Hour + +// Heartbeat records the periodic liveness + feature/provider/mode signal. +// Every attribute value comes from the closed Feature/ProviderFamily/ +// DeploymentMode enums — never a def/model/tool name. +func (r *Recorder) Heartbeat(snap FeatureSnapshot) { + ctx := context.Background() + r.heartbeat.Add(ctx, 1) + for feature, on := range snap.enabled() { + if on { + r.featureEnabled.Add(ctx, 1, metric.WithAttributes(attribute.String(attrFeature, string(feature)))) + } + } + r.providerConfig.Add(ctx, 1, metric.WithAttributes(attribute.String(attrProvider, string(snap.Provider)))) + r.deploymentMode.Add(ctx, 1, metric.WithAttributes(attribute.String(attrMode, string(snap.Mode)))) +} + +// RunHeartbeat fires one heartbeat immediately, then one every interval, +// until ctx is done. interval<=0 disables the ticker (a single fire only — +// mecatequi's shape). Meant to run in its own goroutine, owned by the +// caller (composition), which cancels ctx on shutdown. +func RunHeartbeat(ctx context.Context, r *Recorder, interval time.Duration, snap FeatureSnapshot) { + r.Heartbeat(snap) + if interval <= 0 { + return + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + r.Heartbeat(snap) + } + } +} diff --git a/internal/adapter/productmetrics/heartbeat_test.go b/internal/adapter/productmetrics/heartbeat_test.go new file mode 100644 index 0000000000..b83eb9cb68 --- /dev/null +++ b/internal/adapter/productmetrics/heartbeat_test.go @@ -0,0 +1,48 @@ +package productmetrics + +import ( + "context" + "testing" + "time" +) + +func TestRecorderHeartbeatRecordsClosedLabelsOnly(t *testing.T) { + r, reader := newTestRecorder(t) + r.Heartbeat(FeatureSnapshot{ + Memory: true, MCP: true, + Provider: ProviderAnthropic, Mode: ModeInteractive, + }) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.adoption.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want 1", got) + } + featureAgg := collected["mecatl.adoption.feature_enabled"] + if got := sumPoint(t, featureAgg, "feature", "memory"); got != 1 { + t.Errorf("feature_enabled{feature=memory} = %d, want 1", got) + } + if got := sumPoint(t, featureAgg, "feature", "mcp"); got != 1 { + t.Errorf("feature_enabled{feature=mcp} = %d, want 1", got) + } + // guardrails/scheduling were false in the snapshot: TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent + // (Task 6) is the exhaustive "no other data point" check; this test + // only asserts the enabled ones are present with the right value. + if got := sumPoint(t, collected["mecatl.adoption.provider_configured"], "family", "anthropic"); got != 1 { + t.Errorf("provider_configured{family=anthropic} = %d, want 1", got) + } + if got := sumPoint(t, collected["mecatl.adoption.deployment_mode"], "mode", "interactive"); got != 1 { + t.Errorf("deployment_mode{mode=interactive} = %d, want 1", got) + } +} + +func TestRunHeartbeatFiresImmediatelyThenStopsOnCtxDone(t *testing.T) { + r, reader := newTestRecorder(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancelled BEFORE RunHeartbeat: only the immediate fire happens. + + RunHeartbeat(ctx, r, time.Hour, FeatureSnapshot{Mode: ModeHeadless}) + + if got := sumValue(t, collect(t, reader)["mecatl.adoption.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want exactly 1 (immediate fire only)", got) + } +} diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index eb416bb1ae..f15b65dcd7 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -27,9 +27,9 @@ const ( ) // Recorder is the product-metrics adapter: it implements port.EventSink -// (this file) and will implement port.ToolCallRecorder (a later task's -// toolcall.go), deriving ONLY the bounded counts in the design's catalog. It -// never reads a tool name, session id, model id, or any free-text field. +// (this file) and port.ToolCallRecorder (toolcall.go), deriving ONLY the +// bounded counts in the design's catalog. It never reads a tool name, +// session id, model id, or any free-text field. type Recorder struct { heartbeat metric.Int64Counter featureEnabled metric.Int64Counter @@ -43,9 +43,11 @@ type Recorder struct { teamUsed metric.Int64Counter } -// Compile-time interface check. port.ToolCallRecorder is satisfied once a -// later task adds Recorder.ToolCall; asserting it here would fail to build. -var _ port.EventSink = (*Recorder)(nil) +// Compile-time interface checks. +var ( + _ port.EventSink = (*Recorder)(nil) + _ port.ToolCallRecorder = (*Recorder)(nil) +) // NewRecorder constructs every instrument from the given MeterProvider. It // returns an error if any instrument fails to construct — the OTel meter API diff --git a/internal/adapter/productmetrics/toolcall.go b/internal/adapter/productmetrics/toolcall.go new file mode 100644 index 0000000000..1d7ade3217 --- /dev/null +++ b/internal/adapter/productmetrics/toolcall.go @@ -0,0 +1,17 @@ +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +// ToolCall records ONLY that a tool call happened — no tool name, no +// session id, no result content, no duration. It satisfies +// port.ToolCallRecorder. The three typed parameters it ignores (id, call, +// result) are accepted only because the port's signature requires them; not +// one of their fields is ever read. +func (r *Recorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + r.toolCalls.Add(context.Background(), 1) +} diff --git a/internal/adapter/productmetrics/toolcall_test.go b/internal/adapter/productmetrics/toolcall_test.go new file mode 100644 index 0000000000..6f6d996470 --- /dev/null +++ b/internal/adapter/productmetrics/toolcall_test.go @@ -0,0 +1,24 @@ +package productmetrics + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +func TestRecorderToolCallCountsWithoutIdentity(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCall( + session.SessionID("sensitive-session-id"), + session.ToolCall{Name: "read_secret_file"}, + session.ToolResult{Content: "super secret content", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.ToolCall(session.SessionID("other"), session.ToolCall{Name: "another_tool"}, session.ToolResult{}, 0, 0) + + agg := collect(t, reader)["mecatl.adoption.tool_calls"] + if got := sumValue(t, agg); got != 2 { + t.Errorf("tool_calls = %d, want 2", got) + } +} From 2a08386d09ece585b3dc8aa5e29c2349225b8421 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:02:50 -0400 Subject: [PATCH 09/47] test(productmetrics): exhaustive guard against unbounded/sensitive attributes Co-Authored-By: Claude Sonnet 5 --- .../adapter/productmetrics/bounded_test.go | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 internal/adapter/productmetrics/bounded_test.go diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go new file mode 100644 index 0000000000..ffb80ecd25 --- /dev/null +++ b/internal/adapter/productmetrics/bounded_test.go @@ -0,0 +1,107 @@ +package productmetrics + +import ( + "context" + "strings" + "testing" + "time" + + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +// allowedAttributeKeys is the COMPLETE set of attribute keys any instrument +// in this package may ever carry. A future change that attaches a new label +// must add it here explicitly — the same "closed set is a reviewed +// decision" discipline as internal/adapter/telemetry's attrRole. +var allowedAttributeKeys = map[string]bool{ + attrStop: true, + attrKind: true, + attrFeature: true, + attrProvider: true, + attrMode: true, +} + +// sensitiveMarkers are strings injected into every field the Recorder must +// NEVER read. If any of these ever shows up in a collected metric name or +// attribute value, something started reading a field it shouldn't. +var sensitiveMarkers = []string{ + "sensitive-session-id-marker", + "secret-tool-name-marker", + "secret-tool-content-marker", + "secret-error-text-marker", +} + +func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T) { + r, reader := newTestRecorder(t) + + // Drive every observation path with deliberately sensitive-looking data. + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopError, + Text: "sensitive-session-id-marker should never be read", + Error: "secret-error-text-marker: connection to 10.0.0.5 failed", + Usage: session.Usage{InputTokens: 1, OutputTokens: 1}, + }, + }) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + r.ToolCall( + session.SessionID("sensitive-session-id-marker"), + session.ToolCall{Name: "secret-tool-name-marker"}, + session.ToolResult{Content: "secret-tool-content-marker", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.Heartbeat(FeatureSnapshot{ + Memory: true, Guardrails: true, MCP: true, Scheduling: true, + Provider: ProviderOther, Mode: ModeK8s, + }) + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + assertNoSensitiveSubstring(t, md.Name) + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("metric %s: aggregation is %T, want Sum[int64]", md.Name, md.Data) + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + key := string(kv.Key) + if !allowedAttributeKeys[key] { + t.Errorf("metric %s carries attribute key %q, not in allowedAttributeKeys", md.Name, key) + } + assertNoSensitiveSubstring(t, kv.Value.AsString()) + } + } + // mecatl.adoption.tool_calls carries NO attributes at all — the + // strongest form of "no tool identity ever attaches." + if md.Name == "mecatl.adoption.tool_calls" { + for _, dp := range sum.DataPoints { + if dp.Attributes.Len() != 0 { + t.Errorf("mecatl.adoption.tool_calls data point carries %d attributes, want 0: %v", + dp.Attributes.Len(), dp.Attributes) + } + } + } + } + } +} + +func assertNoSensitiveSubstring(t *testing.T, s string) { + t.Helper() + for _, marker := range sensitiveMarkers { + if strings.Contains(s, marker) { + t.Errorf("value %q contains sensitive marker %q", s, marker) + } + } +} From f5b5df821ff9571a16055c391b7db1ee8a52e63b Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:07:27 -0400 Subject: [PATCH 10/47] fix(productmetrics): guard the privacy test against a vacuous pass Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/bounded_test.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go index ffb80ecd25..68a013906d 100644 --- a/internal/adapter/productmetrics/bounded_test.go +++ b/internal/adapter/productmetrics/bounded_test.go @@ -65,6 +65,14 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T t.Fatalf("Collect: %v", err) } + totalMetrics := 0 + for _, sm := range rm.ScopeMetrics { + totalMetrics += len(sm.Metrics) + } + if totalMetrics < 10 { + t.Fatalf("collected only %d metrics, want at least 10 (the full mecatl.adoption.* instrument set) — the walk below would otherwise pass vacuously", totalMetrics) + } + for _, sm := range rm.ScopeMetrics { for _, md := range sm.Metrics { assertNoSensitiveSubstring(t, md.Name) From 5493b6b8de800d08f037af1ca3ec0391cd83b83c Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:22:07 -0400 Subject: [PATCH 11/47] feat(permconfig): add the strict telemetry.productMetrics: operator schema Adds the operator-tier telemetry: subtree (telemetry.productMetrics.enabled) to permconfig.Config, parsed strictly (unknown keys error) mirroring OpenRouterSection. Also wires the top-level Config.UnmarshalYAML dispatch table (permconfig.go) so the new key is actually decoded, and registers the subtree in internal/configgen's BuildModel/tier-pin guard so the generated settings reference and skeleton stay in sync with the schema (both required for task lint && task test to stay green; this task adds schema only, resolution/wiring for the actual telemetry gate is a later task). Co-Authored-By: Claude Sonnet 5 --- docs/configuration-reference.md | 11 +++++ internal/adapter/permconfig/permconfig.go | 1 + internal/adapter/permconfig/schema.go | 48 +++++++++++++++++++ .../permconfig/telemetry_schema_test.go | 31 ++++++++++++ internal/configgen/build.go | 26 ++++++++++ internal/configgen/configgen_test.go | 1 + internal/configgen/settings.skeleton.yaml | 19 ++++++++ 7 files changed, 137 insertions(+) create mode 100644 internal/adapter/permconfig/telemetry_schema_test.go diff --git a/docs/configuration-reference.md b/docs/configuration-reference.md index 07afda0ec3..f10e8f0d16 100644 --- a/docs/configuration-reference.md +++ b/docs/configuration-reference.md @@ -240,6 +240,17 @@ OPERATOR-TIER OpenRouter downstream-provider routing (issue #480): a per-model p | `openrouter.models..order` | `[]string` | `(absent)` | Order lists downstream provider slugs (lowercase-kebab, e.g. "anthropic", "google-vertex", "deepinfra/turbo") tried in order. Setting it disables OpenRouter's default price load-balancing. Base-slug matching applies: "google-vertex" matches all its regions/variants (service tiers excepted). | | `openrouter.models..allow_fallbacks` | `bool` | `(absent)` | AllowFallbacks, when explicitly false, pins the request to Order with no fallback to other downstreams. Omit the key to keep OpenRouter's default (true); set it to false to disable fallback. | +## `telemetry` + +Tier: **operator** + +OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). Honoured ONLY from the user-global + CLI tiers; a project-tier telemetry: block is IGNORED with a WARN (a project repo cannot flip a user's own telemetry choice in either direction). Omit entirely to fall through to the DO_NOT_TRACK env var and finally the enabled-by-default posture. + +| Key | Type | Default | Description | +| --- | --- | --- | --- | +| `telemetry.productMetrics` | `productmetricssection` | `(absent)` | ProductMetrics is the opt-out product/adoption metrics config. | +| `telemetry.productMetrics.enabled` | `bool` | `(absent)` | Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit false: nil = absent (composition falls through to DO_NOT_TRACK then the enabled-by-default posture); a non-nil value is honoured exactly. | + ## `mcp` Tier: **operator** diff --git a/internal/adapter/permconfig/permconfig.go b/internal/adapter/permconfig/permconfig.go index 41a39a05ca..37223b1344 100644 --- a/internal/adapter/permconfig/permconfig.go +++ b/internal/adapter/permconfig/permconfig.go @@ -163,6 +163,7 @@ func (c *Config) UnmarshalYAML(node ast.Node) error { "learning": newPermconfigNodePointer(&c.Learning), "steer": newPermconfigNodePointer(&c.Steer), "openrouter": newPermconfigNodePointer(&c.OpenRouter), + "telemetry": newPermconfigNodePointer(&c.Telemetry), "mcp": newPermconfigNodePointer(&c.MCP), "retention": newPermconfigNodePointer(&c.Retention), "storage_management": newPermconfigNodePointer(&c.StorageManagement), diff --git a/internal/adapter/permconfig/schema.go b/internal/adapter/permconfig/schema.go index 7cfa32a72e..9deb767157 100644 --- a/internal/adapter/permconfig/schema.go +++ b/internal/adapter/permconfig/schema.go @@ -152,6 +152,14 @@ type Config struct { // key was absent. The composition layer reads + validates the maps; permconfig // only carries them. OpenRouter *OpenRouterSection `yaml:"openrouter"` + // Telemetry holds the OPERATOR-TIER `telemetry:` subtree (opt-out product/ + // adoption metrics). Like OpenRouter/Guardrails/Posture it is honoured ONLY + // from the user-global + CLI tiers; a project-tier file's telemetry: block + // is IGNORED with a WARN (a project repo cannot flip a user's own telemetry + // choice in either direction). Parsed STRICTLY (unknown keys error). A nil + // Telemetry means the key was absent — composition then falls through the + // DO_NOT_TRACK env var and finally defaults to enabled. + Telemetry *TelemetrySection `yaml:"telemetry"` // MCP holds named global Streamable HTTP MCP server profiles. It is strict and // OPERATOR-TIER ONLY: project files cannot choose endpoints, authentication, // credential references, or egress policy. Values are metadata only; parsing @@ -1247,6 +1255,46 @@ func (m *OpenRouterModelRoute) strictFields() map[string]any { } } +// TelemetrySection is the `telemetry:` operator-tier YAML subtree: the opt-out +// switch for community/adoption product metrics. Parsed STRICTLY (unknown +// keys error), mirroring OpenRouterSection/GuardrailsSection. +type TelemetrySection struct { + // ProductMetrics is the opt-out product/adoption metrics config. + ProductMetrics *ProductMetricsSection `yaml:"productMetrics"` +} + +func (s *TelemetrySection) strictFields() map[string]any { + return map[string]any{ + "productMetrics": newPermconfigNodePointer(&s.ProductMetrics), + } +} + +// UnmarshalYAML decodes the telemetry: mapping STRICTLY: an unknown key +// (e.g. a typo'd product-metrics:) is a parse error, same discipline as +// openrouter:/guardrails:. +func (s *TelemetrySection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry", s.strictFields()) +} + +// ProductMetricsSection is the `telemetry.productMetrics:` subtree. +type ProductMetricsSection struct { + // Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit + // false: nil = absent (composition falls through to DO_NOT_TRACK then the + // enabled-by-default posture); a non-nil value is honoured exactly. + Enabled *bool `yaml:"enabled"` +} + +func (s *ProductMetricsSection) strictFields() map[string]any { + return map[string]any{ + "enabled": newPermconfigNodePointer(&s.Enabled), + } +} + +// UnmarshalYAML decodes the productMetrics: mapping STRICTLY. +func (s *ProductMetricsSection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry.productMetrics", s.strictFields()) +} + // UnmarshalYAML decodes an openrouter.models. entry STRICTLY. func (m *OpenRouterModelRoute) UnmarshalYAML(node ast.Node) error { return decodeStrictMapping(node, "openrouter.models[]", m.strictFields()) diff --git a/internal/adapter/permconfig/telemetry_schema_test.go b/internal/adapter/permconfig/telemetry_schema_test.go new file mode 100644 index 0000000000..c3eccbe34c --- /dev/null +++ b/internal/adapter/permconfig/telemetry_schema_test.go @@ -0,0 +1,31 @@ +package permconfig + +import "testing" + +func TestParseYAMLTelemetryProductMetricsEnabled(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enabled: false\n") + cfg, err := parseYAML(data) + if err != nil { + t.Fatalf("parseYAML: %v", err) + } + if cfg.Telemetry == nil || cfg.Telemetry.ProductMetrics == nil { + t.Fatal("Telemetry.ProductMetrics is nil") + } + if cfg.Telemetry.ProductMetrics.Enabled == nil || *cfg.Telemetry.ProductMetrics.Enabled != false { + t.Errorf("Enabled = %v, want explicit false", cfg.Telemetry.ProductMetrics.Enabled) + } +} + +func TestParseYAMLTelemetryUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productmetric:\n enabled: false\n") // typo: productmetric + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown telemetry.productmetric key, got nil") + } +} + +func TestParseYAMLTelemetryProductMetricsUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enable: false\n") // typo: enable + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown enable key, got nil") + } +} diff --git a/internal/configgen/build.go b/internal/configgen/build.go index 6aaa34697d..24fdc69b23 100644 --- a/internal/configgen/build.go +++ b/internal/configgen/build.go @@ -36,6 +36,7 @@ func BuildModel(docs Docs) *Model { steerSubtree(docs), modelsSubtree(docs), openRouterSubtree(docs), + telemetrySubtree(docs), mcpSubtree(docs), }} } @@ -398,6 +399,31 @@ func modelsSubtree(docs Docs) *Subtree { } } +func telemetrySubtree(docs Docs) *Subtree { + fields := fieldsOf("TelemetrySection", permconfig.TelemetrySection{}, docs) + for _, f := range fields { + if f.Key == "productMetrics" { + f.Nested = fieldsOf("ProductMetricsSection", permconfig.ProductMetricsSection{}, docs) + } + } + return &Subtree{ + Key: "telemetry", + Tier: TierOperator, + Doc: "OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). " + + "Honoured ONLY from the user-global + CLI tiers; a project-tier telemetry: block " + + "is IGNORED with a WARN (a project repo cannot flip a user's own telemetry choice " + + "in either direction). Omit entirely to fall through to the DO_NOT_TRACK env var " + + "and finally the enabled-by-default posture.", + CommentedOut: true, + Fields: fields, + Example: []string{ + "telemetry:", + " productMetrics:", + " enabled: false", + }, + } +} + // docFor returns the harvested doc for key, or a fallback when absent. func docFor(docs Docs, key, fallback string) string { if d := docs[key]; d != "" { diff --git a/internal/configgen/configgen_test.go b/internal/configgen/configgen_test.go index ce569a4037..278c2fdf05 100644 --- a/internal/configgen/configgen_test.go +++ b/internal/configgen/configgen_test.go @@ -312,6 +312,7 @@ func TestSubtreeTiersAreAsPinned(t *testing.T) { "steer": configgen.TierOperator, // operator-only: a project cannot flip the mid-run steer surface (issue #512) "models": configgen.TierProject, // operator + project (project within the operator allowlist) "openrouter": configgen.TierOperator, // operator-only: a project cannot steer the OpenRouter downstream provider (issue #480) + "telemetry": configgen.TierOperator, // operator-only: a project cannot flip a user's own telemetry choice "mcp": configgen.TierOperator, // operator-only: endpoints, auth, credentials, and egress policy } got := map[string]configgen.Tier{} diff --git a/internal/configgen/settings.skeleton.yaml b/internal/configgen/settings.skeleton.yaml index 7fdb30f952..8f24cee78d 100644 --- a/internal/configgen/settings.skeleton.yaml +++ b/internal/configgen/settings.skeleton.yaml @@ -358,6 +358,25 @@ #| order: ["anthropic", "google-vertex"] #| allow_fallbacks: false +#| === telemetry === (tier: operator) +#| OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). Honoured +#| ONLY from the user-global + CLI tiers; a project-tier telemetry: block is IGNORED +#| with a WARN (a project repo cannot flip a user's own telemetry choice in either +#| direction). Omit entirely to fall through to the DO_NOT_TRACK env var and finally the +#| enabled-by-default posture. +# telemetry: +# # ProductMetrics is the opt-out product/adoption metrics config. +# productMetrics: +# # Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit false: nil +# # = absent (composition falls through to DO_NOT_TRACK then the enabled-by-default +# # posture); a non-nil value is honoured exactly. +# enabled: false +#| +#| Example: +#| telemetry: +#| productMetrics: +#| enabled: false + #| === mcp === (tier: operator) #| Strict OPERATOR-TIER Streamable HTTP MCP authority configuration. Mode selects one #| mutually exclusive global or session-broker authority; broker mode carries its From 3558511ca1e00ae0b3b4563bab7feca54e47cd9f Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:28:36 -0400 Subject: [PATCH 12/47] feat(permconfig): resolve the operator-tier telemetry.productMetrics opt-out Add the resolver half of the product-metrics config schema landed in Task 7: an operatorTelemetry field, captureTelemetry capture method mirroring captureOpenRouter, and the OperatorProductMetricsEnabled() accessor. Wired into both loadUserRules capture sites (CLI + user-global) and a project-tier WARN-ignore block in loadProjectRules, so a project repo can never flip a user's own product-metrics opt-out in either direction. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/permconfig/resolve.go | 43 ++++++++++++ .../permconfig/telemetry_resolve_test.go | 67 +++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 internal/adapter/permconfig/telemetry_resolve_test.go diff --git a/internal/adapter/permconfig/resolve.go b/internal/adapter/permconfig/resolve.go index 3e7abb936f..d379f48266 100644 --- a/internal/adapter/permconfig/resolve.go +++ b/internal/adapter/permconfig/resolve.go @@ -196,6 +196,15 @@ type Resolver struct { // (explicit files) out-ranks user-global (first-non-nil keeps CLI). operatorOpenRouter *OpenRouterSection + // operatorTelemetry is the OPERATOR-TIER telemetry: subtree, read ONCE at + // construction from the user-global + CLI tiers ONLY (the SOLE capture path + // is captureTelemetry from loadUserRules — mirroring captureOpenRouter). A + // project-tier file's telemetry: block is IGNORED with a WARN in + // loadProjectRules. nil when no operator-tier file carried a telemetry: + // section. CLI (explicit files) out-ranks user-global (first-non-nil keeps + // CLI). + operatorTelemetry *TelemetrySection + // operatorMCP is the first complete operator-tier mcp: subtree. Explicit CLI // files are visited before user-global settings, so precedence is whole-block, // first-non-nil; project mcp blocks are warning-only and never captured. @@ -392,6 +401,19 @@ func (r *Resolver) OperatorOpenRouter() *OpenRouterSection { return r.operatorOpenRouter } +// OperatorProductMetricsEnabled returns the operator-tier +// telemetry.productMetrics.enabled: value (user-global + CLI only), or nil +// when none was configured. It is the SOLE accessor composition uses to +// read the product-metrics opt-out from config — by construction it never +// returns a project-tier value (a project telemetry: block is ignored with +// a WARN in loadProjectRules). nil-safe. Mirrors OperatorOpenRouter(). +func (r *Resolver) OperatorProductMetricsEnabled() *bool { + if r == nil || r.operatorTelemetry == nil || r.operatorTelemetry.ProductMetrics == nil { + return nil + } + return r.operatorTelemetry.ProductMetrics.Enabled +} + // OperatorMCP returns the complete operator-tier mcp subtree, or nil when absent. // It is metadata only and can never originate from project settings. func (r *Resolver) OperatorMCP() *MCPSection { @@ -674,6 +696,11 @@ func (r *Resolver) loadProjectRules(ws tool.WorkspaceReader) ([]governance.Rule, "openrouter: IGNORING a project-tier openrouter: block (operator-tier only — a project repo cannot steer the OpenRouter downstream provider; set openrouter in your user-global settings.yaml)", "file", src.path, "root", ws.Root()) } + if cfg.Telemetry != nil { + r.diag.Log(context.Background(), port.LevelWarn, + "telemetry: IGNORING a project-tier telemetry: block (operator-tier only — a project repo cannot change a user's own product-metrics opt-out in either direction; set telemetry in your user-global settings.yaml)", + "file", src.path, "root", ws.Root()) + } if cfg.MCP != nil { r.diag.Log(context.Background(), port.LevelWarn, "mcp: IGNORING a project-tier mcp: block (operator-tier only — a project repo cannot configure global MCP servers)", @@ -905,6 +932,9 @@ func (r *Resolver) loadUserRules(report *Report) []governance.Rule { r.captureModels(cfg.Models) // Operator-tier openrouter: same first-non-nil-keeps-CLI discipline (issue #480). r.captureOpenRouter(cfg.OpenRouter) + // Operator-tier telemetry (opt-out product metrics): same + // first-non-nil-keeps-CLI discipline as openrouter. + r.captureTelemetry(cfg.Telemetry) // Operator-tier MCP profiles: capture the complete first block; never field-merge. r.captureMCP(cfg.MCP) r.captureRetention(cfg.Retention) @@ -944,6 +974,9 @@ func (r *Resolver) loadUserRules(report *Report) []governance.Rule { r.captureModels(cfg.Models) // User-global openrouter: captured only if no higher CLI file already did. r.captureOpenRouter(cfg.OpenRouter) + // Operator-tier telemetry (opt-out product metrics): same + // first-non-nil-keeps-CLI discipline as openrouter. + r.captureTelemetry(cfg.Telemetry) // User-global MCP: captured only if no higher CLI file already did. r.captureMCP(cfg.MCP) r.captureRetention(cfg.Retention) @@ -1086,6 +1119,16 @@ func (r *Resolver) captureOpenRouter(s *OpenRouterSection) { r.operatorOpenRouter = s } +// captureTelemetry records the FIRST operator-tier telemetry: block seen +// during loadUserRules (CLI files out-rank user-global, so first-non-nil +// keeps CLI). Mirrors captureOpenRouter. +func (r *Resolver) captureTelemetry(s *TelemetrySection) { + if s == nil || r.operatorTelemetry != nil { + return + } + r.operatorTelemetry = s +} + // captureMCP records the first complete operator-tier mcp block. It is called // only by loadUserRules, whose explicit-files-before-user order defines precedence. func (r *Resolver) captureMCP(s *MCPSection) { diff --git a/internal/adapter/permconfig/telemetry_resolve_test.go b/internal/adapter/permconfig/telemetry_resolve_test.go new file mode 100644 index 0000000000..93eac76ad4 --- /dev/null +++ b/internal/adapter/permconfig/telemetry_resolve_test.go @@ -0,0 +1,67 @@ +package permconfig + +import ( + "bytes" + "context" + "strings" + "testing" + + "github.com/stacklok/mecatl/engine/adapter/memfs" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/slogdiag" +) + +// The telemetry: block (specifically telemetry.productMetrics.enabled) is +// OPERATOR-TIER ONLY: a project repo must never be able to flip a user's own +// product-metrics opt-out in either direction. These tests pin the operator +// capture and the project-tier WARN-ignore (the fail-closed core). Mirrors +// openrouter_test.go. + +const operatorTelemetryYAML = "telemetry:\n productMetrics:\n enabled: false\n" + +// TestOperatorProductMetricsEnabledFromCLIHonoured: an OPERATOR-TIER (CLI +// explicit) telemetry: block is read and returned by +// OperatorProductMetricsEnabled(), parsed faithfully. +func TestOperatorProductMetricsEnabledFromCLIHonoured(t *testing.T) { + env := envWithExplicit("/etc/mecatl/telemetry.yaml", operatorTelemetryYAML) + r := newWithEnv(Options{ExplicitFiles: []string{"/etc/mecatl/telemetry.yaml"}}, env) + if r == nil { + t.Fatal("resolver should be non-nil with an explicit file") + } + got := r.OperatorProductMetricsEnabled() + if got == nil || *got != false { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want explicit false", got) + } +} + +// TestOperatorProductMetricsEnabledAbsentIsNil: no telemetry: config anywhere +// yields a nil accessor result. +func TestOperatorProductMetricsEnabledAbsentIsNil(t *testing.T) { + r := newWithEnv(Options{Conventional: true}, fakeEnv()) + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want nil (absent)", got) + } +} + +// TestProjectTierTelemetryBlockIsIgnoredWithWarn is the FAIL-CLOSED CORE: a +// PROJECT-TIER telemetry: block must NEVER become the operator config, and the +// resolver WARNs naming why (a project repo cannot change the user's own +// product-metrics opt-out). +func TestProjectTierTelemetryBlockIsIgnoredWithWarn(t *testing.T) { + var buf bytes.Buffer + diag := slogdiag.New(&buf, false, port.LevelDebug) + + ws := &countingWS{Workspace: memfs.NewWorkspace("/repo")} + ws.seed(t, projectFileMecatl, operatorTelemetryYAML) + + r := newWithEnv(Options{Conventional: true, TrustProject: true, Diagnostics: diag}, fakeEnv()) + _ = r.Resolve(context.Background(), ws) + + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Fatalf("a PROJECT-tier telemetry: block must NOT become the operator config; got %v", got) + } + log := buf.String() + if !strings.Contains(log, "IGNORING a project-tier telemetry") { + t.Fatalf("expected an ignore-WARN naming the project tier; got:\n%s", log) + } +} From 71eeac1e450990c1bdd3a50025434f2e55cdb230 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:35:10 -0400 Subject: [PATCH 13/47] feat(cliconfig): product-metrics opt-out precedence + ToolCallRecorder fan-out Adds ProductMetricsPrecedence + ResolveProductMetricsEnabled (CLI flag > DO_NOT_TRACK env var > operator settings.yaml > default-enabled) and TeeToolCallRecorder, the ToolCallRecorder twin of telemetry.NewSink's EventSink fan-out, in a new internal/cliconfig/productmetrics_config.go. Co-Authored-By: Claude Sonnet 5 --- internal/cliconfig/productmetrics_config.go | 73 +++++++++++++++++++ .../cliconfig/productmetrics_config_test.go | 57 +++++++++++++++ 2 files changed, 130 insertions(+) create mode 100644 internal/cliconfig/productmetrics_config.go create mode 100644 internal/cliconfig/productmetrics_config_test.go diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go new file mode 100644 index 0000000000..fcb2c457fa --- /dev/null +++ b/internal/cliconfig/productmetrics_config.go @@ -0,0 +1,73 @@ +// Package cliconfig provides product-metrics opt-out precedence and a +// ToolCallRecorder fan-out helper. +package cliconfig + +import ( + "os" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// ProductMetricsPrecedence carries the opt-out inputs +// ResolveProductMetricsEnabled folds, highest precedence first: an explicit +// CLI flag, then the DO_NOT_TRACK env var convention (consoledonottrack.com), +// then the operator settings.yaml value, then default-enabled. +type ProductMetricsPrecedence struct { + // FlagSet/FlagValue report whether --product-metrics was explicitly + // passed on the command line and its value. + FlagSet bool + FlagValue bool + // Getenv abstracts os.Getenv for DO_NOT_TRACK / testing. Defaults to + // os.Getenv when nil. + Getenv func(string) string + // SettingsEnabled is permconfig.Resolver.OperatorProductMetricsEnabled() + // — nil when the operator set no telemetry.productMetrics.enabled value. + SettingsEnabled *bool +} + +// ResolveProductMetricsEnabled applies the opt-out precedence documented on +// ProductMetricsPrecedence. Default (nothing set anywhere) is true — product +// metrics are OPT-OUT, not opt-in. +func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { + if p.FlagSet { + return p.FlagValue + } + getenv := p.Getenv + if getenv == nil { + getenv = os.Getenv + } + if getenv("DO_NOT_TRACK") != "" { + return false + } + if p.SettingsEnabled != nil { + return *p.SettingsEnabled + } + return true +} + +// TeeToolCallRecorder combines multiple ToolCallRecorders into one — the +// ToolCallRecorder twin of internal/adapter/telemetry.NewSink's EventSink +// fan-out (no such helper existed before product metrics, because until now +// only one ToolCallRecorder ever observed a given engine). nil entries are +// skipped, so a caller can pass an always-present operator recorder +// alongside an optional product-metrics one without a conditional slice +// build. +func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecorder { + var non []port.ToolCallRecorder + for _, r := range recorders { + if r != nil { + non = append(non, r) + } + } + return multiToolCallRecorder(non) +} + +type multiToolCallRecorder []port.ToolCallRecorder + +func (m multiToolCallRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + for _, r := range m { + r.ToolCall(id, call, result, queued, took) + } +} diff --git a/internal/cliconfig/productmetrics_config_test.go b/internal/cliconfig/productmetrics_config_test.go new file mode 100644 index 0000000000..a76878ccd2 --- /dev/null +++ b/internal/cliconfig/productmetrics_config_test.go @@ -0,0 +1,57 @@ +package cliconfig + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +func boolPtr(b bool) *bool { return &b } + +func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { + getenvSet := func(string) string { return "1" } + getenvUnset := func(string) string { return "" } + + cases := []struct { + name string + p ProductMetricsPrecedence + want bool + }{ + {"flag true wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: true, Getenv: getenvSet, SettingsEnabled: boolPtr(false)}, true}, + {"flag false wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: false, Getenv: getenvUnset, SettingsEnabled: boolPtr(true)}, false}, + {"DO_NOT_TRACK disables when no flag", ProductMetricsPrecedence{Getenv: getenvSet, SettingsEnabled: boolPtr(true)}, false}, + {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: boolPtr(false)}, false}, + {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: nil}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := ResolveProductMetricsEnabled(tc.p); got != tc.want { + t.Errorf("ResolveProductMetricsEnabled(%+v) = %v, want %v", tc.p, got, tc.want) + } + }) + } +} + +func TestTeeToolCallRecorderCallsEveryNonNilRecorder(t *testing.T) { + var calls []string + rec := func(name string) port.ToolCallRecorder { + return recorderFunc(func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + calls = append(calls, name) + }) + } + tee := TeeToolCallRecorder(rec("a"), nil, rec("b")) + tee.ToolCall(session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if len(calls) != 2 || calls[0] != "a" || calls[1] != "b" { + t.Errorf("calls = %v, want [a b] (nil skipped, order preserved)", calls) + } +} + +// recorderFunc adapts a plain func to port.ToolCallRecorder for this test. +type recorderFunc func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) + +func (f recorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f(id, call, result, queued, took) +} From eb3099b0c1bba0d1e767b739e60ed687cd81b2d8 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:40:59 -0400 Subject: [PATCH 14/47] feat(cliconfig): BuildProductMetrics composition helper + disclosure notice Adds the composition-level pipeline builder (install id -> provider -> recorder -> heartbeat goroutine) that a cmd main threads into its EventSink/ToolCallRecorder fan-out, plus the one-time opt-out disclosure notice text. Disabled returns zero handles with a no-op Shutdown; enabled with no baked ingest key surfaces the error rather than disabling silently. Co-Authored-By: Claude Sonnet 5 --- internal/cliconfig/productmetrics.go | 96 +++++++++++++++++++++++ internal/cliconfig/productmetrics_test.go | 36 +++++++++ 2 files changed, 132 insertions(+) create mode 100644 internal/cliconfig/productmetrics.go create mode 100644 internal/cliconfig/productmetrics_test.go diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go new file mode 100644 index 0000000000..e088336ba5 --- /dev/null +++ b/internal/cliconfig/productmetrics.go @@ -0,0 +1,96 @@ +// Package cliconfig assembles the BuildProductMetrics composition helper — +// the full opt-out product-metrics pipeline (install id → provider → +// recorder → heartbeat goroutine) that a cmd main threads into its +// EventSink/ToolCallRecorder fan-out. Kept separate from +// productmetrics_config.go (the precedence/fan-out helpers Task 9 added): +// this file is the thing that actually constructs the pipeline, not the +// pure-function policy that decides whether to. +package cliconfig + +import ( + "context" + "fmt" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// ProductMetricsDisclosureNotice is printed ONCE — the first run after +// product metrics were enabled and this install's telemetry id did not yet +// exist — to stderr, non-blockingly, before any pipeline is built. Opt-out +// telemetry without a visible disclosure is the pattern that burns +// community trust; this is the whole of that disclosure. +const ProductMetricsDisclosureNotice = `mecatl reports anonymous product-adoption metrics (version, OS/arch, which +major features you have enabled, and coarse session/run/tool-call counts — +never a prompt, file path, tool name, or model id) to help Stacklok understand +community adoption. This is on by default. To opt out: pass +--product-metrics=false, set DO_NOT_TRACK=1, or set +telemetry.productMetrics.enabled: false in your settings.yaml. Details: +. +` + +// ProductMetricsHandles bundles the handles a cmd main threads into its +// EventSink/ToolCallRecorder fan-out (via TeeToolCallRecorder / +// internal/adapter/telemetry.NewSink alongside the operator sink) and its +// shutdown defer. Every field is zero-valued when telemetry is disabled. +type ProductMetricsHandles struct { + Sink port.EventSink + ToolCallRecorder port.ToolCallRecorder + // Shutdown flushes + stops the provider. Always non-nil (a no-op when + // disabled), so a caller can defer it unconditionally. + Shutdown func(context.Context) error + // FirstRun is true the first time this install's telemetry id was just + // minted — the caller prints ProductMetricsDisclosureNotice when true. + FirstRun bool +} + +// BuildProductMetrics constructs the full opt-out product-metrics pipeline +// when enabled is true; when false it returns zero handles (the +// byte-identical disabled posture) and no error. heartbeatInterval is +// productmetrics.DefaultHeartbeatInterval for long-running processes, or 0 +// for a single-fire-only short-lived process (mecatequi). heartbeatCtx is +// cancelled by the caller on shutdown to stop the periodic ticker goroutine +// this starts. +func BuildProductMetrics( + ctx, heartbeatCtx context.Context, + enabled bool, + binary productmetrics.Binary, + version string, + heartbeatInterval time.Duration, + snap productmetrics.FeatureSnapshot, +) (ProductMetricsHandles, error) { + noop := func(context.Context) error { return nil } + if !enabled { + return ProductMetricsHandles{Shutdown: noop}, nil + } + + installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + InstallID: installID, + }) + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: provider: %w", err) + } + + recorder, err := productmetrics.NewRecorder(provider.Meter()) + if err != nil { + _ = provider.Shutdown(ctx) + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: recorder: %w", err) + } + + go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) + + return ProductMetricsHandles{ + Sink: recorder, + ToolCallRecorder: recorder, + Shutdown: provider.Shutdown, + FirstRun: firstRun, + }, nil +} diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go new file mode 100644 index 0000000000..090b1ebfdf --- /dev/null +++ b/internal/cliconfig/productmetrics_test.go @@ -0,0 +1,36 @@ +package cliconfig + +import ( + "context" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=false): %v", err) + } + if h.Sink != nil || h.ToolCallRecorder != nil { + t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder: %+v", h) + } + if h.Shutdown == nil { + t.Fatal("Shutdown must be non-nil even when disabled (a no-op)") + } + if err := h.Shutdown(context.Background()); err != nil { + t.Errorf("no-op Shutdown returned an error: %v", err) + } +} + +func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { + // bakedKey is empty in every non-release build/test — enabling must + // surface the error rather than silently disabling, so a caller notices + // its release build is missing the ldflag. + _, err := BuildProductMetrics(context.Background(), context.Background(), true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + if err == nil { + t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") + } +} From ba5d68006f5b25255252a11259d3655aa202bcc2 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Tue, 8 Sep 2026 23:55:11 -0400 Subject: [PATCH 15/47] feat(mecated): wire opt-out product metrics alongside operator telemetry Adds --product-metrics (default true) and threads the already-built productmetrics/cliconfig pipeline into mecated's EventSink/ToolCallRecorder fan-out. Resolves the effective enabled value via a throwaway permconfig resolver (mirroring mecated's own mcplogin.go precedent, since app.Build's internal resolver is never exposed back to run()) and registers the flag in helpmeta.go's completeness table. Co-Authored-By: Claude Sonnet 5 --- cmd/mecated/helpmeta.go | 1 + cmd/mecated/main.go | 93 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 93 insertions(+), 1 deletion(-) diff --git a/cmd/mecated/helpmeta.go b/cmd/mecated/helpmeta.go index f65d8d5cca..7d4dcfbe8f 100644 --- a/cmd/mecated/helpmeta.go +++ b/cmd/mecated/helpmeta.go @@ -107,6 +107,7 @@ var flagMetaByFlag = map[string]flagMeta{ "perf-mcp": {group: groupObservability, common: false, acp: acpExclude}, "goroutine-warn-threshold": {group: groupObservability, common: false, acp: acpExclude}, "goroutine-warn-interval": {group: groupObservability, common: false, acp: acpExclude}, + "product-metrics": {group: groupObservability, common: true, acp: acpInclude}, // ── Driver connectivity (serve-only) ────────────────────────────────── "driver-auth-token": {group: groupDriver, common: false, acp: acpExclude}, diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index e20e25df4d..21e36a6edf 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -50,6 +50,8 @@ import ( "github.com/stacklok/mecatl/internal/adapter/mcpauthority" "github.com/stacklok/mecatl/internal/adapter/mcpbroker" "github.com/stacklok/mecatl/internal/adapter/mcpperf" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" "github.com/stacklok/mecatl/internal/adapter/server" "github.com/stacklok/mecatl/internal/adapter/skills" "github.com/stacklok/mecatl/internal/adapter/slogdiag" @@ -178,6 +180,10 @@ type config struct { otlpProtocol string // OTLP transport: "grpc" (default) or "http" otlpInsecure bool // skip TLS when dialing the OTLP collector (dev only) + // productMetrics reports anonymous product-adoption metrics to Stacklok. + // OPT-OUT: ON by default. See the --product-metrics flag help text. + productMetrics bool + // Runtime-introspection admin surface (loopback only, on the --metrics-addr // listener): pprof + expvar + a runtime/metrics snapshot + a FlightRecorder. // mutexProfileFraction arms runtime.SetMutexProfileFraction (0 = off); @@ -911,6 +917,21 @@ func run(mode commandMode, remaining []string) error { defer obs.recorder.Stop() } + // Product metrics (opt-out, Task 11): resolve the effective enabled value + // and build the pipeline. Extracted into a helper (mirroring + // setupObservability) so run()'s cyclomatic complexity stays under the + // lint gate; the helper owns the resolve/build/disclosure branches and + // logs its own failure, so run() only threads the resulting handles. + pm, cancelHeartbeat, _ := setupProductMetrics(ctx, cfg, diag) + defer cancelHeartbeat() + defer func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if serr := pm.Shutdown(shutdownCtx); serr != nil { + slog.Warn("product metrics shutdown", "err", serr) + } + }() + tracing := telemetry.NewTracing(otel.GetTracerProvider()) // Role-scoped main pair (issue #47): the MAIN engine records through the @@ -930,6 +951,9 @@ func run(mode commandMode, remaining []string) error { slowTurns = telemetry.NewSlowTurnBuffer(telemetry.DefaultSlowTurnCapacity, time.Now) sinks = append(sinks, slowTurns.WithRole(telemetry.RoleMain)) } + if pm.Sink != nil { + sinks = append(sinks, pm.Sink) + } sink := telemetry.NewSink(sinks...) // Child role scoper (issue #47): the composition hands each CHILD engine a @@ -946,7 +970,7 @@ func run(mode commandMode, remaining []string) error { return telemetry.NewSink(childSinks...), scoped } - composition := appConfig(cfg, sink, mainScoped, roleScoper, obs.metrics, diag) + composition := appConfig(cfg, sink, cliconfig.TeeToolCallRecorder(mainScoped, pm.ToolCallRecorder), roleScoper, obs.metrics, diag) built, err := app.Build(ctx, composition) if err != nil { return err @@ -1059,6 +1083,70 @@ func setupObservability(ctx context.Context, cfg config, diag port.Diagnostics) return observability{providers: providers, metrics: metrics, recorder: recorder}, nil } +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on cfg — never a +// model id/alias, only whether each feature is configured at all. +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + provider := productmetrics.ProviderOther + switch { + case cfg.useOpenAI: + provider = productmetrics.ProviderOpenAI + case strings.Contains(strings.ToLower(cfg.defaultProvider), "openrouter"): + provider = productmetrics.ProviderOpenRouter + case strings.Contains(strings.ToLower(cfg.defaultProvider), "openai"): + provider = productmetrics.ProviderOpenAI + case cfg.defaultProvider == "" || strings.Contains(strings.ToLower(cfg.defaultProvider), "anthropic"): + provider = productmetrics.ProviderAnthropic + } + return productmetrics.FeatureSnapshot{ + Memory: cfg.memoryDir != "", + Guardrails: cfg.guardrailsModel != "", + MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, + Scheduling: !cfg.noScheduler, + Provider: provider, + Mode: productmetrics.ModeInteractive, + } +} + +// setupProductMetrics resolves the opt-out product-metrics precedence and +// builds the pipeline (Task 11). It reads the operator's +// telemetry.productMetrics.enabled setting via a THROWAWAY resolver built +// the SAME WAY internal/app/build.go's buildPermResolver constructs its — +// app.Build's own resolver is internal and never exposed back to run(), so +// this narrow read-only resolver mirrors mecated's mcplogin.go precedent +// (loadMCPLoginProfiles). settings.yaml is parsed twice at boot (once here, +// once inside app.Build); an accepted, negligible boot-time cost. +// +// It logs its own build failure and prints the first-run disclosure, so run() +// only threads the resulting handles and the heartbeat-context cancel func +// (both callers must defer unconditionally: the handles' Shutdown is always +// a safe no-op when disabled/errored). The returned error is informational +// only — a caller that just wants the handles can discard it. +func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) (cliconfig.ProductMetricsHandles, func(), error) { + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: cfg.permissionsConventional, + ImportClaude: cfg.importClaudePermissions, + ExplicitFiles: cfg.permissionConfigs, + Diagnostics: diag, + }, xdgconfig.OSEnv) + productMetricsEnabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.cliExplicit["product-metrics"], + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, + productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot(cfg)) + if err != nil { + slog.Warn("product metrics disabled: setup failed", "err", err) + } + if pm.FirstRun { + fmt.Fprint(os.Stderr, cliconfig.ProductMetricsDisclosureNotice) + } + return pm, cancelHeartbeat, err +} + // mecatedServerImplementation is the stable family reported to authenticated clients. const mecatedServerImplementation = "mecated" @@ -1611,6 +1699,9 @@ func parseFlagsModeOut(mode commandMode, argv []string, out io.Writer) (*flag.Fl fs.StringVar(&cfg.otlpProtocol, "otlp-protocol", telemetry.ProtocolGRPC, "OTLP transport: \"grpc\" (default) or \"http\"") fs.BoolVar(&cfg.otlpInsecure, "otlp-insecure", false, "skip TLS when dialing the OTLP collector (development only)") + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.IntVar(&cfg.mutexProfileFraction, "mutex-profile-fraction", 0, "runtime.SetMutexProfileFraction: report 1/N mutex contention events for /debug/pprof/mutex. 0 (default) disables it. Adds per-contention sampling overhead; enable only when investigating lock contention") fs.IntVar(&cfg.blockProfileRate, "block-profile-rate", 0, "runtime.SetBlockProfileRate in nanoseconds: sample one blocking event per N ns blocked for /debug/pprof/block. 0 (default) disables it. Adds per-block-event overhead; enable only when investigating blocking") fs.BoolVar(&cfg.flightRecorder, "flight-recorder", true, "arm the execution-trace FlightRecorder (bounded in-memory ring buffer) so /debug/flightrecorder can snapshot recent activity. ON by default (low, bounded overhead). Pass --flight-recorder=false to disable") From 49c839496f11b20b891e016a3a65b6bc3fa8754d Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 00:16:29 -0400 Subject: [PATCH 16/47] feat(mecatui): wire opt-out product metrics alongside operator telemetry Adds --product-metrics (default true, embedded-server only) and threads the already-built productmetrics/cliconfig pipeline into mecatui's embedded server's EventSink/ToolCallRecorder fan-out. Unlike mecated, mecatui wires composition.Sink/ToolCallRecorder BEFORE embed.Start rather than at run() itself, because the perf-observability feature (independently OFF by default) can also overwrite those same fields inside embed.Start's setupPerf/wirePerfSinks. wirePerfSinks now captures whatever cfg.Sink/cfg.ToolCallRecorder already held before reassigning them, so the product-metrics tap survives untouched when perf is off (the default) and gets folded in via cliconfig.TeeToolCallRecorder alongside the perf metrics when perf is also enabled. The effective enabled value resolves through a throwaway permconfig resolver built with mecatui's own hardcoded embedded discovery posture (Conventional/ImportClaude: true), mirroring mecated's precedent since app.Build's own resolver is never exposed back to main.go. The disclosure notice goes through diag.Log (never stderr, which would corrupt the Bubble Tea alt-screen) and registers the flag in helpmeta.go's completeness table. Co-Authored-By: Claude Sonnet 5 --- cmd/mecatui/config.go | 17 +++++ cmd/mecatui/embed/embed.go | 15 +++- cmd/mecatui/embed/perf_internal_test.go | 94 +++++++++++++++++++++++++ cmd/mecatui/helpmeta.go | 1 + cmd/mecatui/main.go | 83 +++++++++++++++++++++- 5 files changed, 207 insertions(+), 3 deletions(-) diff --git a/cmd/mecatui/config.go b/cmd/mecatui/config.go index e849d5b438..f649cb60f6 100644 --- a/cmd/mecatui/config.go +++ b/cmd/mecatui/config.go @@ -157,6 +157,16 @@ type config struct { noSteer bool noSteerFlagSet bool + // productMetrics reports anonymous product-adoption metrics to Stacklok + // for the embedded server only (ignored under `mecatui connect`, which + // hosts no local engine). OPT-OUT: ON by default. See the + // --product-metrics flag help text. productMetricsFlagSet records an + // explicit --product-metrics so CLI out-ranks the operator-global + // settings.yaml telemetry.productMetrics.enabled: key (mirrors + // noSteerFlagSet). + productMetrics bool + productMetricsFlagSet bool + // resumeID and resumeLatest select an existing owned main chat for static // startup adoption. They are shared by embedded and connect modes and mutually // exclusive; the first prompt still owns all run-entry attachment/revalidation. @@ -460,6 +470,8 @@ func parseTransportFlags(mode transportMode, out io.Writer, args []string, brows fs.BoolVar(&cfg.noCommands, "no-commands", false, "embedded server only: disable slash-command expansion entirely") fs.StringVar(&cfg.skillsDir, "skills-dir", "", "embedded server only: directory of skill units (/SKILL.md); empty = the conventional dirs (e.g. .claude/skills)") fs.BoolVar(&cfg.noSkills, "no-skills", false, "embedded server only: disable skill discovery (the Skill tool) entirely") + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") fs.BoolVar(&cfg.perf, "perf", false, "embedded server only: expose the private perf-observability admin surface (/metrics, /debug/pprof, /debug/vars, /debug/flightrecorder) and wire domain metrics into the engine. OFF by default. Empty --perf-addr uses a per-instance UNIX socket. SECURITY: UNAUTHENTICATED — its output can embed prompt text/file paths/goroutine stacks") fs.StringVar(&cfg.perfAddr, "perf-addr", "", "embedded server only: explicit loopback host:port for the --perf admin surface (empty = private per-instance UNIX socket, or ephemeral 127.0.0.1 TCP with --perf-mcp). Use 127.0.0.1:0 for explicit ephemeral TCP. Non-loopback addresses are refused. Only consulted with --perf") @@ -644,6 +656,11 @@ func recordExplicitFlag(f *flag.Flag, cfg *config) { case "no-steer": // Record an explicit --no-steer so CLI out-ranks the settings.yaml steer: key. cfg.noSteerFlagSet = true + case "product-metrics": + // Record an explicit --product-metrics so CLI out-ranks the settings.yaml + // telemetry.productMetrics.enabled: key (ResolveProductMetricsEnabled's + // highest-precedence input). + cfg.productMetricsFlagSet = true case "reasoning-effort": cfg.reasoningEffortFlagSet = true case "default-provider": diff --git a/cmd/mecatui/embed/embed.go b/cmd/mecatui/embed/embed.go index 8b15d6e79d..ab7cd5b29e 100644 --- a/cmd/mecatui/embed/embed.go +++ b/cmd/mecatui/embed/embed.go @@ -39,6 +39,7 @@ import ( "github.com/stacklok/mecatl/internal/adapter/slogdiag" "github.com/stacklok/mecatl/internal/adapter/telemetry" "github.com/stacklok/mecatl/internal/app" + "github.com/stacklok/mecatl/internal/cliconfig" ) // socketName is the fixed socket filename inside the per-process temp directory. @@ -547,9 +548,21 @@ func setupPerf(ctx context.Context, perf PerfConfig, cfg *app.Config, runtimeDir // It returns the slow-turn buffer (nil when the perf MCP server is off) for the // mcpperf Deps wiring. func wirePerfSinks(cfg *app.Config, metrics *telemetry.Metrics, tracing port.EventSink, mountMCP bool) *telemetry.SlowTurnBuffer { + // Capture whatever cfg.Sink/cfg.ToolCallRecorder ALREADY held before either + // field is reassigned below — the product-metrics tap main.go wired onto + // composition BEFORE Start (and thus before setupPerf/wirePerfSinks ran), + // when perf is also enabled. Folding it in here (rather than overwriting) + // keeps the tap alive alongside the perf metrics; a nil oldSink/ + // oldToolCallRecorder (perf-only, no product metrics) is the byte-identical + // prior behaviour. + oldSink := cfg.Sink + oldToolCallRecorder := cfg.ToolCallRecorder mainScoped := metrics.WithRole(telemetry.RoleMain) var slowTurns *telemetry.SlowTurnBuffer sinks := []port.EventSink{mainScoped, tracing} + if oldSink != nil { + sinks = append(sinks, oldSink) + } if mountMCP { // The ring stores scalars only (redaction by shape) and spawns no // goroutine — goleak-clean. Built only when the MCP server will read it. @@ -557,7 +570,7 @@ func wirePerfSinks(cfg *app.Config, metrics *telemetry.Metrics, tracing port.Eve sinks = append(sinks, slowTurns.WithRole(telemetry.RoleMain)) } cfg.Sink = telemetry.NewSink(sinks...) - cfg.ToolCallRecorder = mainScoped + cfg.ToolCallRecorder = cliconfig.TeeToolCallRecorder(mainScoped, oldToolCallRecorder) // Schedule metrics (issue #233, Phase 2b): wire the metrics callback over the // telemetry adapter's EmitSchedule, mirroring MetricsRoleScoper. Schedule // metrics are NOT a role-family; this is a separate schedule-lifecycle diff --git a/cmd/mecatui/embed/perf_internal_test.go b/cmd/mecatui/embed/perf_internal_test.go index 1cea183513..7d340b8de6 100644 --- a/cmd/mecatui/embed/perf_internal_test.go +++ b/cmd/mecatui/embed/perf_internal_test.go @@ -6,7 +6,9 @@ import ( "path/filepath" "strings" "testing" + "time" + "github.com/stacklok/mecatl/engine/session" "github.com/stacklok/mecatl/internal/app" ) @@ -36,6 +38,33 @@ func TestSetupPerfDisabledLeavesTelemetrySeamsNil(t *testing.T) { } } +// TestSetupPerfDisabledPreservesPreexistingTap pins the perf-off half of the +// Task 12 product-metrics-tap preservation contract: with PerfConfig.Enabled +// false (mecatui's default), setupPerf's early return must leave a +// preexisting cfg.Sink/cfg.ToolCallRecorder — set by main.go BEFORE embed.Start +// when the product-metrics pipeline is enabled — completely untouched, so it +// survives byte-identically into the running engine. +func TestSetupPerfDisabledPreservesPreexistingTap(t *testing.T) { + preexistingSink := fakeSink{emitted: new(bool)} + preexistingRecorder := fakeRecorder{recorded: new(bool)} + cfg := app.Config{ + Workspace: t.TempDir(), + Model: "mock", + Sink: preexistingSink, + ToolCallRecorder: preexistingRecorder, + } + + if _, err := setupPerf(context.Background(), PerfConfig{}, &cfg, t.TempDir()); err != nil { + t.Fatalf("setupPerf(disabled): %v", err) + } + if cfg.Sink != preexistingSink { + t.Errorf("perf-off cfg.Sink = %#v, want the untouched preexisting sink", cfg.Sink) + } + if cfg.ToolCallRecorder != preexistingRecorder { + t.Errorf("perf-off cfg.ToolCallRecorder = %#v, want the untouched preexisting recorder", cfg.ToolCallRecorder) + } +} + func TestListenPrivateUnixRejectsUnsafeCollisions(t *testing.T) { for _, kind := range []string{"file", "symlink"} { t.Run(kind, func(t *testing.T) { @@ -68,3 +97,68 @@ func TestSetupPerfRejectsNonLoopbackWithoutMCP(t *testing.T) { t.Fatalf("setupPerf error = %v, want non-loopback refusal", err) } } + +// fakeSink and fakeRecorder are minimal port.EventSink/port.ToolCallRecorder +// probes recording whether they were invoked. +type fakeSink struct{ emitted *bool } + +func (f fakeSink) Emit(context.Context, session.Event) { *f.emitted = true } + +type fakeRecorder struct{ recorded *bool } + +func (f fakeRecorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + *f.recorded = true +} + +// TestWirePerfSinksFoldsInPreexistingTap pins the product-metrics-tap +// preservation contract (Task 12): whatever cfg.Sink/cfg.ToolCallRecorder ALREADY +// held before setupPerf/wirePerfSinks ran (main.go wires the product-metrics +// pipeline onto composition BEFORE embed.Start, i.e. before this ever runs) MUST +// still receive every event/tool-call fan-out AFTER perf wiring — not be +// silently overwritten. Exercises the real setupPerf → wirePerfSinks path with +// PerfConfig.Enabled true (the perf-on case; the perf-off case is covered by +// TestSetupPerfDisabledLeavesTelemetrySeamsNil next to it, which asserts a +// preexisting cfg.Sink is untouched because setupPerf never mutates cfg at all). +func TestWirePerfSinksFoldsInPreexistingTap(t *testing.T) { + var preexistingEmitted, preexistingRecorded bool + preexistingSink := fakeSink{emitted: &preexistingEmitted} + preexistingRecorder := fakeRecorder{recorded: &preexistingRecorded} + + cfg := app.Config{ + Workspace: t.TempDir(), + Model: "mock", + Sink: preexistingSink, + ToolCallRecorder: preexistingRecorder, + } + + // A dedicated short-path temp dir for the admin unix socket: t.TempDir() + // embeds this test's (long) name in the path, which overflows the OS unix + // socket path length limit. + runtimeDir, err := os.MkdirTemp("", "embedperf") + if err != nil { + t.Fatalf("MkdirTemp: %v", err) + } + t.Cleanup(func() { _ = os.RemoveAll(runtimeDir) }) + + ps, err := setupPerf(context.Background(), PerfConfig{Enabled: true}, &cfg, runtimeDir) + if err != nil { + t.Fatalf("setupPerf(enabled): %v", err) + } + t.Cleanup(func() { ps.teardown(context.Background()) }) + + if cfg.Sink == nil { + t.Fatal("perf-on cfg.Sink is nil after wiring") + } + cfg.Sink.Emit(context.Background(), session.Event{}) + if !preexistingEmitted { + t.Error("perf-on wiring dropped the preexisting EventSink instead of folding it in") + } + + if cfg.ToolCallRecorder == nil { + t.Fatal("perf-on cfg.ToolCallRecorder is nil after wiring") + } + cfg.ToolCallRecorder.ToolCall(session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + if !preexistingRecorded { + t.Error("perf-on wiring dropped the preexisting ToolCallRecorder instead of teeing it in") + } +} diff --git a/cmd/mecatui/helpmeta.go b/cmd/mecatui/helpmeta.go index 186ea2cf83..c62b094af5 100644 --- a/cmd/mecatui/helpmeta.go +++ b/cmd/mecatui/helpmeta.go @@ -164,6 +164,7 @@ var flagApplicabilityByFlag = map[string]flagApplicability{ "perf-addr": {group: groupObservability, common: false, local: true, connect: false}, "perf-goroutine-warn-threshold": {group: groupObservability, common: false, local: true, connect: false}, "perf-mcp": {group: groupObservability, common: false, local: true, connect: false}, + "product-metrics": {group: groupObservability, common: true, local: true, connect: false}, // ── Info (meta-flags) ─────────────────────────────────────────────────── "help-all": {group: groupInfo, common: false, local: true, connect: true}, diff --git a/cmd/mecatui/main.go b/cmd/mecatui/main.go index 94c5fe415e..8a8dac51ff 100644 --- a/cmd/mecatui/main.go +++ b/cmd/mecatui/main.go @@ -46,6 +46,8 @@ import ( "github.com/stacklok/mecatl/internal/adapter/clientauth" "github.com/stacklok/mecatl/internal/adapter/credentialstore" "github.com/stacklok/mecatl/internal/adapter/mcpauthority" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" "github.com/stacklok/mecatl/internal/adapter/slogdiag" "github.com/stacklok/mecatl/internal/adapter/xdgconfig" "github.com/stacklok/mecatl/internal/app" @@ -939,11 +941,28 @@ func resolveTransport(ctx context.Context, cfg config) (target string, dial clie cfg = applyTrustPrompt(cfg, diag) composition := embeddedConfig(cfg, diag) + // Product metrics (opt-out, Task 12, mirroring mecated's Task 11 wiring): + // resolve the effective enabled value and build the pipeline BEFORE + // embed.Start, so setupPerf/wirePerfSinks (perf is OFF by default) can see + // composition.Sink/ToolCallRecorder already populated and fold them in + // rather than overwrite them when perf IS also enabled. + pm, cancelHeartbeat := setupProductMetrics(ctx, cfg, diag) + if pm.Sink != nil { + composition.Sink = pm.Sink + } + composition.ToolCallRecorder = pm.ToolCallRecorder srv, err := embed.Start(ctx, composition, perfConfig(cfg, perfLogger)) if err != nil { + cancelHeartbeat() + shutdownProductMetrics(pm) _ = diagCloser.Close() return target, client.DialConfig{}, noop, fmt.Errorf("start embedded server: %w", err) } + if pm.FirstRun { + // Through diag, never stderr: stderr would corrupt the Bubble Tea + // alt-screen once the TUI program starts. + diag.Log(ctx, port.LevelInfo, cliconfig.ProductMetricsDisclosureNotice) + } if toFile { // One line, written to the FILE sink (never the TUI), so an operator can find // where the embedded server's diagnostics went. @@ -967,14 +986,74 @@ func resolveTransport(ctx context.Context, cfg config) (target string, dial clie } // The embedded server has no auth/TLS — it is a private UNIX socket dialled // plaintext, the same single-user loopback trust model mecated uses. Cleanup - // closes the server AND the diagnostics log file (a no-op closer for the - // discard/quiet paths), so a clean exit leaks no fd. + // closes the server, stops the product-metrics heartbeat/pipeline, AND closes + // the diagnostics log file (a no-op closer for the discard/quiet paths), so a + // clean exit leaks no fd/goroutine. return srv.Target(), client.DialConfig{Server: srv.Target()}, func() { _ = srv.Close() + cancelHeartbeat() + shutdownProductMetrics(pm) _ = diagCloser.Close() }, nil } +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports for the embedded server. mecatui's feature-flag +// detection is out of scope for this task (the same simplification mecated's +// Task 11 made): only Mode is populated here; Memory/Guardrails/MCP/Scheduling +// stay false and Provider stays the zero value. +func productMetricsSnapshot() productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{Mode: productmetrics.ModeInteractive} +} + +// setupProductMetrics resolves the opt-out product-metrics precedence and builds +// the pipeline (Task 12, mirroring mecated's Task 11 setupProductMetrics) for the +// EMBEDDED server only — `mecatui connect` hosts no local engine and never calls +// this. It reads the operator's telemetry.productMetrics.enabled setting via a +// THROWAWAY resolver mirroring embeddedConfig's own hardcoded discovery posture +// (Conventional/ImportClaude: true, no explicit files, no project trust — +// app.Build's own resolver is internal and never exposed back to main.go, the +// same mecated mcplogin.go precedent). settings.yaml is parsed twice at boot +// (once here, once inside app.Build via embed.Start → app.Build); an accepted, +// negligible boot-time cost. +// +// It logs its own build failure via diag (NEVER stderr — stderr would corrupt +// the Bubble Tea alt-screen) and does NOT print the disclosure notice itself; +// the caller prints cliconfig.ProductMetricsDisclosureNotice through diag.Log +// when the returned handles' FirstRun is true, after the embedded server has +// started successfully. The returned cancel func must be called/deferred +// unconditionally by the caller (Shutdown on the handles is always a safe +// no-op when disabled/errored). +func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) (cliconfig.ProductMetricsHandles, func()) { + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: true, + ImportClaude: true, + Diagnostics: diag, + }, xdgconfig.OSEnv) + productMetricsEnabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.productMetricsFlagSet, + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, + productmetrics.BinaryMecatui, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot()) + if err != nil { + diag.Log(ctx, port.LevelWarn, "mecatui: product metrics disabled: setup failed", "err", err.Error()) + } + return pm, cancelHeartbeat +} + +// shutdownProductMetrics bounds pm.Shutdown the same way mecated's run() bounds +// its own product-metrics Shutdown defer (5s), for use on every path +// resolveTransport returns (the error path and the success cleanup closure). +func shutdownProductMetrics(pm cliconfig.ProductMetricsHandles) { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = pm.Shutdown(shutdownCtx) +} + // applyTrustPrompt runs the pre-TUI first-encounter workspace-trust gate // (Workspace-Trust Phase 2c) and returns cfg with trustProject set when the // operator (or an already-existing trust grant) trusts the run. It builds the From c6a5748fb4dde1903b838ad6a3c066bdd7d24113 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 00:33:18 -0400 Subject: [PATCH 17/47] feat(mecatequi,mecak8s): wire opt-out product metrics via cliconfig Extend buildObservability in both binaries to take a port.Diagnostics and build the opt-out product-metrics pipeline (cliconfig.BuildProductMetrics) alongside the existing opt-in OTLP telemetry, fanning both into the engine's EventSink/ToolCallRecorder (nil-guarded, since TeeToolCallRecorder is not nil-safe over an all-nil input). Adds --product-metrics (default true) to both flag surfaces, flushes the pipeline on exit/SIGTERM, and prints the first-run disclosure notice to stderr. Co-Authored-By: Claude Sonnet 5 --- cmd/mecak8s/flags.go | 19 ++++- cmd/mecak8s/main.go | 2 +- cmd/mecak8s/observability.go | 128 +++++++++++++++++++++++++++----- cmd/mecak8s/telemetry_test.go | 6 +- cmd/mecatequi/flags.go | 21 +++++- cmd/mecatequi/main.go | 2 +- cmd/mecatequi/observability.go | 126 ++++++++++++++++++++++++++----- cmd/mecatequi/telemetry_test.go | 2 +- 8 files changed, 261 insertions(+), 45 deletions(-) diff --git a/cmd/mecak8s/flags.go b/cmd/mecak8s/flags.go index 5e708ab928..ff569fe57e 100644 --- a/cmd/mecak8s/flags.go +++ b/cmd/mecak8s/flags.go @@ -294,6 +294,13 @@ type config struct { otlpMetricsEndpoint string otlpMetricsProtocol string otlpShutdownTimeout time.Duration + + // productMetrics reports anonymous product-adoption metrics to Stacklok. + // OPT-OUT: ON by default. See the --product-metrics flag help text. + productMetrics bool + // productMetricsSet records whether --product-metrics was explicitly passed, + // so ResolveProductMetricsEnabled can let CLI out-rank DO_NOT_TRACK/settings. + productMetricsSet bool } // stringList is a repeatable string flag.Value, preserving order across @@ -468,6 +475,9 @@ func parseFlags(argv []string) (config, error) { fs.StringVar(&cfg.otlpMetricsProtocol, "otlp-metrics-protocol", "grpc", "OTLP transport for metrics: \"grpc\" (default) or \"http\"") fs.DurationVar(&cfg.otlpShutdownTimeout, "otlp-shutdown-timeout", 5*time.Second, "bound on the telemetry flush at SIGTERM (so a dead collector cannot hang shutdown). 0 disables the bound") + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.Usage = func() { _, _ = fmt.Fprint(fs.Output(), "Usage: mecak8s [flags]\n\n") cliconfig.PrintDefaults(fs.Output(), fs) @@ -500,6 +510,8 @@ func parseFlags(argv []string) (config, error) { cfg.reasoningEffortFlagSet = true case "subagent-model-router": cfg.subagentModelRouterSet = true + case "product-metrics": + cfg.productMetricsSet = true } markRetentionCLIFlag(&cfg.retentionCLISet, fl.Name) if fl.Name == "schedule-fire-retention" { @@ -715,8 +727,11 @@ func appConfig(cfg config, diag port.Diagnostics, obs observability) app.Config Diagnostics: diag, // Observability (issue #343, ADR 0098): OPT-IN. With no --otlp-* flags the // handles are zero-valued (nil) — the byte-identical no-metrics posture. - Sink: obs.Sink, - ToolCallRecorder: obs.ToolCallRecorder, + // The opt-out product-metrics Sink/ToolCallRecorder are folded in + // alongside (nil-guarded fan-out): both nil reproduces the + // byte-identical no-telemetry posture exactly. + Sink: productMetricsSink(obs), + ToolCallRecorder: productMetricsRecorder(obs), MetricsRoleScoper: obs.MetricsRoleScoper, SessionLoadFailureMetricsEmitter: obs.SessionLoadFailureMetricsEmitter, } diff --git a/cmd/mecak8s/main.go b/cmd/mecak8s/main.go index 0eb770fb6a..d1bcc0c430 100644 --- a/cmd/mecak8s/main.go +++ b/cmd/mecak8s/main.go @@ -76,7 +76,7 @@ func run() error { // flags this is a no-op (byte-identical default). The flush defer runs BEFORE // built.Close() (LIFO), so the OTLP flush completes before the service tears // down on the SIGTERM path. - obs, oerr := buildObservability(ctx, cfg) + obs, oerr := buildObservability(ctx, cfg, diag) if oerr != nil { return fmt.Errorf("telemetry: %w", oerr) } diff --git a/cmd/mecak8s/observability.go b/cmd/mecak8s/observability.go index 4f76ab6ea5..c368bb2ecb 100644 --- a/cmd/mecak8s/observability.go +++ b/cmd/mecak8s/observability.go @@ -6,24 +6,44 @@ import ( "io" "time" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/adapter/telemetry" + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" + "github.com/stacklok/mecatl/internal/buildinfo" "github.com/stacklok/mecatl/internal/cliconfig" ) // observability carries the telemetry handles run() threads into appConfig and // serve(), plus the flush-on-SIGTERM Shutdown. With no --otlp-* / --metrics-addr -// flags every field is zero-valued and Shutdown is a no-op (the byte-identical -// no-telemetry posture). +// flags every HeadlessTelemetryHandles field is zero-valued and Shutdown is a +// no-op (the byte-identical no-telemetry posture). productMetrics carries the +// opt-out product-adoption metrics handles (issue #343 follow-up): Shutdown is +// always non-nil (a no-op when disabled) so the caller can defer it +// unconditionally. type observability struct { cliconfig.HeadlessTelemetryHandles + productMetrics cliconfig.ProductMetricsHandles } // buildObservability wires the OPT-IN OTLP telemetry pipeline for mecak8s via // the shared cliconfig.HeadlessTelemetry helper (the SAME Setup→NewMetrics→ -// WithRole→scoper path mecated wires inline). With no --otlp-* endpoints the -// helper returns zero handles (byte-identical default). The caller owns the -// Shutdown defer (flush on SIGTERM). The /metrics loopback listener is wired -// separately in serve() from the returned Registry. -func buildObservability(ctx context.Context, cfg config) (observability, error) { +// WithRole→scoper path mecated wires inline), plus the OPT-OUT product-metrics +// pipeline. With no --otlp-* endpoints the OTLP helper returns zero handles +// (byte-identical default). The caller owns the Shutdown defer (flush on +// SIGTERM). The /metrics loopback listener is wired separately in serve() from +// the returned Registry. +// +// Product metrics: mecak8s is long-running, so heartbeatCtx is the SAME +// signal-driven ctx run() already has in scope (cancelled by its own +// signalCtx()/stop() chain on SIGTERM — no separate cancel function needed) +// and heartbeatInterval is productmetrics.DefaultHeartbeatInterval (the +// steady-state ticker). A build failure (e.g. --product-metrics=true forced on +// with no baked ingest key) is NEVER fatal — product metrics are best-effort — +// so it degrades to a no-op Shutdown rather than failing this function's error +// return (which stays meaningful for the OTLP half only). +func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecak8s", OTLPTraceEndpoint: cfg.otlpEndpoint, @@ -39,24 +59,98 @@ func buildObservability(ctx context.Context, cfg config) (observability, error) if err != nil { return observability{}, err } - return observability{HeadlessTelemetryHandles: h}, nil + + // Product metrics (opt-out): resolve the effective enabled value via a + // THROWAWAY resolver mirroring mecated's setupProductMetrics precedent — + // app.Build's own resolver is internal and never exposed back here, so + // this narrow read-only resolver re-parses the same operator + // settings.yaml (an accepted, negligible boot-time cost, same as the + // other two mains). + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: cfg.permissionsConventional, + ImportClaude: cfg.importClaudePermissions, + ExplicitFiles: []string(cfg.permissionConfigs), + Diagnostics: diag, + }, xdgconfig.OSEnv) + enabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.productMetricsSet, + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, + productmetrics.BinaryMecak8s, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeK8s}) + if pmErr != nil { + // Mirror the existing telemetry-setup-failure posture: a warning, never + // a fatal error — product metrics are best-effort and must not block + // the daemon from starting. + diag.Log(ctx, port.LevelWarn, "product metrics disabled: setup failed", "err", pmErr) + pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} + } + + return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } -// flushTelemetry runs the telemetry Shutdown (flush) with a bounded ctx so a -// dead collector cannot hang SIGTERM shutdown. Safe on a zero observability -// (Shutdown is a no-op when telemetry is disabled). A flush failure is logged -// and never aborts — telemetry is best-effort at shutdown. +// flushTelemetry runs the OTLP + product-metrics Shutdown (flush) with a +// bounded ctx so a dead collector cannot hang SIGTERM shutdown. Safe on a zero +// observability (both Shutdowns are no-ops when telemetry is disabled). A +// flush failure is logged and never aborts — telemetry is best-effort at +// shutdown. The product-metrics first-run disclosure notice is printed to +// stderr here too (mecak8s already writes plain informational lines to +// stderr in main.go's fmt.Fprintf calls; slog goes to the structured logger). func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { - if obs.Shutdown == nil { - return - } ctx := context.Background() if timeout > 0 { var cancel context.CancelFunc ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } - if err := obs.Shutdown(ctx); err != nil { - _, _ = fmt.Fprintf(stderr, "mecak8s: telemetry flush: %v\n", err) + if obs.Shutdown != nil { + if err := obs.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecak8s: telemetry flush: %v\n", err) + } + } + if obs.productMetrics.Shutdown != nil { + if err := obs.productMetrics.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecak8s: product metrics flush: %v\n", err) + } + } + if obs.productMetrics.FirstRun { + _, _ = fmt.Fprint(stderr, cliconfig.ProductMetricsDisclosureNotice) + } +} + +// productMetricsSink fans obs.Sink (the OTLP sink, nil when telemetry is off) +// together with obs.productMetrics.Sink (nil when product metrics are off) +// into one EventSink. telemetry.NewSink's fanOut.Emit calls every wrapped +// sink unconditionally, so a nil element would panic — both are filtered into +// a non-nil-only slice first (mirroring mecated/mecatui's Task 11/12 pattern). +// With both nil this returns nil, reproducing the byte-identical +// no-telemetry Sink posture exactly. +func productMetricsSink(obs observability) port.EventSink { + var sinks []port.EventSink + if obs.Sink != nil { + sinks = append(sinks, obs.Sink) + } + if obs.productMetrics.Sink != nil { + sinks = append(sinks, obs.productMetrics.Sink) + } + if len(sinks) == 0 { + return nil + } + return telemetry.NewSink(sinks...) +} + +// productMetricsRecorder fans obs.ToolCallRecorder together with +// obs.productMetrics.ToolCallRecorder via cliconfig.TeeToolCallRecorder — but +// ONLY when at least one is non-nil. TeeToolCallRecorder always returns a +// non-nil multiToolCallRecorder interface value even over an all-nil input +// (a typed-nil-slice wrapper, not a nil interface), which would break the +// byte-identical no-telemetry ToolCallRecorder-is-nil posture when both +// sources are off. With both nil this returns nil. +func productMetricsRecorder(obs observability) port.ToolCallRecorder { + if obs.ToolCallRecorder == nil && obs.productMetrics.ToolCallRecorder == nil { + return nil } + return cliconfig.TeeToolCallRecorder(obs.ToolCallRecorder, obs.productMetrics.ToolCallRecorder) } diff --git a/cmd/mecak8s/telemetry_test.go b/cmd/mecak8s/telemetry_test.go index fa51b96e3e..31cebef8b1 100644 --- a/cmd/mecak8s/telemetry_test.go +++ b/cmd/mecak8s/telemetry_test.go @@ -136,7 +136,7 @@ func TestTelemetryDefaultIsNil(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), cfg) + obs, err := buildObservability(context.Background(), cfg, port.NopDiagnostics{}) if err != nil { t.Fatalf("buildObservability: %v", err) } @@ -191,7 +191,7 @@ func TestTelemetryMetricsAddrServesPrometheus(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), cfg) + obs, err := buildObservability(context.Background(), cfg, port.NopDiagnostics{}) if err != nil { t.Fatalf("buildObservability: %v", err) } @@ -305,7 +305,7 @@ func TestTelemetryPushesRunMetricsOnExit(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), cfg) + obs, err := buildObservability(context.Background(), cfg, port.NopDiagnostics{}) if err != nil { t.Fatalf("buildObservability: %v", err) } diff --git a/cmd/mecatequi/flags.go b/cmd/mecatequi/flags.go index 2e1226309f..2e39b740a6 100644 --- a/cmd/mecatequi/flags.go +++ b/cmd/mecatequi/flags.go @@ -161,6 +161,13 @@ type flags struct { otlpMetricsEndpoint string otlpMetricsProtocol string otlpShutdownTimeout time.Duration + + // productMetrics reports anonymous product-adoption metrics to Stacklok. + // OPT-OUT: ON by default. See the --product-metrics flag help text. + productMetrics bool + // productMetricsSet records whether --product-metrics was explicitly passed, + // so ResolveProductMetricsEnabled can let CLI out-rank DO_NOT_TRACK/settings. + productMetricsSet bool } // parseFlags turns argv into a flags value, resolving env-derived defaults and @@ -237,6 +244,9 @@ func parseFlags(argv []string) (flags, error) { fs.StringVar(&f.otlpMetricsProtocol, "otlp-metrics-protocol", "grpc", "OTLP transport for metrics: \"grpc\" (default) or \"http\"") fs.DurationVar(&f.otlpShutdownTimeout, "otlp-shutdown-timeout", 5*time.Second, "bound on the telemetry flush at exit (so a dead collector cannot hang the run). 0 disables the bound (flush until it completes); the flush runs BEFORE the diff/summary emit defer unwinds") + fs.BoolVar(&f.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.Usage = usageEpilogue(fs) if err := fs.Parse(argv); err != nil { @@ -266,6 +276,8 @@ func parseFlags(argv []string) (flags, error) { // --out-summary=- selects it. The unset default also resolves to "-" // but keeps the indented JSON — default behavior unchanged. f.summaryCompact = f.outSummary == "-" + case "product-metrics": + f.productMetricsSet = true } if fl.Name == "reasoning-effort" { f.reasoningEffortFlagSet = true @@ -449,9 +461,12 @@ func appConfig(f flags, diag port.Diagnostics, obs observability) app.Config { Diagnostics: diag, // Observability (issue #343, ADR 0098): OPT-IN OTLP push. With no --otlp-* // flags the handles are zero-valued (nil Sink/ToolCallRecorder/ - // MetricsRoleScoper) — the byte-identical no-telemetry posture. - Sink: obs.Sink, - ToolCallRecorder: obs.ToolCallRecorder, + // MetricsRoleScoper) — the byte-identical no-telemetry posture. The + // opt-out product-metrics Sink/ToolCallRecorder are folded in alongside + // (nil-guarded fan-out): both nil reproduces the byte-identical + // no-telemetry posture exactly. + Sink: productMetricsSink(obs), + ToolCallRecorder: productMetricsRecorder(obs), MetricsRoleScoper: obs.MetricsRoleScoper, SessionLoadFailureMetricsEmitter: obs.SessionLoadFailureMetricsEmitter, } diff --git a/cmd/mecatequi/main.go b/cmd/mecatequi/main.go index c09e6fbb50..40205504e5 100644 --- a/cmd/mecatequi/main.go +++ b/cmd/mecatequi/main.go @@ -73,7 +73,7 @@ func realMain(argv []string, stdout, stderr io.Writer) int { // Observability (issue #343, ADR 0098): OPT-IN OTLP push. Built right after // flag parse so the flush-on-exit defer covers EVERY exit path (setup-failure // included). With no --otlp-* flags this is a no-op (byte-identical default). - obs, oerr := buildObservability(context.Background(), f) + obs, oerr := buildObservability(context.Background(), f, diag) if oerr != nil { _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry: %v\n", oerr) return 2 diff --git a/cmd/mecatequi/observability.go b/cmd/mecatequi/observability.go index b7dba950a6..eedc188646 100644 --- a/cmd/mecatequi/observability.go +++ b/cmd/mecatequi/observability.go @@ -6,24 +6,43 @@ import ( "io" "time" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/adapter/telemetry" + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" + "github.com/stacklok/mecatl/internal/buildinfo" "github.com/stacklok/mecatl/internal/cliconfig" ) // observability carries the telemetry handles realMain threads into appConfig, // plus the flush-on-exit Shutdown the main owns. With no --otlp-* flags every -// field is zero-valued and Shutdown is a no-op (the byte-identical no-telemetry -// posture). +// HeadlessTelemetryHandles field is zero-valued and Shutdown is a no-op (the +// byte-identical no-telemetry posture). productMetrics carries the opt-out +// product-adoption metrics handles (issue #343 follow-up): Shutdown is always +// non-nil (a no-op when disabled) so the caller can defer it unconditionally. type observability struct { cliconfig.HeadlessTelemetryHandles + productMetrics cliconfig.ProductMetricsHandles } // buildObservability wires the OPT-IN OTLP telemetry pipeline for mecatequi via // the shared cliconfig.HeadlessTelemetry helper (the SAME Setup→NewMetrics→ -// WithRole→scoper path mecated wires inline). It keeps appConfig a pure mapping -// over flags + these handles. With no --otlp-* endpoints the helper returns zero -// handles, so the no-telemetry default is byte-identical. The caller owns the -// Shutdown defer (flush-before-exit). -func buildObservability(ctx context.Context, f flags) (observability, error) { +// WithRole→scoper path mecated wires inline), plus the OPT-OUT product-metrics +// pipeline. It keeps appConfig a pure mapping over flags + these handles. With +// no --otlp-* endpoints the OTLP helper returns zero handles, so the +// no-telemetry default is byte-identical. The caller owns the Shutdown defer +// (flush-before-exit). +// +// Product metrics: mecatequi is single-shot/short-lived, so heartbeatCtx is +// context.Background() (nothing to cancel — the process exits right after) +// and heartbeatInterval is 0 (a single immediate fire only, no ticker), +// mirroring mecatequi's own push-before-exit OTLP shape. A build failure (e.g. +// --product-metrics=true forced on with no baked ingest key) is NEVER fatal — +// product metrics are best-effort — so it degrades to a no-op Shutdown rather +// than failing this function's error return (which stays meaningful for the +// OTLP half only). +func buildObservability(ctx context.Context, f flags, diag port.Diagnostics) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecatequi", OTLPTraceEndpoint: f.otlpEndpoint, @@ -36,24 +55,97 @@ func buildObservability(ctx context.Context, f flags) (observability, error) { if err != nil { return observability{}, err } - return observability{HeadlessTelemetryHandles: h}, nil + + // Product metrics (opt-out): resolve the effective enabled value via a + // THROWAWAY resolver mirroring mecated's setupProductMetrics / + // mecatui's setupProductMetrics precedent — app.Build's own resolver is + // internal and never exposed back here, so this narrow read-only resolver + // re-parses the same operator settings.yaml (an accepted, negligible + // boot-time cost, same as the other two mains). + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: true, + ExplicitFiles: []string(f.permissionConfigs), + Diagnostics: diag, + }, xdgconfig.OSEnv) + enabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: f.productMetricsSet, + FlagValue: f.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + pm, pmErr := cliconfig.BuildProductMetrics(ctx, context.Background(), enabled, + productmetrics.BinaryMecatequi, buildinfo.BuildID, 0, /* single fire, short-lived */ + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}) + if pmErr != nil { + // Mirror the existing telemetry-setup-failure posture: a warning, never + // a fatal error — product metrics are best-effort and must not block a + // CI run. + diag.Log(ctx, port.LevelWarn, "product metrics disabled: setup failed", "err", pmErr) + pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} + } + + return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } -// flushTelemetry runs the telemetry Shutdown (flush) with a bounded ctx so a -// dead collector cannot hang the run. It is safe to call on a zero observability -// (Shutdown is a no-op when telemetry is disabled). A flush failure is logged -// to stderr and never aborts — telemetry is best-effort at exit. +// flushTelemetry runs the OTLP + product-metrics Shutdown (flush) with a +// bounded ctx so a dead collector cannot hang the run. It is safe to call on a +// zero observability (both Shutdowns are no-ops when telemetry is disabled). A +// flush failure is logged to stderr and never aborts — telemetry is +// best-effort at exit. The product-metrics first-run disclosure notice is +// printed to stderr here too (mecatequi already writes plain informational +// lines to stderr — see emitAuthFileWarning/verdictLine). func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { - if obs.Shutdown == nil { - return - } ctx := context.Background() if timeout > 0 { var cancel context.CancelFunc ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } - if err := obs.Shutdown(ctx); err != nil { - _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry flush: %v\n", err) + if obs.Shutdown != nil { + if err := obs.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry flush: %v\n", err) + } + } + if obs.productMetrics.Shutdown != nil { + if err := obs.productMetrics.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: product metrics flush: %v\n", err) + } + } + if obs.productMetrics.FirstRun { + _, _ = fmt.Fprint(stderr, cliconfig.ProductMetricsDisclosureNotice) + } +} + +// productMetricsSink fans obs.Sink (the OTLP sink, nil when telemetry is off) +// together with obs.productMetrics.Sink (nil when product metrics are off) +// into one EventSink. telemetry.NewSink's fanOut.Emit calls every wrapped +// sink unconditionally, so a nil element would panic — both are filtered into +// a non-nil-only slice first (mirroring mecated/mecatui's Task 11/12 pattern). +// With both nil this returns nil, reproducing the byte-identical +// no-telemetry Sink posture exactly. +func productMetricsSink(obs observability) port.EventSink { + var sinks []port.EventSink + if obs.Sink != nil { + sinks = append(sinks, obs.Sink) + } + if obs.productMetrics.Sink != nil { + sinks = append(sinks, obs.productMetrics.Sink) + } + if len(sinks) == 0 { + return nil + } + return telemetry.NewSink(sinks...) +} + +// productMetricsRecorder fans obs.ToolCallRecorder together with +// obs.productMetrics.ToolCallRecorder via cliconfig.TeeToolCallRecorder — but +// ONLY when at least one is non-nil. TeeToolCallRecorder always returns a +// non-nil multiToolCallRecorder interface value even over an all-nil input +// (a typed-nil-slice wrapper, not a nil interface), which would break the +// byte-identical no-telemetry ToolCallRecorder-is-nil posture when both +// sources are off. With both nil this returns nil. +func productMetricsRecorder(obs observability) port.ToolCallRecorder { + if obs.ToolCallRecorder == nil && obs.productMetrics.ToolCallRecorder == nil { + return nil } + return cliconfig.TeeToolCallRecorder(obs.ToolCallRecorder, obs.productMetrics.ToolCallRecorder) } diff --git a/cmd/mecatequi/telemetry_test.go b/cmd/mecatequi/telemetry_test.go index 4fd88509b8..35135076aa 100644 --- a/cmd/mecatequi/telemetry_test.go +++ b/cmd/mecatequi/telemetry_test.go @@ -136,7 +136,7 @@ func TestTelemetryDefaultIsNil(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), f) + obs, err := buildObservability(context.Background(), f, newDiagnostics()) if err != nil { t.Fatalf("buildObservability: %v", err) } From 3b26b02fd75c83b3120f4b1feb45ce239641ce19 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 00:42:30 -0400 Subject: [PATCH 18/47] fix(mecak8s): print the product-metrics disclosure notice at startup, not shutdown mecak8s is a long-running daemon, so printing the first-run disclosure only in flushTelemetry left it invisible until process exit (potentially days/weeks later) and never printed at all on a SIGKILL/OOM-kill. Move the print into buildObservability, right after BuildProductMetrics returns, mirroring cmd/mecated/main.go's setupProductMetrics shape. mecatequi is unaffected (single-shot process; setup and flush are seconds apart there). Co-Authored-By: Claude Sonnet 5 --- cmd/mecak8s/observability.go | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/cmd/mecak8s/observability.go b/cmd/mecak8s/observability.go index c368bb2ecb..e9b57f2df6 100644 --- a/cmd/mecak8s/observability.go +++ b/cmd/mecak8s/observability.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "io" + "os" "time" "github.com/stacklok/mecatl/engine/port" @@ -43,6 +44,15 @@ type observability struct { // with no baked ingest key) is NEVER fatal — product metrics are best-effort — // so it degrades to a no-op Shutdown rather than failing this function's error // return (which stays meaningful for the OTLP half only). +// +// The first-run disclosure notice is printed HERE, at setup time — NOT in +// flushTelemetry — because mecak8s is a long-running daemon (unlike +// mecatequi's single-shot process, where setup and flush are seconds apart): +// printing only at shutdown would leave the notice invisible for as long as +// the process runs (potentially days/weeks) and never printed at all on a +// SIGKILL/OOM-kill with no graceful shutdown path. This mirrors +// cmd/mecated/main.go's setupProductMetrics, which prints at setup for the +// same reason. func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecak8s", @@ -87,6 +97,13 @@ func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) diag.Log(ctx, port.LevelWarn, "product metrics disabled: setup failed", "err", pmErr) pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} } + if pm.FirstRun { + // stderr, not diag: mecak8s already writes plain informational lines to + // stderr elsewhere (e.g. boundedClose's timeout line in main.go), and the + // disclosure banner is a one-time, human-facing notice rather than a + // structured operational log line. + _, _ = fmt.Fprint(os.Stderr, cliconfig.ProductMetricsDisclosureNotice) + } return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } @@ -95,9 +112,8 @@ func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) // bounded ctx so a dead collector cannot hang SIGTERM shutdown. Safe on a zero // observability (both Shutdowns are no-ops when telemetry is disabled). A // flush failure is logged and never aborts — telemetry is best-effort at -// shutdown. The product-metrics first-run disclosure notice is printed to -// stderr here too (mecak8s already writes plain informational lines to -// stderr in main.go's fmt.Fprintf calls; slog goes to the structured logger). +// shutdown. (The first-run disclosure notice is printed at buildObservability +// setup time, not here — see that function's doc comment.) func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { ctx := context.Background() if timeout > 0 { @@ -115,9 +131,6 @@ func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) _, _ = fmt.Fprintf(stderr, "mecak8s: product metrics flush: %v\n", err) } } - if obs.productMetrics.FirstRun { - _, _ = fmt.Fprint(stderr, cliconfig.ProductMetricsDisclosureNotice) - } } // productMetricsSink fans obs.Sink (the OTLP sink, nil when telemetry is off) From f3945f22aa0d22ecfee1044a851118dc4c2779d9 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 00:57:36 -0400 Subject: [PATCH 19/47] feat(productmetrics): --product-metrics-dry-run audit mode across all four binaries Adds DryRunRecorder (port.EventSink + port.ToolCallRecorder) which logs every would-be product-metrics observation via port.Diagnostics instead of exporting it over OTLP, reading only the same bounded fields the real Recorder reads (event type, stop reason, token counts by kind, feature/provider/mode enums) -- never a tool name, session id, or free text. BuildProductMetrics gains a dryRun bool (after enabled) and a port.Diagnostics parameter; when dryRun is true it short-circuits before any install-id read/write or real OTLP provider construction, firing one representative Heartbeat instead of a full ticker. Wires --product-metrics-dry-run (default false) into mecated, mecatui, mecatequi, and mecak8s, threading each binary's existing diag into its BuildProductMetrics call site. Co-Authored-By: Claude Sonnet 5 --- cmd/mecak8s/flags.go | 6 + cmd/mecak8s/observability.go | 4 +- cmd/mecated/helpmeta.go | 1 + cmd/mecated/main.go | 10 +- cmd/mecatequi/flags.go | 6 + cmd/mecatequi/observability.go | 4 +- cmd/mecatui/config.go | 6 + cmd/mecatui/helpmeta.go | 1 + cmd/mecatui/main.go | 4 +- internal/adapter/productmetrics/dryrun.go | 85 +++++++++++++++ .../adapter/productmetrics/dryrun_test.go | 103 ++++++++++++++++++ internal/cliconfig/productmetrics.go | 18 ++- internal/cliconfig/productmetrics_test.go | 48 +++++++- 13 files changed, 283 insertions(+), 13 deletions(-) create mode 100644 internal/adapter/productmetrics/dryrun.go create mode 100644 internal/adapter/productmetrics/dryrun_test.go diff --git a/cmd/mecak8s/flags.go b/cmd/mecak8s/flags.go index ff569fe57e..6fe7b537ba 100644 --- a/cmd/mecak8s/flags.go +++ b/cmd/mecak8s/flags.go @@ -301,6 +301,10 @@ type config struct { // productMetricsSet records whether --product-metrics was explicitly passed, // so ResolveProductMetricsEnabled can let CLI out-rank DO_NOT_TRACK/settings. productMetricsSet bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool } // stringList is a repeatable string flag.Value, preserving order across @@ -477,6 +481,8 @@ func parseFlags(argv []string) (config, error) { fs.BoolVar(&cfg.productMetrics, "product-metrics", true, "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") fs.Usage = func() { _, _ = fmt.Fprint(fs.Output(), "Usage: mecak8s [flags]\n\n") diff --git a/cmd/mecak8s/observability.go b/cmd/mecak8s/observability.go index e9b57f2df6..8244e4a934 100644 --- a/cmd/mecak8s/observability.go +++ b/cmd/mecak8s/observability.go @@ -87,9 +87,9 @@ func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) FlagValue: cfg.productMetrics, SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), }) - pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, + pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, cfg.productMetricsDryRun, productmetrics.BinaryMecak8s, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productmetrics.FeatureSnapshot{Mode: productmetrics.ModeK8s}) + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeK8s}, diag) if pmErr != nil { // Mirror the existing telemetry-setup-failure posture: a warning, never // a fatal error — product metrics are best-effort and must not block diff --git a/cmd/mecated/helpmeta.go b/cmd/mecated/helpmeta.go index 7d4dcfbe8f..8028975ccb 100644 --- a/cmd/mecated/helpmeta.go +++ b/cmd/mecated/helpmeta.go @@ -108,6 +108,7 @@ var flagMetaByFlag = map[string]flagMeta{ "goroutine-warn-threshold": {group: groupObservability, common: false, acp: acpExclude}, "goroutine-warn-interval": {group: groupObservability, common: false, acp: acpExclude}, "product-metrics": {group: groupObservability, common: true, acp: acpInclude}, + "product-metrics-dry-run": {group: groupObservability, common: false, acp: acpExclude}, // ── Driver connectivity (serve-only) ────────────────────────────────── "driver-auth-token": {group: groupDriver, common: false, acp: acpExclude}, diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index 21e36a6edf..76dd01955a 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -183,6 +183,10 @@ type config struct { // productMetrics reports anonymous product-adoption metrics to Stacklok. // OPT-OUT: ON by default. See the --product-metrics flag help text. productMetrics bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool // Runtime-introspection admin surface (loopback only, on the --metrics-addr // listener): pprof + expvar + a runtime/metrics snapshot + a FlightRecorder. @@ -1135,9 +1139,9 @@ func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), }) heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) - pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productMetricsSnapshot(cfg)) + productMetricsSnapshot(cfg), diag) if err != nil { slog.Warn("product metrics disabled: setup failed", "err", err) } @@ -1701,6 +1705,8 @@ func parseFlagsModeOut(mode commandMode, argv []string, out io.Writer) (*flag.Fl fs.BoolVar(&cfg.productMetrics, "product-metrics", true, "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") fs.IntVar(&cfg.mutexProfileFraction, "mutex-profile-fraction", 0, "runtime.SetMutexProfileFraction: report 1/N mutex contention events for /debug/pprof/mutex. 0 (default) disables it. Adds per-contention sampling overhead; enable only when investigating lock contention") fs.IntVar(&cfg.blockProfileRate, "block-profile-rate", 0, "runtime.SetBlockProfileRate in nanoseconds: sample one blocking event per N ns blocked for /debug/pprof/block. 0 (default) disables it. Adds per-block-event overhead; enable only when investigating blocking") diff --git a/cmd/mecatequi/flags.go b/cmd/mecatequi/flags.go index 2e39b740a6..fc64e76c71 100644 --- a/cmd/mecatequi/flags.go +++ b/cmd/mecatequi/flags.go @@ -168,6 +168,10 @@ type flags struct { // productMetricsSet records whether --product-metrics was explicitly passed, // so ResolveProductMetricsEnabled can let CLI out-rank DO_NOT_TRACK/settings. productMetricsSet bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool } // parseFlags turns argv into a flags value, resolving env-derived defaults and @@ -246,6 +250,8 @@ func parseFlags(argv []string) (flags, error) { fs.BoolVar(&f.productMetrics, "product-metrics", true, "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&f.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") fs.Usage = usageEpilogue(fs) diff --git a/cmd/mecatequi/observability.go b/cmd/mecatequi/observability.go index eedc188646..8507930525 100644 --- a/cmd/mecatequi/observability.go +++ b/cmd/mecatequi/observability.go @@ -72,9 +72,9 @@ func buildObservability(ctx context.Context, f flags, diag port.Diagnostics) (ob FlagValue: f.productMetrics, SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), }) - pm, pmErr := cliconfig.BuildProductMetrics(ctx, context.Background(), enabled, + pm, pmErr := cliconfig.BuildProductMetrics(ctx, context.Background(), enabled, f.productMetricsDryRun, productmetrics.BinaryMecatequi, buildinfo.BuildID, 0, /* single fire, short-lived */ - productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}) + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}, diag) if pmErr != nil { // Mirror the existing telemetry-setup-failure posture: a warning, never // a fatal error — product metrics are best-effort and must not block a diff --git a/cmd/mecatui/config.go b/cmd/mecatui/config.go index f649cb60f6..28eeddbd0c 100644 --- a/cmd/mecatui/config.go +++ b/cmd/mecatui/config.go @@ -166,6 +166,10 @@ type config struct { // noSteerFlagSet). productMetrics bool productMetricsFlagSet bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool // resumeID and resumeLatest select an existing owned main chat for static // startup adoption. They are shared by embedded and connect modes and mutually @@ -472,6 +476,8 @@ func parseTransportFlags(mode transportMode, out io.Writer, args []string, brows fs.BoolVar(&cfg.noSkills, "no-skills", false, "embedded server only: disable skill discovery (the Skill tool) entirely") fs.BoolVar(&cfg.productMetrics, "product-metrics", true, "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") fs.BoolVar(&cfg.perf, "perf", false, "embedded server only: expose the private perf-observability admin surface (/metrics, /debug/pprof, /debug/vars, /debug/flightrecorder) and wire domain metrics into the engine. OFF by default. Empty --perf-addr uses a per-instance UNIX socket. SECURITY: UNAUTHENTICATED — its output can embed prompt text/file paths/goroutine stacks") fs.StringVar(&cfg.perfAddr, "perf-addr", "", "embedded server only: explicit loopback host:port for the --perf admin surface (empty = private per-instance UNIX socket, or ephemeral 127.0.0.1 TCP with --perf-mcp). Use 127.0.0.1:0 for explicit ephemeral TCP. Non-loopback addresses are refused. Only consulted with --perf") diff --git a/cmd/mecatui/helpmeta.go b/cmd/mecatui/helpmeta.go index c62b094af5..c854962b88 100644 --- a/cmd/mecatui/helpmeta.go +++ b/cmd/mecatui/helpmeta.go @@ -165,6 +165,7 @@ var flagApplicabilityByFlag = map[string]flagApplicability{ "perf-goroutine-warn-threshold": {group: groupObservability, common: false, local: true, connect: false}, "perf-mcp": {group: groupObservability, common: false, local: true, connect: false}, "product-metrics": {group: groupObservability, common: true, local: true, connect: false}, + "product-metrics-dry-run": {group: groupObservability, common: false, local: true, connect: false}, // ── Info (meta-flags) ─────────────────────────────────────────────────── "help-all": {group: groupInfo, common: false, local: true, connect: true}, diff --git a/cmd/mecatui/main.go b/cmd/mecatui/main.go index 8a8dac51ff..a268acccfa 100644 --- a/cmd/mecatui/main.go +++ b/cmd/mecatui/main.go @@ -1036,9 +1036,9 @@ func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), }) heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) - pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, productmetrics.BinaryMecatui, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productMetricsSnapshot()) + productMetricsSnapshot(), diag) if err != nil { diag.Log(ctx, port.LevelWarn, "mecatui: product metrics disabled: setup failed", "err", err.Error()) } diff --git a/internal/adapter/productmetrics/dryrun.go b/internal/adapter/productmetrics/dryrun.go new file mode 100644 index 0000000000..a5841c6d0f --- /dev/null +++ b/internal/adapter/productmetrics/dryrun.go @@ -0,0 +1,85 @@ +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// DryRunRecorder implements the same two ports as Recorder (port.EventSink + +// port.ToolCallRecorder) but logs every would-be observation via an injected +// port.Diagnostics instead of exporting it over OTLP — the +// --product-metrics-dry-run audit path, so a skeptical operator can see +// exactly what this pipeline would have sent without trusting the docs. It +// logs ONLY the same bounded fields Recorder ever reads (event type, stop +// reason, token counts by kind, feature/provider/mode enum values) — never a +// tool name, session id, or free-text content, mirroring Recorder's own +// privacy discipline exactly. +type DryRunRecorder struct { + diag port.Diagnostics +} + +// Compile-time interface checks. +var ( + _ port.EventSink = (*DryRunRecorder)(nil) + _ port.ToolCallRecorder = (*DryRunRecorder)(nil) +) + +// NewDryRunRecorder builds a DryRunRecorder over the given Diagnostics sink. +func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder { + return &DryRunRecorder{diag: diag} +} + +// Emit logs the bounded event type (and, for EvResult, the stop reason and +// token counts by kind) — the exact same fields Recorder.Emit reads. +func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") + case session.EvResult: + d.emitResult(ctx, ev.Result) + case session.EvSubagentStart: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") + case session.EvTeamStart: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record team_used+1") + } +} + +func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload) { + if res == nil { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed", + "stop", string(session.StopNone)) + return + } + u := res.Usage + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed + tokens", + "stop", string(res.Stop), + "input_tokens", u.InputTokens, + "output_tokens", u.OutputTokens, + "cache_read_tokens", u.CacheReadTokens, + "cache_write_tokens", u.CacheWriteTokens, + "reasoning_tokens", u.ReasoningTokens) +} + +// ToolCall logs only that a call happened — no name, no session id, no +// content, no duration — matching Recorder.ToolCall's restraint exactly. +func (d *DryRunRecorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1") +} + +// Heartbeat logs the closed-enum feature/provider/mode signal, matching +// Recorder.Heartbeat's fields exactly. +func (d *DryRunRecorder) Heartbeat(snap FeatureSnapshot) { + enabled := make([]string, 0, 4) + for f, on := range snap.enabled() { + if on { + enabled = append(enabled, string(f)) + } + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record heartbeat", + "features_enabled", enabled, + "provider_family", string(snap.Provider), + "deployment_mode", string(snap.Mode)) +} diff --git a/internal/adapter/productmetrics/dryrun_test.go b/internal/adapter/productmetrics/dryrun_test.go new file mode 100644 index 0000000000..1df9009765 --- /dev/null +++ b/internal/adapter/productmetrics/dryrun_test.go @@ -0,0 +1,103 @@ +package productmetrics + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +type capturingDiag struct { + lines []string + args [][]any +} + +func (c *capturingDiag) Log(_ context.Context, _ port.Level, msg string, args ...any) { + c.lines = append(c.lines, msg) + c.args = append(c.args, args) +} +func (c *capturingDiag) With(...any) port.Diagnostics { return c } + +func TestDryRunRecorderLogsInsteadOfExporting(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "sensitive-name"}, session.ToolResult{Content: "sensitive-content"}, 0, time.Millisecond) + + if len(diag.lines) != 2 { + t.Fatalf("got %d logged lines, want 2: %v", len(diag.lines), diag.lines) + } + for i, line := range diag.lines { + if strings.Contains(line, "sensitive") { + t.Errorf("dry-run log line leaked sensitive content: %q", line) + } + for _, a := range diag.args[i] { + if s, ok := a.(string); ok && strings.Contains(s, "sensitive") { + t.Errorf("dry-run log args leaked sensitive content: %v", diag.args[i]) + } + } + } +} + +// TestDryRunRecorderImplementsPorts pins the compile-time interface guards +// (var _ port.EventSink = ...) via an explicit assignment, so a signature +// drift on either port fails this test with a clear message rather than only +// the package-level var block. +func TestDryRunRecorderImplementsPorts(_ *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + var _ port.EventSink = r + var _ port.ToolCallRecorder = r +} + +// TestDryRunRecorderResultNeverLeaksFreeText covers the EvResult branch (not +// exercised by the brief's original test) with a non-nil ResultPayload, +// asserting the log carries only the bounded stop/token fields and never the +// free-text Text/Error fields on ResultPayload. +func TestDryRunRecorderResultNeverLeaksFreeText(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopEndTurn, + Text: "sensitive final answer text", + Usage: session.Usage{ + InputTokens: 10, + OutputTokens: 20, + }, + }, + }) + + if len(diag.lines) != 1 { + t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) + } + for _, a := range diag.args[0] { + if s, ok := a.(string); ok && strings.Contains(s, "sensitive") { + t.Errorf("dry-run log args leaked ResultPayload.Text: %v", diag.args[0]) + } + } +} + +func TestDryRunRecorderHeartbeatLogsOnlyEnums(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Heartbeat(FeatureSnapshot{ + Memory: true, + Guardrails: false, + MCP: true, + Scheduling: false, + Provider: ProviderAnthropic, + Mode: ModeInteractive, + }) + + if len(diag.lines) != 1 { + t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) + } +} diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index e088336ba5..0d71fe716a 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -52,18 +52,34 @@ type ProductMetricsHandles struct { // for a single-fire-only short-lived process (mecatequi). heartbeatCtx is // cancelled by the caller on shutdown to stop the periodic ticker goroutine // this starts. +// +// When dryRun is true (and enabled is also true), this builds a +// productmetrics.DryRunRecorder over diag instead of the real OTLP +// pipeline — the --product-metrics-dry-run audit path: no install-id +// read/write, no real provider, no real heartbeat ticker. It fires exactly +// ONE representative Heartbeat call in its own goroutine (a dry run only +// needs to show one sample, not simulate the full cadence) and returns +// handles wrapping the DryRunRecorder as both Sink and ToolCallRecorder. func BuildProductMetrics( ctx, heartbeatCtx context.Context, - enabled bool, + enabled, dryRun bool, binary productmetrics.Binary, version string, heartbeatInterval time.Duration, snap productmetrics.FeatureSnapshot, + diag port.Diagnostics, ) (ProductMetricsHandles, error) { noop := func(context.Context) error { return nil } if !enabled { return ProductMetricsHandles{Shutdown: noop}, nil } + if dryRun { + rec := productmetrics.NewDryRunRecorder(diag) + go func() { + rec.Heartbeat(snap) + }() + return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil + } installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() if err != nil { diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go index 090b1ebfdf..57a5ef8de9 100644 --- a/internal/cliconfig/productmetrics_test.go +++ b/internal/cliconfig/productmetrics_test.go @@ -4,12 +4,13 @@ import ( "context" "testing" + "github.com/stacklok/mecatl/engine/port" "github.com/stacklok/mecatl/internal/adapter/productmetrics" ) func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { - h, err := BuildProductMetrics(context.Background(), context.Background(), false, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + h, err := BuildProductMetrics(context.Background(), context.Background(), false, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) if err != nil { t.Fatalf("BuildProductMetrics(enabled=false): %v", err) } @@ -28,9 +29,48 @@ func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { // bakedKey is empty in every non-release build/test — enabling must // surface the error rather than silently disabling, so a caller notices // its release build is missing the ldflag. - _, err := BuildProductMetrics(context.Background(), context.Background(), true, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) if err == nil { t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") } } + +// TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider proves the +// dry-run branch takes the DryRunRecorder short-circuit BEFORE the +// install-id/provider construction that requires a baked ingest key — so +// dryRun=true must succeed with no error even though the enabled-for-real +// case (above) fails closed with no baked key. +func TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), true, true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=true, dryRun=true): %v", err) + } + if h.Sink == nil || h.ToolCallRecorder == nil { + t.Errorf("dry-run handles must carry a non-nil Sink/ToolCallRecorder: %+v", h) + } + if h.Shutdown == nil { + t.Fatal("Shutdown must be non-nil in dry-run mode (a no-op)") + } + if err := h.Shutdown(context.Background()); err != nil { + t.Errorf("dry-run no-op Shutdown returned an error: %v", err) + } + if h.FirstRun { + t.Error("dry-run must never mint/read an install id, so FirstRun must stay false") + } +} + +// TestBuildProductMetricsDisabledDryRunStillNoop proves dryRun is inert when +// enabled is false — the disabled posture must stay byte-identical +// regardless of the dry-run flag's value. +func TestBuildProductMetricsDisabledDryRunStillNoop(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), false, true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=false, dryRun=true): %v", err) + } + if h.Sink != nil || h.ToolCallRecorder != nil { + t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder even with dryRun=true: %+v", h) + } +} From 9cbe99874fe26597cfc1ea1de786ce761a057b8a Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 01:14:33 -0400 Subject: [PATCH 20/47] docs: add ADR and user-docs for opt-out product/adoption metrics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds ADR 0317 recording the shipped product-metrics design (isolation architecture, the exact mecatl.adoption.* catalog, opt-out precedence, operator-tier-only settings gate, privacy guard test, dry-run mode, and why opt-out is defensible here), extends the existing observability.md user-docs page with a "Product / adoption metrics (opt-out)" section, and fills in ProductMetricsDisclosureNotice's doc-link placeholder. Also excludes docs/superpowers/ (the SDD workflow's committed plan/spec artifacts) from the matlatl docs corpus, matching its existing transient-artifact exclusions — required for `task docs --strict` to pass now that this feature's plan/spec files are the only content under that path. Co-Authored-By: Claude Sonnet 5 --- .matlatlignore | 9 + docs/adr/0317-product-metrics.md | 303 ++++++++++++++++++ docs/adr/README.md | 1 + internal/cliconfig/productmetrics.go | 2 +- .../building/what-you-get/observability.md | 16 + 5 files changed, 330 insertions(+), 1 deletion(-) create mode 100644 docs/adr/0317-product-metrics.md diff --git a/.matlatlignore b/.matlatlignore index 6d41e7fede..386b2e5e14 100644 --- a/.matlatlignore +++ b/.matlatlignore @@ -66,3 +66,12 @@ website/CLAUDE.md # Exclude both the committed reports and their ignored comparison copies. sdk/typescript/etc/*.api.md sdk/typescript/.api-extractor-temp/ + +# docs/superpowers/{plans,specs}/ are the SDD (spec-driven-development) workflow's +# committed historical record of a feature's implementation plan and design spec — +# a point-in-time process artifact analogous to review-*.md/handoff-*.md above, not +# navigational product documentation. They are intentionally standalone (no inbound +# or outbound doc links); the decisions they record land in a proper docs/adr/ entry +# instead, which DOES stay in the corpus. Left in, each new plan/spec adds a fresh +# orphan/unreachable pair that blocks `check --strict`. +docs/superpowers/ diff --git a/docs/adr/0317-product-metrics.md b/docs/adr/0317-product-metrics.md new file mode 100644 index 0000000000..2d0fde6d63 --- /dev/null +++ b/docs/adr/0317-product-metrics.md @@ -0,0 +1,303 @@ +# ADR 0317 — Product (adoption) metrics over OTLP + +- Status: Accepted +- Date: 2026-09-09 +- Scope: `internal/adapter/productmetrics` (new), `internal/cliconfig`, `internal/adapter/permconfig` (new `telemetry:` operator section), `cmd/mecated`, `cmd/mecatui`, `cmd/mecatequi`, `cmd/mecak8s` +- Supersedes: — +- Superseded by: — + +## Context + +mecatl had no visibility into community adoption: no install counts, no +feature-adoption signal, no aggregate usage depth. Stacklok's infra team +stood up a dedicated, internet-facing OTLP/HTTP metrics ingest at +`https://metrics.stacklok.com/v1/metrics` specifically for mecatl binaries +running on infrastructure Stacklok does not control (`stacklok/infra#5604`): +API-key-gated at the edge (`x-mecatl-metrics-key` header, stripped before the +collector) and server-side filtered to accept only metric names matching +`^mecatl\..*`. + +This is a genuinely different concern from mecatl's existing operator-facing +observability (ADR 0018/0045/0098, and the three-channel taxonomy in +ADR 0020: `port.Diagnostics`, `port.ToolCallRecorder`, `port.EventSink`→ +`internal/adapter/telemetry`). That pipeline exists so an *operator* can +observe *their own* deployment, pointed at *their own* collector. Product +metrics is Stacklok observing aggregate, anonymous community adoption across +every install — a different audience, a different destination, and a +different consent model. Reusing any part of the existing three-channel +pipeline as a transport for this data would be a category error: an +operator's own `--otlp-endpoint` must have zero effect on what does or +doesn't reach Stacklok, and enabling/disabling product metrics must have +zero effect on what the operator's own collector receives. The two cannot +be allowed to share a `MeterProvider`, a struct, or a destination, or that +guarantee becomes an implementation accident instead of a structural fact. + +`toolhive-core` (already an mecatl dependency) ships `telemetry/providers`: a +small, already-reviewed OTel SDK-wiring layer (`providers.NewCompositeProvider`) +that builds a `metric.MeterProvider` from an options struct (endpoint, +headers, service name/version, custom resource attributes) without ever +installing it as the process-global provider — the natural building block +for this pipeline's OTLP/HTTP exporter, so mecatl does not hand-roll a third +OTLP wiring implementation next to the two it already has +(`internal/adapter/telemetry/otlp.go`'s inline construction, and this one). + +## Decision + +### A fully independent adapter: `internal/adapter/productmetrics` + +Zero import relationship with `internal/adapter/telemetry`. It owns: + +- Its own `metric.MeterProvider` (`Provider`, wrapping + `toolhive-core/telemetry/providers.CompositeProvider`), built against a + **hardcoded** endpoint (`https://metrics.stacklok.com/v1/metrics`) and a + **hardcoded** header key baked into the binary at build time via + `-X …productmetrics.bakedKey=…` (`Taskfile.yml`'s `BUILD_LDFLAGS`) — neither + is operator-configurable, and there is exactly one place this data can go. + An empty baked key (every local/dev/CI-test build) makes `NewProvider` + refuse to construct at all: a non-release build can never accidentally + phone home. +- Its own `Recorder`, implementing `port.EventSink` (`Emit`) and + `port.ToolCallRecorder` (`ToolCall`) — the same two seams + `internal/adapter/telemetry` taps, but a type whose public API cannot + accept a bare free-text `string` anywhere (see "Privacy guard" below). +- Its own heartbeat ticker (`RunHeartbeat`): fires once immediately, then + every `DefaultHeartbeatInterval` (24h) for long-running processes + (`mecated`, `mecatui`, `mecak8s`); a single fire with no ticker + (`interval<=0`) plus flush-before-exit for the short-lived `mecatequi`, + mirroring the OTLP-push-with-flush precedent ADR 0098 already established + for that binary's shape. +- Its own install-identity file (`installid.go`). + +Composition combines the two independent sinks with a trivial fan-out helper +in `internal/cliconfig` (`BuildProductMetrics`, `TeeToolCallRecorder`) — the +existing Rule-of-Three home for cross-binary telemetry wiring per ADR 0098. +`internal/app` stays import-free of `productmetrics`, exactly as it is of +`telemetry` today. Each `cmd/*/main.go` builds its existing operator +telemetry pipeline unchanged, and — only when product metrics are enabled — +separately constructs a `productmetrics.Recorder` (or, under +`--product-metrics-dry-run`, a `DryRunRecorder`) and tees it in alongside via +`TeeToolCallRecorder` / `internal/adapter/telemetry.NewSink`'s existing +`EventSink` fan-out. + +An operator who disables their own OTLP export still has product metrics +flow (if enabled) to Stacklok; an operator who fully disables product +metrics has zero effect on their own OTLP/Prometheus pipeline. The two +cannot leak into each other because they share no struct, provider, +registry, or destination — only the same two read-only observation points +every consumer of those ports already receives independently, per +composition's existing fan-out discipline. + +### The shipped `mecatl.adoption.*` catalog + +All instrument names are namespaced under `mecatl.adoption.*` — passes the +collector's `^mecatl\..*` filter, and is visually/query-wise distinct from +the operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family, so nobody +looking at either series family can mistake one for the other. + +**Resource attributes** (set once per process via `productmetrics.Config`, +not per-metric labels): `service.name=mecatl`, `service.version`, +`mecatl.install.id` (a random v4 UUID), `mecatl.binary` (one of the closed +`Binary` set: `mecated`/`mecatui`/`mecatequi`/`mecak8s`). + +**Heartbeat** (`metrics.go`/`heartbeat.go` — on start, then every ~24h for +long-running processes; single fire for `mecatequi`): + +| Instrument | Kind | Attributes | Notes | +|---|---|---|---| +| `mecatl.adoption.heartbeat` | counter, +1 per fire | none | install/liveness signal | +| `mecatl.adoption.feature_enabled` | counter, +1 per enabled feature per heartbeat | `feature` | closed `Feature` set — see refinement below | +| `mecatl.adoption.provider_configured` | counter | `family` | closed `ProviderFamily`: `anthropic`/`openai`/`openrouter`/`other` — never a model id/alias | +| `mecatl.adoption.deployment_mode` | counter | `mode` | closed `DeploymentMode`: `interactive`/`headless`/`k8s` | + +**Coarse usage** (`metrics.go`/`toolcall.go` — derived from the +`port.EventSink`/`port.ToolCallRecorder` tap, exported on the provider's +normal periodic-reader cadence since these are cumulative counters): + +| Instrument | Kind | Attributes | Fires on | +|---|---|---|---| +| `mecatl.adoption.sessions_started` | counter | none | `EvSessionInit` | +| `mecatl.adoption.runs_completed` | counter | `stop` (reuses `session.StopReason`) | `EvResult` | +| `mecatl.adoption.tool_calls` | counter | none — **no tool/MCP-server name label at all** | every `ToolCallRecorder.ToolCall` | +| `mecatl.adoption.tokens` | counter | `kind` (`input`/`output`/`cache_read`/`cache_write`/`reasoning`) | `EvResult`'s `Usage` | +| `mecatl.adoption.subagent_used` | counter | none | `EvSubagentStart` | +| `mecatl.adoption.team_used` | counter | none | `EvTeamStart` | + +**Deliberate scope refinement from the original design spec.** The spec's +illustrative heartbeat catalog listed `teams`/`subagents`/`learning` as +feature-enabled flags alongside `memory`/`guardrails`/`mcp`/`scheduling`. +During implementation those three were dropped from the heartbeat: there is +no reliable per-CLI-flag boolean signal for "is Team/Subagent/learning +enabled" the way there is for a settings toggle like memory or guardrails — +those features are always *available*, not gated by a single config +boolean, so a `feature_enabled{feature="teams"}` counter would either always +fire (uninformative) or require inventing a proxy signal (dishonest). The +shipped `productmetrics.Feature` closed set is exactly four values: +`memory`, `guardrails`, `mcp`, `scheduling` (`config.go`). Team/Subagent +adoption is instead captured honestly via the coarse-usage event tap — +`mecatl.adoption.subagent_used`/`team_used` fire once per run the first time +that delegation family is actually invoked — which is a truer adoption +signal than a static capability flag. This ADR records the *shipped* set; +readers should treat the design spec (`docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md`) +as historical context, not the current catalog. + +Nothing here is free text, a session/run/model identifier, a tool or MCP +server name, a file path, a prompt, or an output. Every label value is drawn +from a closed Go-level enum already defined in `internal/adapter/productmetrics/config.go` +or reused from `engine/session` (`StopReason`). + +### Opt-out precedence and the operator-tier-only settings gate + +Product metrics are **enabled by default** (opt-out). `internal/cliconfig.ResolveProductMetricsEnabled` +(`productmetrics_config.go`) folds four inputs, highest precedence first: + +1. An explicit CLI flag: `--product-metrics=false` (all four binaries). +2. The `DO_NOT_TRACK` environment variable (any non-empty value) — the + cross-ecosystem convention (consoledonottrack.com), so the one env var + that already opts CI fleets and dev machines out of *other* tools' + telemetry covers mecatl too, with no mecatl-specific variable to + remember. (A dedicated `MECATL_PRODUCT_METRICS=0` was deliberately not + added on top of it — one standard signal beats two overlapping ones.) +3. `telemetry.productMetrics.enabled: false` in the **operator-tier** + settings file (`~/.config/mecatl/settings.yaml` + CLI-loaded equivalents) + — `permconfig.Resolver.OperatorProductMetricsEnabled()`. +4. Default: enabled. + +The settings toggle is **operator-tier only**, the same trust boundary as +`guardrails:`/`openrouter:` (AGENTS.md's existing operator-tier-only +precedent). A project-tier `.mecatl/settings.yaml` can neither enable nor +disable it for a user — parsed with the same WARN-and-ignore discipline as +the other operator-only subtrees (`internal/adapter/permconfig/resolve.go`'s +`captureTelemetry` + the project-tier ignore WARN). This direction matters +specifically because it is the *reverse* of most tighten-only project gates: +a project silently overriding a user's own telemetry opt-out (in *either* +direction — forcing it on, or forcing it off to hide activity from an +otherwise-informed operator) would itself be a trust violation, so the +whole subtree is simply inadmissible from a project file. + +### Privacy guard test discipline + +A reflect-based guard test (mirroring the existing `attrRole`/`attrStop` +bounded-label discipline elsewhere in the codebase, and the +`engine/port/diagnostics_imports_test.go` import-tripwire pattern) asserts +that `Recorder`'s entire public API accepts no bare free-text `string` +parameter — only bounded enum types (Go string-alias types with a small +closed value set) and counts/durations. This makes "no PII can flow through +this type" a property CI checks on every future change to the package, not +just a code-review norm that erodes over time. `DryRunRecorder` mirrors +`Recorder`'s restraint field-for-field (it logs the exact same bounded set +`Recorder` would have recorded, via `port.Diagnostics`, never more). + +Additional structural safeguards: exporter failures are silent to the app +(lazy-dial exporter, matching the existing OTLP exporter pattern — a dead +`metrics.stacklok.com` never blocks or slows a session); `Shutdown` is +bounded so a hung network path can never delay process exit; the +`MeterProvider` is never installed as the process-global provider (mirrors +`internal/adapter/telemetry`'s own discipline), so it structurally cannot +collide with, or be mistaken for, an operator's own OTel setup. + +### The dry-run audit mode + +`--product-metrics-dry-run` (all four binaries) builds a +`productmetrics.DryRunRecorder` in place of the real OTLP `Recorder` +(`BuildProductMetrics` in `internal/cliconfig/productmetrics.go`): no +install-id read/write, no real provider construction, no real heartbeat +ticker — it fires exactly one representative `Heartbeat` in its own +goroutine (a dry run only needs to demonstrate one sample, not simulate the +full ~24h cadence) and logs every would-be observation through the injected +`port.Diagnostics` instead of exporting it. It logs precisely the same +bounded fields the real `Recorder` would have read (event type, stop +reason, token counts by kind, feature/provider/mode enum values) — never +more. This exists so a skeptical operator can verify the "no PII, here is +exactly what leaves this process" claim by running the binary once, rather +than trusting this document. + +### Why opt-out is defensible here + +Opt-out telemetry is usually a trust liability. Four things together make it +defensible in this specific case, and all four are load-bearing — remove any +one and the balance shifts back toward requiring opt-in: + +1. **A visible, non-blocking first-run disclosure.** The first time a + binary is about to actually send product metrics (telemetry enabled, and + this install's telemetry-id file did not yet exist), it prints + `cliconfig.ProductMetricsDisclosureNotice` once to stderr — what is + collected, that it is on by default, and the exact three ways to turn it + off. It never blocks. An opt-out default with no visible disclosure is + the pattern that burns community trust; this is the whole of that + disclosure, and it is not optional or hidden in a man page. +2. **`DO_NOT_TRACK` support**, so the opt-out is not mecatl-specific + knowledge — anyone who already opts every other tool in their environment + out of telemetry gets mecatl covered for free, with zero new config to + learn. +3. **A reviewable, tested, closed catalog.** Every metric this pipeline can + ever emit is enumerated in this document and enforced by the privacy + guard test above — an operator (or a contributor reading this ADR) does + not have to trust a claim; they can read the closed `Feature`/ + `ProviderFamily`/`DeploymentMode`/token-kind enums and the guard test that + pins them, and know that is the entire surface, mechanically, not just as + of today. +4. **The dry-run self-verification path.** `--product-metrics-dry-run` lets + anyone confirm the catalog claim empirically on their own machine before + trusting `--product-metrics` for real, rather than trusting either the + docs or the code review that produced them. + +## Consequences + +- A new direct dependency surface: `github.com/stacklok/toolhive-core/telemetry/providers` + (already present at `v0.0.43`) is now also used by `internal/adapter/productmetrics`, + in addition to `internal/adapter/telemetry`'s own inline OTLP construction — + two independent OTLP wiring call sites in the same binary, by design, never + merged. +- A new operator-tier-only YAML subtree (`telemetry.productMetrics.enabled`) + and its own strict-decode schema (`permconfig.TelemetrySection`/ + `ProductMetricsSection`) — a project-tier `telemetry:` block is parsed and + then discarded with a WARN, never silently accepted. +- Four new CLI flags across four binaries (`--product-metrics`, + `--product-metrics-dry-run`), all defaulting to the byte-identical + disabled-pipeline posture when a non-release build carries no baked ingest + key (`bakedKey == ""` refuses `NewProvider` outright) — a local `go build`/ + `go test`/CI build can never phone home regardless of flag state. +- A new small persisted file per install + (`$XDG_STATE_HOME/mecatl/telemetry-id`, a bare random v4 UUID) — trivially + reset by deleting it, and carrying no machine or user information. +- ADR-0027 List-1 (resource inventory) is NOT extended: the heartbeat + ticker's lifetime matches the process (owned by the caller's + `heartbeatCtx`, cancelled on shutdown alongside the rest of composition's + shutdown sequence) and the `Provider`/`Recorder` pair holds no cross-run + state a restart would need to rehydrate — this pipeline is stateless + across restart by design, the same reasoning ADR 0098 used for its own + periodic-reader rows. +- Going forward, any NEW metric added to this package must (a) be added to + the catalog table above, (b) use only an existing or newly-defined closed + enum type (never a bare `string`), and (c) pass the privacy guard test + unchanged — the test is the enforcement mechanism, this ADR is the record. +- A per-run cost/dollar metric and any richer per-feature usage counters + (per-MCP-transport type, per-model-family latency) remain explicitly out + of scope, mirroring the deferred #192 cost-metric gap in the operator + pipeline — start narrow, revisit only if a concrete adoption question the + current catalog cannot answer comes up. + +## See also + +- [ADR 0098 — Telemetry for the headless binaries](./0098-headless-telemetry.md) + — the operator-facing OTLP/Prometheus pipeline and the + `internal/cliconfig` Rule-of-Three home this ADR's composition helper + (`BuildProductMetrics`) reuses structurally, but shares NO runtime state + with. Prior art deliberately not reused: this is a new, separate channel, + not a fourth use of the existing telemetry pipeline. +- [ADR 0020 — Diagnostics, audit, and the global-slog ban](./0020-diagnostics.md) + — the three-channel observability taxonomy (`port.Diagnostics`, + `port.ToolCallRecorder`, `port.EventSink`→`internal/adapter/telemetry`) + this decision deliberately does NOT add a fourth row to; product metrics + taps the same two ports (`EventSink`, `ToolCallRecorder`) every existing + consumer already receives independently, but through its own entirely + separate adapter, `MeterProvider`, and destination. +- [ADR 0018 — Performance & observability](./0018-perf-observability.md) +- `user-docs/building/what-you-get/observability.md` — the "Product / + adoption metrics (opt-out)" section added alongside the existing + operator-facing channels. +- `docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md` — the + original design spec this ADR records the shipped outcome of (see the + scope-refinement note above for the one deliberate deviation). +- `stacklok/infra#5604` — the dedicated public OTLP ingest this pipeline + reports to. diff --git a/docs/adr/README.md b/docs/adr/README.md index b62691aa5c..dd4939b7f8 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -189,6 +189,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) +- [0317 — Product (adoption) metrics over OTLP](./0317-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 0d71fe716a..d7b5866580 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -27,7 +27,7 @@ never a prompt, file path, tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass --product-metrics=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: -. +see docs/adr/0317-product-metrics.md. ` // ProductMetricsHandles bundles the handles a cmd main threads into its diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 6824766264..05ca4ee733 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -231,6 +231,22 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor --- +## Product / adoption metrics (opt-out) + +The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. + +**What's collected:** version, OS/arch, which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason), tool calls executed (no tool name), token counts by kind, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, tool name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0317](https://github.com/stacklok/mecatl/blob/main/docs/adr/0317-product-metrics.md). + +**It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: + +- `--product-metrics=false` on the command line (all four binaries). +- The `DO_NOT_TRACK` environment variable (any non-empty value) — the same convention other tools already respect. +- `telemetry.productMetrics.enabled: false` in your **operator-tier** `~/.config/mecatl/settings.yaml`. This setting is operator-tier only: a project repo's `.mecatl/settings.yaml` cannot change your telemetry choice in either direction. + +**Self-verify before trusting it.** `--product-metrics-dry-run` prints every observation this pipeline would have sent to stderr instead of exporting it, so you can check the "no PII" claim yourself rather than take the docs' word for it. + +--- + ## What's next To configure Mecatl for production, see the deployment guide for how to wire an OTLP collector, configure the admin listener, and set up session persistence with `jsonlstore`. From e5ff04ea87065a468001cd90d2deee44c0430d6d Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 08:11:24 -0400 Subject: [PATCH 21/47] fix(docs): renumber ADR 0317 to 0319 to resolve a merge collision main gained ADR 0317 (canonical-shell-command-tool) and 0318 while this branch was open; renumber the product-metrics ADR to the next free slot and fix every cross-reference (docs/adr/README.md, the disclosure notice string, user-docs). Caught by CI's TestRealADRNumberUniqueness. Co-Authored-By: Claude Sonnet 5 --- docs/adr/{0317-product-metrics.md => 0319-product-metrics.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/adr/{0317-product-metrics.md => 0319-product-metrics.md} (100%) diff --git a/docs/adr/0317-product-metrics.md b/docs/adr/0319-product-metrics.md similarity index 100% rename from docs/adr/0317-product-metrics.md rename to docs/adr/0319-product-metrics.md From 035f3d2060b43bcc32098d7b065e7e12345139ee Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 08:11:40 -0400 Subject: [PATCH 22/47] fix(docs): fix cross-references after the ADR renumbering to 0319 Update docs/adr/README.md's index entry, the disclosure-notice string, and the user-docs link to point at the renumbered ADR file. Co-Authored-By: Claude Sonnet 5 --- docs/adr/0319-product-metrics.md | 2 +- docs/adr/README.md | 2 +- internal/cliconfig/productmetrics.go | 2 +- user-docs/building/what-you-get/observability.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/0319-product-metrics.md b/docs/adr/0319-product-metrics.md index 2d0fde6d63..759586f076 100644 --- a/docs/adr/0319-product-metrics.md +++ b/docs/adr/0319-product-metrics.md @@ -1,4 +1,4 @@ -# ADR 0317 — Product (adoption) metrics over OTLP +# ADR 0319 — Product (adoption) metrics over OTLP - Status: Accepted - Date: 2026-09-09 diff --git a/docs/adr/README.md b/docs/adr/README.md index 655eaefada..eaaf774508 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -191,7 +191,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) -- [0317 — Product (adoption) metrics over OTLP](./0317-product-metrics.md) +- [0319 — Product (adoption) metrics over OTLP](./0319-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index d7b5866580..b610581de4 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -27,7 +27,7 @@ never a prompt, file path, tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass --product-metrics=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: -see docs/adr/0317-product-metrics.md. +see docs/adr/0319-product-metrics.md. ` // ProductMetricsHandles bundles the handles a cmd main threads into its diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 05ca4ee733..34622499cb 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -235,7 +235,7 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. -**What's collected:** version, OS/arch, which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason), tool calls executed (no tool name), token counts by kind, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, tool name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0317](https://github.com/stacklok/mecatl/blob/main/docs/adr/0317-product-metrics.md). +**What's collected:** version, OS/arch, which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason), tool calls executed (no tool name), token counts by kind, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, tool name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0319](https://github.com/stacklok/mecatl/blob/main/docs/adr/0319-product-metrics.md). **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: From e19e8f5d230d682f1bfb0c430ad15d792905cae8 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 09:32:42 -0400 Subject: [PATCH 23/47] fix(product-metrics): correct shutdown order, headless mode, dedup, DO_NOT_TRACK, dry-run heartbeat Address review findings from PR #1278: - Fix the mecated shutdown-defer order so cancelHeartbeat() runs before pm.Shutdown() (defers are LIFO), matching mecatui's correct order. - Derive productMetricsSnapshot's Mode from cfg.headless instead of hardcoding ModeInteractive, so headless mecated deployments report correctly. - Dedup subagent_used/team_used to one count per run (keyed by the loop-stamped RunID, cleared on EvResult), matching their documented "at least once per run" semantics instead of counting every EvSubagentStart/EvTeamStart. - Treat only a truthy DO_NOT_TRACK value as an opt-out ("0"/"false" no longer disable telemetry), per the consoledonottrack.com convention. - Run the dry-run pipeline's one-shot Heartbeat call synchronously instead of in an unawaited goroutine, so it can't be lost on a short-lived process like mecatequi. Co-Authored-By: Claude Sonnet 5 --- cmd/mecated/main.go | 8 +- cmd/mecated/productmetrics_test.go | 21 +++++ internal/adapter/productmetrics/metrics.go | 84 +++++++++++++++++-- .../adapter/productmetrics/metrics_test.go | 36 ++++++++ internal/cliconfig/productmetrics.go | 12 +-- internal/cliconfig/productmetrics_config.go | 16 +++- .../cliconfig/productmetrics_config_test.go | 3 + 7 files changed, 165 insertions(+), 15 deletions(-) create mode 100644 cmd/mecated/productmetrics_test.go diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index 76dd01955a..2a52f72ffb 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -927,7 +927,6 @@ func run(mode commandMode, remaining []string) error { // lint gate; the helper owns the resolve/build/disclosure branches and // logs its own failure, so run() only threads the resulting handles. pm, cancelHeartbeat, _ := setupProductMetrics(ctx, cfg, diag) - defer cancelHeartbeat() defer func() { shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() @@ -935,6 +934,7 @@ func run(mode commandMode, remaining []string) error { slog.Warn("product metrics shutdown", "err", serr) } }() + defer cancelHeartbeat() tracing := telemetry.NewTracing(otel.GetTracerProvider()) @@ -1091,6 +1091,10 @@ func setupObservability(ctx context.Context, cfg config, diag port.Diagnostics) // metrics heartbeat reports, from fields already resolved on cfg — never a // model id/alias, only whether each feature is configured at all. func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + mode := productmetrics.ModeInteractive + if cfg.headless { + mode = productmetrics.ModeHeadless + } provider := productmetrics.ProviderOther switch { case cfg.useOpenAI: @@ -1108,7 +1112,7 @@ func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, Scheduling: !cfg.noScheduler, Provider: provider, - Mode: productmetrics.ModeInteractive, + Mode: mode, } } diff --git a/cmd/mecated/productmetrics_test.go b/cmd/mecated/productmetrics_test.go new file mode 100644 index 0000000000..592039cc89 --- /dev/null +++ b/cmd/mecated/productmetrics_test.go @@ -0,0 +1,21 @@ +package main + +import ( + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// TestProductMetricsSnapshotModeReflectsHeadless pins that the deployment +// mode reported to the product-metrics heartbeat matches --headless — a +// headless mecated (autonomous/CI deployment) must report ModeHeadless, not +// the default ModeInteractive, or the adoption dashboard's headless/ +// interactive split is corrupted for every headless mecated server. +func TestProductMetricsSnapshotModeReflectsHeadless(t *testing.T) { + if got := productMetricsSnapshot(config{}).Mode; got != productmetrics.ModeInteractive { + t.Errorf("Mode = %q, want %q (default interactive)", got, productmetrics.ModeInteractive) + } + if got := productMetricsSnapshot(config{headless: true}).Mode; got != productmetrics.ModeHeadless { + t.Errorf("Mode = %q, want %q (--headless)", got, productmetrics.ModeHeadless) + } +} diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index f15b65dcd7..306183b74b 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -3,6 +3,7 @@ package productmetrics import ( "context" "fmt" + "sync" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/metric" @@ -41,6 +42,24 @@ type Recorder struct { tokens metric.Int64Counter subagentUsed metric.Int64Counter teamUsed metric.Int64Counter + + // runFamiliesUsed dedups subagentUsed/teamUsed to their documented + // "at least once per run" semantics: a run's first EvSubagentStart or + // EvTeamStart increments the counter, later ones in the SAME run (e.g. a + // fan-out of concurrent Subagent calls) do not. Keyed by the loop-stamped + // session.Event.RunID (opaque, not a session id) — never a tool name, + // session id, or free-text field, preserving this package's no-PII + // invariant. Bounded to concurrently-live runs: each run's entry is + // cleared on its EvResult. + mu sync.Mutex + runFamiliesUsed map[string]usedFamilies +} + +// usedFamilies tracks, per live run, which delegation families have already +// been counted at least once. +type usedFamilies struct { + subagent bool + team bool } // Compile-time interface checks. @@ -54,7 +73,7 @@ var ( // is fallible. func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { meter := mp.Meter(meterName) - r := &Recorder{} + r := &Recorder{runFamiliesUsed: make(map[string]usedFamilies)} var err error if r.heartbeat, err = meter.Int64Counter("mecatl.adoption.heartbeat", @@ -102,20 +121,73 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { } // Emit derives coarse, bounded counts from a single domain Event. It reads -// ONLY ev.Type, ev.Result.Stop, and ev.Result.Usage — never a session id, -// model id/alias, tool name, or any free-text field (ev.Result.Text/Error -// are never touched). +// ONLY ev.Type, ev.RunID, ev.Result.Stop, and ev.Result.Usage — never a +// session id, model id/alias, tool name, or any free-text field +// (ev.Result.Text/Error are never touched). ev.RunID is an opaque per-run +// correlation id (ADR 0249), not a session id, and is used ONLY to dedup +// subagentUsed/teamUsed to one count per run (see firstInRun); it never +// becomes an attribute value. func (r *Recorder) Emit(ctx context.Context, ev session.Event) { switch ev.Type { case session.EvSessionInit: r.sessionsStarted.Add(ctx, 1) case session.EvResult: r.recordResult(ctx, ev.Result) + r.clearRun(ev.RunID) case session.EvSubagentStart: - r.subagentUsed.Add(ctx, 1) + if r.firstInRun(ev.RunID, familySubagent) { + r.subagentUsed.Add(ctx, 1) + } case session.EvTeamStart: - r.teamUsed.Add(ctx, 1) + if r.firstInRun(ev.RunID, familyTeam) { + r.teamUsed.Add(ctx, 1) + } + } +} + +// delegationFamily is the closed set of families dedup'd per run. +type delegationFamily int + +const ( + familySubagent delegationFamily = iota + familyTeam +) + +// firstInRun reports whether this is the first time, within the run +// identified by runID, that family has been observed — marking it seen as a +// side effect. An empty runID (no run context to dedup against) always +// counts, matching the pre-dedup behavior. +func (r *Recorder) firstInRun(runID string, family delegationFamily) bool { + if runID == "" { + return true + } + r.mu.Lock() + defer r.mu.Unlock() + u := r.runFamiliesUsed[runID] + var seen *bool + switch family { + case familySubagent: + seen = &u.subagent + case familyTeam: + seen = &u.team + } + if *seen { + return false + } + *seen = true + r.runFamiliesUsed[runID] = u + return true +} + +// clearRun drops the run's dedup entry once it has ended (EvResult), so +// runFamiliesUsed stays bounded to concurrently-live runs. +func (r *Recorder) clearRun(runID string) { + if runID == "" { + return } + r.mu.Lock() + delete(r.runFamiliesUsed, runID) + r.mu.Unlock() } func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload) { diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go index e32320f245..323dc86872 100644 --- a/internal/adapter/productmetrics/metrics_test.go +++ b/internal/adapter/productmetrics/metrics_test.go @@ -137,3 +137,39 @@ func TestRecorderEmitSubagentAndTeamUsed(t *testing.T) { t.Errorf("team_used = %d, want 1", got) } } + +// TestRecorderSubagentUsedCountsOncePerRun pins the documented "at least +// once per run" semantics: a run fanning out several concurrent Subagent +// calls (explicitly supported, e.g. the child concurrency gate) must count +// once, not once per EvSubagentStart. +func TestRecorderSubagentUsedCountsOncePerRun(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + + if got := sumValue(t, collect(t, reader)["mecatl.adoption.subagent_used"]); got != 1 { + t.Errorf("subagent_used = %d, want 1 (deduped within one run)", got) + } +} + +// TestRecorderSubagentUsedCountsEachDistinctRun proves the dedup is scoped +// to a run, not global: two separate runs each using Subagent count twice, +// and a run's dedup state is dropped on EvResult so a later run with the +// same RunID (unlikely in practice, but bounds correctness) still counts. +func TestRecorderSubagentUsedCountsEachDistinctRun(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-2"}) + + if got := sumValue(t, collect(t, reader)["mecatl.adoption.subagent_used"]); got != 2 { + t.Errorf("subagent_used = %d, want 2 (two distinct runs)", got) + } + + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + + if got := sumValue(t, collect(t, reader)["mecatl.adoption.subagent_used"]); got != 3 { + t.Errorf("subagent_used = %d, want 3 (run-1's dedup entry cleared on its EvResult)", got) + } +} diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index b610581de4..6947bd4e63 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -57,9 +57,11 @@ type ProductMetricsHandles struct { // productmetrics.DryRunRecorder over diag instead of the real OTLP // pipeline — the --product-metrics-dry-run audit path: no install-id // read/write, no real provider, no real heartbeat ticker. It fires exactly -// ONE representative Heartbeat call in its own goroutine (a dry run only -// needs to show one sample, not simulate the full cadence) and returns -// handles wrapping the DryRunRecorder as both Sink and ToolCallRecorder. +// ONE representative Heartbeat call SYNCHRONOUSLY, before returning (a dry +// run only needs to show one sample, not simulate the full cadence, and a +// short-lived process like mecatequi can exit before an unawaited goroutine +// ever runs) and returns handles wrapping the DryRunRecorder as both Sink +// and ToolCallRecorder. func BuildProductMetrics( ctx, heartbeatCtx context.Context, enabled, dryRun bool, @@ -75,9 +77,7 @@ func BuildProductMetrics( } if dryRun { rec := productmetrics.NewDryRunRecorder(diag) - go func() { - rec.Heartbeat(snap) - }() + rec.Heartbeat(snap) return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil } diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go index fcb2c457fa..11f5a0ac66 100644 --- a/internal/cliconfig/productmetrics_config.go +++ b/internal/cliconfig/productmetrics_config.go @@ -4,12 +4,26 @@ package cliconfig import ( "os" + "strings" "time" "github.com/stacklok/mecatl/engine/port" "github.com/stacklok/mecatl/engine/session" ) +// doNotTrackOptOut reports whether a DO_NOT_TRACK env value means "opt out", +// per the consoledonottrack.com convention: unset/empty and the conventional +// "off" spellings ("0", "false", case-insensitive) are NOT an opt-out; any +// other value is. +func doNotTrackOptOut(v string) bool { + switch strings.ToLower(v) { + case "", "0", "false": + return false + default: + return true + } +} + // ProductMetricsPrecedence carries the opt-out inputs // ResolveProductMetricsEnabled folds, highest precedence first: an explicit // CLI flag, then the DO_NOT_TRACK env var convention (consoledonottrack.com), @@ -38,7 +52,7 @@ func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { if getenv == nil { getenv = os.Getenv } - if getenv("DO_NOT_TRACK") != "" { + if doNotTrackOptOut(getenv("DO_NOT_TRACK")) { return false } if p.SettingsEnabled != nil { diff --git a/internal/cliconfig/productmetrics_config_test.go b/internal/cliconfig/productmetrics_config_test.go index a76878ccd2..3d0f753f21 100644 --- a/internal/cliconfig/productmetrics_config_test.go +++ b/internal/cliconfig/productmetrics_config_test.go @@ -24,6 +24,9 @@ func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { {"DO_NOT_TRACK disables when no flag", ProductMetricsPrecedence{Getenv: getenvSet, SettingsEnabled: boolPtr(true)}, false}, {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: boolPtr(false)}, false}, {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: nil}, true}, + {"DO_NOT_TRACK=0 is not an opt-out", ProductMetricsPrecedence{Getenv: func(string) string { return "0" }, SettingsEnabled: boolPtr(true)}, true}, + {"DO_NOT_TRACK=false is not an opt-out", ProductMetricsPrecedence{Getenv: func(string) string { return "false" }, SettingsEnabled: boolPtr(true)}, true}, + {"DO_NOT_TRACK=true disables", ProductMetricsPrecedence{Getenv: func(string) string { return "true" }, SettingsEnabled: boolPtr(true)}, false}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { From 080398f9ac46306db9b5da436a0c5319ab4bac0b Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 10:03:09 -0400 Subject: [PATCH 24/47] refactor(productmetrics): rename the mecatl.adoption.* namespace to mecatl.product.* MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mecatl.usage.* was considered and rejected: it would read ambiguously next to resource-usage metrics (CPU/RSS) in the same dashboard. mecatl.product.* is unambiguous and matches the feature's own name throughout the ADR/docs. Kept flat (no further category level under product.*) — ten instruments doesn't earn the added structure yet. Co-Authored-By: Claude Sonnet 5 --- docs/adr/0319-product-metrics.md | 26 +++++++++---------- .../adapter/productmetrics/bounded_test.go | 8 +++--- .../adapter/productmetrics/heartbeat_test.go | 10 +++---- internal/adapter/productmetrics/metrics.go | 20 +++++++------- .../adapter/productmetrics/metrics_test.go | 18 ++++++------- .../adapter/productmetrics/provider_test.go | 2 +- .../adapter/productmetrics/toolcall_test.go | 2 +- 7 files changed, 43 insertions(+), 43 deletions(-) diff --git a/docs/adr/0319-product-metrics.md b/docs/adr/0319-product-metrics.md index 759586f076..85c4ee413a 100644 --- a/docs/adr/0319-product-metrics.md +++ b/docs/adr/0319-product-metrics.md @@ -87,9 +87,9 @@ registry, or destination — only the same two read-only observation points every consumer of those ports already receives independently, per composition's existing fan-out discipline. -### The shipped `mecatl.adoption.*` catalog +### The shipped `mecatl.product.*` catalog -All instrument names are namespaced under `mecatl.adoption.*` — passes the +All instrument names are namespaced under `mecatl.product.*` — passes the collector's `^mecatl\..*` filter, and is visually/query-wise distinct from the operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family, so nobody looking at either series family can mistake one for the other. @@ -104,10 +104,10 @@ long-running processes; single fire for `mecatequi`): | Instrument | Kind | Attributes | Notes | |---|---|---|---| -| `mecatl.adoption.heartbeat` | counter, +1 per fire | none | install/liveness signal | -| `mecatl.adoption.feature_enabled` | counter, +1 per enabled feature per heartbeat | `feature` | closed `Feature` set — see refinement below | -| `mecatl.adoption.provider_configured` | counter | `family` | closed `ProviderFamily`: `anthropic`/`openai`/`openrouter`/`other` — never a model id/alias | -| `mecatl.adoption.deployment_mode` | counter | `mode` | closed `DeploymentMode`: `interactive`/`headless`/`k8s` | +| `mecatl.product.heartbeat` | counter, +1 per fire | none | install/liveness signal | +| `mecatl.product.feature_enabled` | counter, +1 per enabled feature per heartbeat | `feature` | closed `Feature` set — see refinement below | +| `mecatl.product.provider_configured` | counter | `family` | closed `ProviderFamily`: `anthropic`/`openai`/`openrouter`/`other` — never a model id/alias | +| `mecatl.product.deployment_mode` | counter | `mode` | closed `DeploymentMode`: `interactive`/`headless`/`k8s` | **Coarse usage** (`metrics.go`/`toolcall.go` — derived from the `port.EventSink`/`port.ToolCallRecorder` tap, exported on the provider's @@ -115,12 +115,12 @@ normal periodic-reader cadence since these are cumulative counters): | Instrument | Kind | Attributes | Fires on | |---|---|---|---| -| `mecatl.adoption.sessions_started` | counter | none | `EvSessionInit` | -| `mecatl.adoption.runs_completed` | counter | `stop` (reuses `session.StopReason`) | `EvResult` | -| `mecatl.adoption.tool_calls` | counter | none — **no tool/MCP-server name label at all** | every `ToolCallRecorder.ToolCall` | -| `mecatl.adoption.tokens` | counter | `kind` (`input`/`output`/`cache_read`/`cache_write`/`reasoning`) | `EvResult`'s `Usage` | -| `mecatl.adoption.subagent_used` | counter | none | `EvSubagentStart` | -| `mecatl.adoption.team_used` | counter | none | `EvTeamStart` | +| `mecatl.product.sessions_started` | counter | none | `EvSessionInit` | +| `mecatl.product.runs_completed` | counter | `stop` (reuses `session.StopReason`) | `EvResult` | +| `mecatl.product.tool_calls` | counter | none — **no tool/MCP-server name label at all** | every `ToolCallRecorder.ToolCall` | +| `mecatl.product.tokens` | counter | `kind` (`input`/`output`/`cache_read`/`cache_write`/`reasoning`) | `EvResult`'s `Usage` | +| `mecatl.product.subagent_used` | counter | none | `EvSubagentStart` | +| `mecatl.product.team_used` | counter | none | `EvTeamStart` | **Deliberate scope refinement from the original design spec.** The spec's illustrative heartbeat catalog listed `teams`/`subagents`/`learning` as @@ -134,7 +134,7 @@ fire (uninformative) or require inventing a proxy signal (dishonest). The shipped `productmetrics.Feature` closed set is exactly four values: `memory`, `guardrails`, `mcp`, `scheduling` (`config.go`). Team/Subagent adoption is instead captured honestly via the coarse-usage event tap — -`mecatl.adoption.subagent_used`/`team_used` fire once per run the first time +`mecatl.product.subagent_used`/`team_used` fire once per run the first time that delegation family is actually invoked — which is a truer adoption signal than a static capability flag. This ADR records the *shipped* set; readers should treat the design spec (`docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md`) diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go index 68a013906d..f226f2f6c7 100644 --- a/internal/adapter/productmetrics/bounded_test.go +++ b/internal/adapter/productmetrics/bounded_test.go @@ -70,7 +70,7 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T totalMetrics += len(sm.Metrics) } if totalMetrics < 10 { - t.Fatalf("collected only %d metrics, want at least 10 (the full mecatl.adoption.* instrument set) — the walk below would otherwise pass vacuously", totalMetrics) + t.Fatalf("collected only %d metrics, want at least 10 (the full mecatl.product.* instrument set) — the walk below would otherwise pass vacuously", totalMetrics) } for _, sm := range rm.ScopeMetrics { @@ -91,12 +91,12 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T assertNoSensitiveSubstring(t, kv.Value.AsString()) } } - // mecatl.adoption.tool_calls carries NO attributes at all — the + // mecatl.product.tool_calls carries NO attributes at all — the // strongest form of "no tool identity ever attaches." - if md.Name == "mecatl.adoption.tool_calls" { + if md.Name == "mecatl.product.tool_calls" { for _, dp := range sum.DataPoints { if dp.Attributes.Len() != 0 { - t.Errorf("mecatl.adoption.tool_calls data point carries %d attributes, want 0: %v", + t.Errorf("mecatl.product.tool_calls data point carries %d attributes, want 0: %v", dp.Attributes.Len(), dp.Attributes) } } diff --git a/internal/adapter/productmetrics/heartbeat_test.go b/internal/adapter/productmetrics/heartbeat_test.go index b83eb9cb68..56880b2e5c 100644 --- a/internal/adapter/productmetrics/heartbeat_test.go +++ b/internal/adapter/productmetrics/heartbeat_test.go @@ -14,10 +14,10 @@ func TestRecorderHeartbeatRecordsClosedLabelsOnly(t *testing.T) { }) collected := collect(t, reader) - if got := sumValue(t, collected["mecatl.adoption.heartbeat"]); got != 1 { + if got := sumValue(t, collected["mecatl.product.heartbeat"]); got != 1 { t.Errorf("heartbeat = %d, want 1", got) } - featureAgg := collected["mecatl.adoption.feature_enabled"] + featureAgg := collected["mecatl.product.feature_enabled"] if got := sumPoint(t, featureAgg, "feature", "memory"); got != 1 { t.Errorf("feature_enabled{feature=memory} = %d, want 1", got) } @@ -27,10 +27,10 @@ func TestRecorderHeartbeatRecordsClosedLabelsOnly(t *testing.T) { // guardrails/scheduling were false in the snapshot: TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent // (Task 6) is the exhaustive "no other data point" check; this test // only asserts the enabled ones are present with the right value. - if got := sumPoint(t, collected["mecatl.adoption.provider_configured"], "family", "anthropic"); got != 1 { + if got := sumPoint(t, collected["mecatl.product.provider_configured"], "family", "anthropic"); got != 1 { t.Errorf("provider_configured{family=anthropic} = %d, want 1", got) } - if got := sumPoint(t, collected["mecatl.adoption.deployment_mode"], "mode", "interactive"); got != 1 { + if got := sumPoint(t, collected["mecatl.product.deployment_mode"], "mode", "interactive"); got != 1 { t.Errorf("deployment_mode{mode=interactive} = %d, want 1", got) } } @@ -42,7 +42,7 @@ func TestRunHeartbeatFiresImmediatelyThenStopsOnCtxDone(t *testing.T) { RunHeartbeat(ctx, r, time.Hour, FeatureSnapshot{Mode: ModeHeadless}) - if got := sumValue(t, collect(t, reader)["mecatl.adoption.heartbeat"]); got != 1 { + if got := sumValue(t, collect(t, reader)["mecatl.product.heartbeat"]); got != 1 { t.Errorf("heartbeat = %d, want exactly 1 (immediate fire only)", got) } } diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index 306183b74b..fb8dd39b2d 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -76,44 +76,44 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { r := &Recorder{runFamiliesUsed: make(map[string]usedFamilies)} var err error - if r.heartbeat, err = meter.Int64Counter("mecatl.adoption.heartbeat", + if r.heartbeat, err = meter.Int64Counter("mecatl.product.heartbeat", metric.WithDescription("Process liveness heartbeat.")); err != nil { return nil, fmt.Errorf("productmetrics: heartbeat counter: %w", err) } - if r.featureEnabled, err = meter.Int64Counter("mecatl.adoption.feature_enabled", + if r.featureEnabled, err = meter.Int64Counter("mecatl.product.feature_enabled", metric.WithDescription("Major feature enabled, by closed feature name, per heartbeat.")); err != nil { return nil, fmt.Errorf("productmetrics: feature_enabled counter: %w", err) } - if r.providerConfig, err = meter.Int64Counter("mecatl.adoption.provider_configured", + if r.providerConfig, err = meter.Int64Counter("mecatl.product.provider_configured", metric.WithDescription("Configured LLM provider family, by closed family name, per heartbeat.")); err != nil { return nil, fmt.Errorf("productmetrics: provider_configured counter: %w", err) } - if r.deploymentMode, err = meter.Int64Counter("mecatl.adoption.deployment_mode", + if r.deploymentMode, err = meter.Int64Counter("mecatl.product.deployment_mode", metric.WithDescription("Process deployment mode, by closed mode name, per heartbeat.")); err != nil { return nil, fmt.Errorf("productmetrics: deployment_mode counter: %w", err) } - if r.sessionsStarted, err = meter.Int64Counter("mecatl.adoption.sessions_started", + if r.sessionsStarted, err = meter.Int64Counter("mecatl.product.sessions_started", metric.WithDescription("Total sessions started.")); err != nil { return nil, fmt.Errorf("productmetrics: sessions_started counter: %w", err) } - if r.runsCompleted, err = meter.Int64Counter("mecatl.adoption.runs_completed", + if r.runsCompleted, err = meter.Int64Counter("mecatl.product.runs_completed", metric.WithDescription("Total runs completed, by bounded stop reason.")); err != nil { return nil, fmt.Errorf("productmetrics: runs_completed counter: %w", err) } - if r.toolCalls, err = meter.Int64Counter("mecatl.adoption.tool_calls", + if r.toolCalls, err = meter.Int64Counter("mecatl.product.tool_calls", metric.WithDescription("Total tool calls executed (no tool identity attached).")); err != nil { return nil, fmt.Errorf("productmetrics: tool_calls counter: %w", err) } - if r.tokens, err = meter.Int64Counter("mecatl.adoption.tokens", + if r.tokens, err = meter.Int64Counter("mecatl.product.tokens", metric.WithDescription("Total tokens accounted, by bounded kind."), metric.WithUnit("{token}")); err != nil { return nil, fmt.Errorf("productmetrics: tokens counter: %w", err) } - if r.subagentUsed, err = meter.Int64Counter("mecatl.adoption.subagent_used", + if r.subagentUsed, err = meter.Int64Counter("mecatl.product.subagent_used", metric.WithDescription("Runs that used the Subagent delegation family at least once.")); err != nil { return nil, fmt.Errorf("productmetrics: subagent_used counter: %w", err) } - if r.teamUsed, err = meter.Int64Counter("mecatl.adoption.team_used", + if r.teamUsed, err = meter.Int64Counter("mecatl.product.team_used", metric.WithDescription("Runs that used the Team delegation family at least once.")); err != nil { return nil, fmt.Errorf("productmetrics: team_used counter: %w", err) } diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go index 323dc86872..ffad2aa238 100644 --- a/internal/adapter/productmetrics/metrics_test.go +++ b/internal/adapter/productmetrics/metrics_test.go @@ -70,9 +70,9 @@ func TestRecorderEmitSessionsStarted(t *testing.T) { r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) - agg, ok := collect(t, reader)["mecatl.adoption.sessions_started"] + agg, ok := collect(t, reader)["mecatl.product.sessions_started"] if !ok { - t.Fatal("mecatl.adoption.sessions_started missing") + t.Fatal("mecatl.product.sessions_started missing") } if got := sumValue(t, agg); got != 2 { t.Errorf("sessions_started = %d, want 2", got) @@ -90,7 +90,7 @@ func TestRecorderEmitRunsCompletedByStopReason(t *testing.T) { Result: &session.ResultPayload{Stop: session.StopError}, }) - agg := collect(t, reader)["mecatl.adoption.runs_completed"] + agg := collect(t, reader)["mecatl.product.runs_completed"] if got := sumPoint(t, agg, "stop", "end_turn"); got != 1 { t.Errorf("runs_completed{stop=end_turn} = %d, want 1", got) } @@ -115,7 +115,7 @@ func TestRecorderEmitTokensByKind(t *testing.T) { }, }) - agg := collect(t, reader)["mecatl.adoption.tokens"] + agg := collect(t, reader)["mecatl.product.tokens"] cases := map[string]int64{"input": 100, "output": 50, "cache_read": 20, "cache_write": 5, "reasoning": 10} for kind, want := range cases { if got := sumPoint(t, agg, "kind", kind); got != want { @@ -130,10 +130,10 @@ func TestRecorderEmitSubagentAndTeamUsed(t *testing.T) { r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) collected := collect(t, reader) - if got := sumValue(t, collected["mecatl.adoption.subagent_used"]); got != 1 { + if got := sumValue(t, collected["mecatl.product.subagent_used"]); got != 1 { t.Errorf("subagent_used = %d, want 1", got) } - if got := sumValue(t, collected["mecatl.adoption.team_used"]); got != 1 { + if got := sumValue(t, collected["mecatl.product.team_used"]); got != 1 { t.Errorf("team_used = %d, want 1", got) } } @@ -148,7 +148,7 @@ func TestRecorderSubagentUsedCountsOncePerRun(t *testing.T) { r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) - if got := sumValue(t, collect(t, reader)["mecatl.adoption.subagent_used"]); got != 1 { + if got := sumValue(t, collect(t, reader)["mecatl.product.subagent_used"]); got != 1 { t.Errorf("subagent_used = %d, want 1 (deduped within one run)", got) } } @@ -162,14 +162,14 @@ func TestRecorderSubagentUsedCountsEachDistinctRun(t *testing.T) { r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-2"}) - if got := sumValue(t, collect(t, reader)["mecatl.adoption.subagent_used"]); got != 2 { + if got := sumValue(t, collect(t, reader)["mecatl.product.subagent_used"]); got != 2 { t.Errorf("subagent_used = %d, want 2 (two distinct runs)", got) } r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) - if got := sumValue(t, collect(t, reader)["mecatl.adoption.subagent_used"]); got != 3 { + if got := sumValue(t, collect(t, reader)["mecatl.product.subagent_used"]); got != 3 { t.Errorf("subagent_used = %d, want 3 (run-1's dedup entry cleared on its EvResult)", got) } } diff --git a/internal/adapter/productmetrics/provider_test.go b/internal/adapter/productmetrics/provider_test.go index 71777b2abd..62c736f335 100644 --- a/internal/adapter/productmetrics/provider_test.go +++ b/internal/adapter/productmetrics/provider_test.go @@ -45,7 +45,7 @@ func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { defer p.Shutdown(context.Background()) meter := p.Meter().Meter("test") - counter, cerr := meter.Int64Counter("mecatl.adoption.test") + counter, cerr := meter.Int64Counter("mecatl.product.test") if cerr != nil { t.Fatalf("Int64Counter: %v", cerr) } diff --git a/internal/adapter/productmetrics/toolcall_test.go b/internal/adapter/productmetrics/toolcall_test.go index 6f6d996470..8a96fb2362 100644 --- a/internal/adapter/productmetrics/toolcall_test.go +++ b/internal/adapter/productmetrics/toolcall_test.go @@ -17,7 +17,7 @@ func TestRecorderToolCallCountsWithoutIdentity(t *testing.T) { ) r.ToolCall(session.SessionID("other"), session.ToolCall{Name: "another_tool"}, session.ToolResult{}, 0, 0) - agg := collect(t, reader)["mecatl.adoption.tool_calls"] + agg := collect(t, reader)["mecatl.product.tool_calls"] if got := sumValue(t, agg); got != 2 { t.Errorf("tool_calls = %d, want 2", got) } From 9bb3cff99804d23de26550e510e6e2e9d98b399d Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Wed, 9 Sep 2026 13:31:17 -0400 Subject: [PATCH 25/47] fix(productmetrics): drop the per-install id from exported metrics This pipeline's destination is a Prometheus-remote-write backend (stacklok/infra#5604), where every OTel resource attribute becomes a permanent label on EVERY instrument's time series. Attaching a random per-install UUID there would multiply active series by (installs x instrument count) with no bound as adoption grows -- an unbounded-cardinality cost for a precision (exact unique-install counts) the design never actually required. Remove Config.InstallID and the mecatl.install.id resource attribute entirely. installid.go's local file still exists to detect first-run for the disclosure notice, but its UUID value is now discarded rather than threaded to NewProvider. Unique-install counts become an approximation from heartbeat volume/cadence instead of an exact count. Also fixes two stale doc references caught while updating: the DO_NOT_TRACK docs said "any non-empty value" (already contradicted by an earlier fix that excludes "0"/"false"), and consoledonottrack.com has been domain-squatted since this was written -- both now point at the live donottrack.sh convention text. Co-Authored-By: Claude Sonnet 5 --- docs/adr/0319-product-metrics.md | 35 +++++++++++++------ internal/adapter/productmetrics/config.go | 2 -- internal/adapter/productmetrics/installid.go | 13 +++++-- internal/adapter/productmetrics/provider.go | 13 +++++-- .../adapter/productmetrics/provider_test.go | 5 ++- internal/cliconfig/productmetrics.go | 12 ++++--- internal/cliconfig/productmetrics_config.go | 4 +-- .../building/what-you-get/observability.md | 2 +- 8 files changed, 59 insertions(+), 27 deletions(-) diff --git a/docs/adr/0319-product-metrics.md b/docs/adr/0319-product-metrics.md index 85c4ee413a..701b2e269e 100644 --- a/docs/adr/0319-product-metrics.md +++ b/docs/adr/0319-product-metrics.md @@ -66,7 +66,8 @@ Zero import relationship with `internal/adapter/telemetry`. It owns: (`interval<=0`) plus flush-before-exit for the short-lived `mecatequi`, mirroring the OTLP-push-with-flush precedent ADR 0098 already established for that binary's shape. -- Its own install-identity file (`installid.go`). +- Its own local install-identity file (`installid.go`) — a first-run marker + only, deliberately never exported (see the cardinality note below). Composition combines the two independent sinks with a trivial fan-out helper in `internal/cliconfig` (`BuildProductMetrics`, `TeeToolCallRecorder`) — the @@ -96,8 +97,20 @@ looking at either series family can mistake one for the other. **Resource attributes** (set once per process via `productmetrics.Config`, not per-metric labels): `service.name=mecatl`, `service.version`, -`mecatl.install.id` (a random v4 UUID), `mecatl.binary` (one of the closed -`Binary` set: `mecated`/`mecatui`/`mecatequi`/`mecak8s`). +`mecatl.binary` (one of the closed `Binary` set: +`mecated`/`mecatui`/`mecatequi`/`mecak8s`). + +**No per-install identifier is attached, deliberately.** This pipeline's +destination is a Prometheus-remote-write backend (`stacklok/infra#5604`), +where every resource attribute becomes a permanent label on *every* +instrument's time series. A random per-install UUID would multiply active +series by (installs × instrument count) with no bound as adoption grows — +an unbounded-cardinality cost for a precision (exact unique-install counts) +the design never actually required. `installid.go` still persists a local +UUID, but purely as a first-run marker for the disclosure notice (§ below) +— its value is never read back by `NewProvider` or attached to anything +exported. Unique-install counts are approximated from heartbeat volume/ +cadence instead of counted exactly. **Heartbeat** (`metrics.go`/`heartbeat.go` — on start, then every ~24h for long-running processes; single fire for `mecatequi`): @@ -151,12 +164,13 @@ Product metrics are **enabled by default** (opt-out). `internal/cliconfig.Resolv (`productmetrics_config.go`) folds four inputs, highest precedence first: 1. An explicit CLI flag: `--product-metrics=false` (all four binaries). -2. The `DO_NOT_TRACK` environment variable (any non-empty value) — the - cross-ecosystem convention (consoledonottrack.com), so the one env var - that already opts CI fleets and dev machines out of *other* tools' - telemetry covers mecatl too, with no mecatl-specific variable to - remember. (A dedicated `MECATL_PRODUCT_METRICS=0` was deliberately not - added on top of it — one standard signal beats two overlapping ones.) +2. The `DO_NOT_TRACK` environment variable set to a truthy value (`""`/`"0"`/ + `"false"`, case-insensitive, are NOT an opt-out) — the cross-ecosystem + convention (donottrack.sh), so the one env var that already opts CI + fleets and dev machines out of *other* tools' telemetry covers mecatl + too, with no mecatl-specific variable to remember. (A dedicated + `MECATL_PRODUCT_METRICS=0` was deliberately not added on top of it — one + standard signal beats two overlapping ones.) 3. `telemetry.productMetrics.enabled: false` in the **operator-tier** settings file (`~/.config/mecatl/settings.yaml` + CLI-loaded equivalents) — `permconfig.Resolver.OperatorProductMetricsEnabled()`. @@ -259,7 +273,8 @@ one and the balance shifts back toward requiring opt-in: `go test`/CI build can never phone home regardless of flag state. - A new small persisted file per install (`$XDG_STATE_HOME/mecatl/telemetry-id`, a bare random v4 UUID) — trivially - reset by deleting it, and carrying no machine or user information. + reset by deleting it, carrying no machine or user information, and never + read back for export: it exists purely as a local first-run marker. - ADR-0027 List-1 (resource inventory) is NOT extended: the heartbeat ticker's lifetime matches the process (owned by the caller's `heartbeatCtx`, cancelled on shutdown alongside the rest of composition's diff --git a/internal/adapter/productmetrics/config.go b/internal/adapter/productmetrics/config.go index 40506f3af1..2a871b113e 100644 --- a/internal/adapter/productmetrics/config.go +++ b/internal/adapter/productmetrics/config.go @@ -87,6 +87,4 @@ type Config struct { Binary Binary // Version is the mecatl build version (resource attribute service.version). Version string - // InstallID is this process's persisted anonymous install identifier. - InstallID string } diff --git a/internal/adapter/productmetrics/installid.go b/internal/adapter/productmetrics/installid.go index 863be18884..f4e8282e7b 100644 --- a/internal/adapter/productmetrics/installid.go +++ b/internal/adapter/productmetrics/installid.go @@ -21,9 +21,16 @@ const installIDRelPath = "mecatl/telemetry-id" // absent or unparseable. The id is a bare random v4 UUID: it carries no // machine or user information, and is trivially reset by deleting the file // (the next opt-in mints a new one). firstRun is true whenever a new id was -// just minted — the caller uses it to decide whether to print the one-time -// disclosure notice. readFile/writeFile/mkdirAll are injected for testing; -// LoadOrCreateInstallIDDefault binds the real filesystem. +// just minted — the caller uses it ONLY to decide whether to print the +// one-time disclosure notice. readFile/writeFile/mkdirAll are injected for +// testing; LoadOrCreateInstallIDDefault binds the real filesystem. +// +// The returned id is deliberately never threaded into any exported metric +// attribute or resource (see provider.go): this pipeline's destination is a +// Prometheus-remote-write backend, where a per-install identifier would +// become a permanent, unbounded-cardinality label on every instrument. The +// file still exists purely as a local first-run marker for the disclosure +// notice — its actual UUID value has no other consumer. func LoadOrCreateInstallID( env xdgconfig.ResolveEnv, readFile func(string) ([]byte, error), diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go index fdb8471fef..bf7ee2cf75 100644 --- a/internal/adapter/productmetrics/provider.go +++ b/internal/adapter/productmetrics/provider.go @@ -52,6 +52,16 @@ func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { // pointed at an httptest.Server) must explicitly opt into WithInsecure, // or the exporter tries TLS against a plaintext listener and every // export fails. The real production endpoint is always https://. + // + // Deliberately NOT included: any per-install identifier. This pipeline's + // destination is a Prometheus-remote-write backend (stacklok/infra#5604), + // where every resource attribute becomes a permanent label on EVERY + // instrument's time series — attaching a random per-install value here + // would multiply active-series count by (installs × instrument count), + // an unbounded-cardinality cost with no bound as adoption grows. Only + // mecatl.binary (a small closed enum) is attached; unique-install + // counting is approximated from heartbeat volume instead (see + // installid.go's doc comment). composite, err := providers.NewCompositeProvider(ctx, providers.WithServiceName("mecatl"), providers.WithServiceVersion(cfg.Version), @@ -60,8 +70,7 @@ func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { providers.WithInsecure(strings.HasPrefix(endpoint, "http://")), providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), providers.WithCustomAttributes(map[string]string{ - "mecatl.install.id": cfg.InstallID, - "mecatl.binary": string(cfg.Binary), + "mecatl.binary": string(cfg.Binary), }), ) if err != nil { diff --git a/internal/adapter/productmetrics/provider_test.go b/internal/adapter/productmetrics/provider_test.go index 62c736f335..e8c4062490 100644 --- a/internal/adapter/productmetrics/provider_test.go +++ b/internal/adapter/productmetrics/provider_test.go @@ -35,9 +35,8 @@ func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { defer func() { endpoint = origEndpoint }() p, err := NewProvider(context.Background(), Config{ - Binary: BinaryMecated, - Version: "test", - InstallID: "11111111-1111-1111-1111-111111111111", + Binary: BinaryMecated, + Version: "test", }) if err != nil { t.Fatalf("NewProvider: %v", err) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 6947bd4e63..b7d4ce2601 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -81,15 +81,19 @@ func BuildProductMetrics( return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil } - installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + // LoadOrCreateInstallIDDefault still runs (and persists its file) purely + // to detect first-run for the disclosure notice below — the returned id + // value itself is deliberately discarded, never threaded to NewProvider: + // see provider.go's doc comment on why a per-install identifier must + // never become a Prometheus-remote-write label. + _, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() if err != nil { return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) } provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ - Binary: binary, - Version: version, - InstallID: installID, + Binary: binary, + Version: version, }) if err != nil { return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: provider: %w", err) diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go index 11f5a0ac66..93986ffad1 100644 --- a/internal/cliconfig/productmetrics_config.go +++ b/internal/cliconfig/productmetrics_config.go @@ -12,7 +12,7 @@ import ( ) // doNotTrackOptOut reports whether a DO_NOT_TRACK env value means "opt out", -// per the consoledonottrack.com convention: unset/empty and the conventional +// per the donottrack.sh convention: unset/empty and the conventional // "off" spellings ("0", "false", case-insensitive) are NOT an opt-out; any // other value is. func doNotTrackOptOut(v string) bool { @@ -26,7 +26,7 @@ func doNotTrackOptOut(v string) bool { // ProductMetricsPrecedence carries the opt-out inputs // ResolveProductMetricsEnabled folds, highest precedence first: an explicit -// CLI flag, then the DO_NOT_TRACK env var convention (consoledonottrack.com), +// CLI flag, then the DO_NOT_TRACK env var convention (donottrack.sh), // then the operator settings.yaml value, then default-enabled. type ProductMetricsPrecedence struct { // FlagSet/FlagValue report whether --product-metrics was explicitly diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 34622499cb..a46f642281 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -240,7 +240,7 @@ The four channels above are all **operator-facing**: they help you observe your **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: - `--product-metrics=false` on the command line (all four binaries). -- The `DO_NOT_TRACK` environment variable (any non-empty value) — the same convention other tools already respect. +- The `DO_NOT_TRACK` environment variable set to a truthy value (`"0"`/`"false"` do not opt out) — the same convention other tools already respect. - `telemetry.productMetrics.enabled: false` in your **operator-tier** `~/.config/mecatl/settings.yaml`. This setting is operator-tier only: a project repo's `.mecatl/settings.yaml` cannot change your telemetry choice in either direction. **Self-verify before trusting it.** `--product-metrics-dry-run` prints every observation this pipeline would have sent to stderr instead of exporting it, so you can check the "no PII" claim yourself rather than take the docs' word for it. From 9ef77863a31ff3d60b15285faea1960d9ee8c470 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 17:54:31 -0400 Subject: [PATCH 26/47] docs(plan): add follow-on plan for install.id reinstatement + activation metrics Co-Authored-By: Claude Sonnet 5 --- ...10-product-metrics-activation-extension.md | 1320 +++++++++++++++++ 1 file changed, 1320 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md diff --git a/docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md b/docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md new file mode 100644 index 0000000000..d00975b26f --- /dev/null +++ b/docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md @@ -0,0 +1,1320 @@ +# Product Metrics — Activation/Retention/Reliability Extension Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extend the already-shipped `internal/adapter/productmetrics` pipeline (PR #1278) to answer the product team's activation/time-to-value/retention/reliability questions: reinstate a per-install identity (accepting the cardinality cost, now that it's been sized and understood), and add `had_tool_call`, tool category/outcome, `run_duration`, `tool_calls_per_run`, and `time_to_first_value`. + +**Architecture:** A small, additive `engine/port` capability (`RunAwareToolCallRecorder`, mirroring the existing `HookApprovalLearner` optional-interface precedent) lets the already-shipped `Recorder` correlate a tool call to the run that made it — the one piece of engine-layer work this requires. Everything else is confined to `internal/adapter/productmetrics`, `internal/cliconfig`, and a new Helm template for `mecak8s`'s install-id provisioning. + +**Tech Stack:** Go 1.26, the existing `internal/adapter/productmetrics` package (Tasks 1-15 of the prior plan, already merged), Helm (`deploy/helm/mecak8s/`). + +**Spec:** This plan's own context section below records every design decision reached in conversation; there is no separate written design doc for this increment — the conversation that produced it is the record. + +## Global Constraints + +- **No new free-text/PII surface.** Tool category comes from a structural `mcp__` prefix check (confirmed via `internal/adapter/mcp/tool.go:89`'s `"mcp__" + server + "__" + toolName` construction) — never a maintained allowlist, never the raw MCP server/tool name. Everything else stays a bounded enum or a count/duration, per the existing package-wide guard test discipline (Task 6 of the prior plan). +- **`install.id` is now a deliberate, accepted exception** to "every attribute is bounded" — reinstate it as a resource attribute (undoing the earlier removal), now that its cardinality cost has been sized (~$1,930/month at 100K installs, worst-case 24/7 uptime, full catalog, on the actual AMP pricing model) and accepted. +- **The engine port change must be purely additive.** `RunAwareToolCallRecorder` is a NEW, STANDALONE interface (not embedding `ToolCallRecorder`), type-asserted at the one dispatch call site — mirrors `engine/port/hookrunner.go`'s `HookApprovalLearner` exactly. No existing `ToolCallRecorder` implementer (the operator `internal/adapter/telemetry.Metrics`/`RoleMetrics`, `jsonlstore`, `redisstore`, etc.) needs to change at all. This is an `Added` (minor) change per `engine/COMPATIBILITY.md` — run `task api:update` and add an `engine/CHANGELOG.md` entry. +- **`task lint && task test` must stay green after every task.** `task api:check` (part of `task test`) must pass after the engine port change. + +--- + +### Task 1: Engine port extension — `RunAwareToolCallRecorder` + +**Files:** +- Modify: `engine/port/log.go` (add the new interface, do NOT touch `ToolCallRecorder`) +- Modify: `engine/agent/dispatch.go` (the one call site, `execute`, currently around line 1293) +- Test: `engine/agent/run_aware_tool_call_recorder_test.go` (new) +- Modify: `engine/CHANGELOG.md`, run `task api:update` to regenerate `engine/api/*.txt` + +**Interfaces:** +- Produces: `type RunAwareToolCallRecorder interface { ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) }` in `engine/port`. + +- [ ] **Step 1: Write the failing test** + +```go +package agent + +import ( + "context" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +// runAwareFakeRecorder implements BOTH port.ToolCallRecorder and the new +// port.RunAwareToolCallRecorder, recording which method the dispatcher chose. +type runAwareFakeRecorder struct { + plainCalls int + runAwareCalls int + lastRunID string +} + +func (f *runAwareFakeRecorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + f.plainCalls++ +} + +func (f *runAwareFakeRecorder) ToolCallForRun(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + f.runAwareCalls++ + f.lastRunID = runID +} + +// TestExecutePrefersRunAwareToolCallRecorderWhenImplemented pins that the +// dispatcher, at its one ToolCallRecorder call site, calls ToolCallForRun +// (never both) when the injected recorder implements it, passing the SAME +// RunID the enclosing Run already carries — and falls back to the plain +// ToolCall for a recorder that does not implement the richer interface +// (every existing ToolCallRecorder implementer is unaffected). +func TestExecutePrefersRunAwareToolCallRecorderWhenImplemented(t *testing.T) { + rec := &runAwareFakeRecorder{} + eng, sess := newTestEngineWithToolCallRecorder(t, rec) // see Step 3 for this test helper's real signature, sourced from an existing dispatch_test.go helper + runID := driveOneToolCallingTurn(t, eng, sess) // helper: drives a turn that calls a tool at least once + + if rec.plainCalls != 0 { + t.Errorf("plainCalls = %d, want 0 (RunAwareToolCallRecorder must be preferred)", rec.plainCalls) + } + if rec.runAwareCalls == 0 { + t.Fatal("runAwareCalls = 0, want at least 1") + } + if rec.lastRunID != runID { + t.Errorf("lastRunID = %q, want %q (the enclosing Run's own id)", rec.lastRunID, runID) + } +} + +func TestExecuteFallsBackToPlainToolCallRecorder(t *testing.T) { + // A recorder implementing ONLY port.ToolCallRecorder (not the richer + // interface) must keep working exactly as before — confirmed via the + // EXISTING plain-ToolCallRecorder test fixture already in this package + // (find it by name in dispatch_test.go and reuse it directly rather than + // inventing a new one). +} +``` + +Note to implementer: `newTestEngineWithToolCallRecorder`/`driveOneToolCallingTurn` are placeholder helper NAMES — before writing this file, grep `engine/agent/*_test.go` for the EXISTING test harness this package already uses to build a test `*Engine` and drive a tool-calling turn (there is one; every dispatch test in this package uses it), and write these two tests using the REAL existing helpers/fixtures, not new ones. `TestExecuteFallsBackToPlainToolCallRecorder`'s body is intentionally left for you to fill in using that same real harness with a recorder implementing only the base interface — assert its existing plain-`ToolCall` path still fires exactly as it does today (this is a regression guard, not new behavior). + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd engine && go test ./agent/... -run TestExecutePrefersRunAwareToolCallRecorderWhenImplemented -v` +Expected: FAIL — `port.RunAwareToolCallRecorder` undefined, or the dispatcher doesn't yet type-assert for it. + +- [ ] **Step 3: Add the port interface** + +In `engine/port/log.go`, immediately after the existing `ToolCallRecorder` interface, add: + +```go +// RunAwareToolCallRecorder is an OPTIONAL capability a ToolCallRecorder may +// ALSO implement to additionally receive the RunID of the run that made the +// call (the same opaque per-run correlation id carried on session.Event.RunID, +// ADR 0249) — the one thing ToolCall's signature cannot express, since a +// SessionID can span many sequential runs over a session's lifetime and +// ToolCall alone gives no way to tell which run a given call belongs to. +// +// The engine TYPE-ASSERTS this interface on Deps.ToolCallRecorder and calls +// ToolCallForRun INSTEAD OF ToolCall (never both) when implemented — so a +// recorder that implements only the base ToolCallRecorder is wholly +// unaffected (no method added to ToolCallRecorder: that would be a breaking +// change, mirroring the HookApprovalLearner precedent in hookrunner.go). +type RunAwareToolCallRecorder interface { + // ToolCallForRun is ToolCall's signature plus the leading runID — the + // same value the enclosing Run stamps onto every session.Event.RunID it + // emits. Consumers that need to correlate a tool call to the run that + // made it (e.g. "did this run have at least one successful tool call") + // use this instead of ToolCall. + ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) +} +``` + +- [ ] **Step 4: Wire the type-assertion at the dispatch call site** + +In `engine/agent/dispatch.go`, replace: + +```go + if e.deps.ToolCallRecorder != nil { + e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + } +``` + +with: + +```go + if e.deps.ToolCallRecorder != nil { + if aware, ok := e.deps.ToolCallRecorder.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(r.RunID(), sess.ID, c, res, queued, dur) + } else { + e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + } + } +``` + +(`r` is the enclosing `execute` method's existing `*Run` parameter — already in scope two lines below this call site where `e.emit(r, ...)` is called; `port` is already imported in this file — confirm, and add the import if for some reason it is not.) + +- [ ] **Step 5: Run test to verify it passes** + +Run: `cd engine && go test ./agent/... -run 'TestExecutePrefersRunAwareToolCallRecorderWhenImplemented|TestExecuteFallsBackToPlainToolCallRecorder' -v` +Expected: PASS (both tests) + +- [ ] **Step 6: Run the full engine test suite to catch any regression** + +Run: `cd engine && go test ./... -race` +Expected: PASS, no regressions (this is an additive interface change; every existing `ToolCallRecorder` consumer's tests should be untouched). + +- [ ] **Step 7: Update the engine API compatibility surface** + +Run: `task api:update` (regenerates `engine/api/*.txt`). Then add an entry to `engine/CHANGELOG.md` classified as `Added` (minor) per `engine/COMPATIBILITY.md`'s rules, e.g.: + +```markdown +## Unreleased +### Added +- `port.RunAwareToolCallRecorder`: an optional `ToolCallRecorder` extension + that additionally receives the calling run's `RunID`, letting a consumer + correlate a tool call to the run that made it. Purely additive — no + existing `ToolCallRecorder` implementer is affected. +``` + +- [ ] **Step 8: Run `task api:check` and repo-wide `task lint`** + +Run: `task api:check` (should now pass against the regenerated `engine/api/*.txt`) and `task lint` (0 issues expected). + +- [ ] **Step 9: Commit** + +```bash +git add engine/port/log.go engine/agent/dispatch.go engine/agent/run_aware_tool_call_recorder_test.go engine/CHANGELOG.md engine/api/ +git commit -m "feat(engine): add the optional RunAwareToolCallRecorder port capability" +``` + +--- + +### Task 2: Reinstate `install.id` + +**Files:** +- Modify: `internal/adapter/productmetrics/config.go` (restore `Config.InstallID`) +- Modify: `internal/adapter/productmetrics/provider.go` (restore the resource attribute) +- Modify: `internal/adapter/productmetrics/provider_test.go` (restore the test's `InstallID` field) +- Modify: `internal/cliconfig/productmetrics.go` (thread `installID` back into `Config`) +- Modify: `internal/adapter/productmetrics/installid.go` (doc comment: remove the "deliberately never threaded" language — it's threaded again now) +- Test: existing `installid_test.go` unaffected (the persistence mechanism itself never changed) + +**Interfaces:** +- Produces: `Config.InstallID string` restored; `NewProvider`'s resource attributes include `"mecatl.install.id": cfg.InstallID` again. + +- [ ] **Step 1: Restore `Config.InstallID`** + +In `internal/adapter/productmetrics/config.go`, restore the field: + +```go +// Config configures a Provider/Recorder pair for one process. +type Config struct { + // Binary identifies which of the four entry points this process is. + Binary Binary + // Version is the mecatl build version (resource attribute service.version). + Version string + // InstallID is this process's persisted (or externally-provisioned, for + // mecak8s — see Task 7) anonymous install identifier. Reinstated as a + // resource attribute after being sized and accepted: ~$1,930/month at + // 100K installs under worst-case 24/7 uptime on the actual AMP pricing + // model (see the ADR's updated cost-analysis section, Task 8). + InstallID string +} +``` + +- [ ] **Step 2: Write the failing test (provider_test.go)** + +Restore the `InstallID` field to the existing `TestNewProviderExportsToConfiguredEndpoint` test's `Config{...}` literal: + +```go + p, err := NewProvider(context.Background(), Config{ + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", + }) +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go build ./...` +Expected: FAIL — `unknown field InstallID` (Config doesn't have it back yet if you did Step 2 before Step 1 — do Step 1 first; this ordering note exists so the two steps are both concrete, not because there is a real red-green gap here — `Config`'s field addition and the test asserting it stay in the SAME commit). + +- [ ] **Step 4: Restore the resource attribute in `provider.go`** + +In `internal/adapter/productmetrics/provider.go`, restore: + +```go + composite, err := providers.NewCompositeProvider(ctx, + providers.WithServiceName("mecatl"), + providers.WithServiceVersion(cfg.Version), + providers.WithOTLPEndpoint(endpoint), + providers.WithMetricsEnabled(true), + providers.WithInsecure(strings.HasPrefix(endpoint, "http://")), + providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), + providers.WithCustomAttributes(map[string]string{ + "mecatl.install.id": cfg.InstallID, + "mecatl.binary": string(cfg.Binary), + }), + ) +``` + +removing the "Deliberately NOT included" doc comment above it (or rewriting it — see Step 5). + +- [ ] **Step 5: Rewrite the doc comment explaining the reinstated decision** + +Replace the comment block above the `NewCompositeProvider` call with: + +```go + // mecatl.install.id is a per-install random UUID, deliberately attached + // as a resource attribute (so it flattens onto every instrument this + // provider exports). This was removed once (see git history) over + // unbounded-cardinality concerns on the Prometheus-remote-write + // destination (stacklok/infra#5604), then reinstated after the actual + // cost was sized against real AMP pricing and accepted — see the ADR's + // cost-analysis section for the numbers. mecak8s provisions this value + // differently (a stable per-Helm-release ConfigMap, not this package's + // local install-id file — see internal/cliconfig's mecak8s wiring and + // deploy/helm/mecak8s/templates/install-id-configmap.yaml), since a + // pod-local file would mint a new id on every pod restart. +``` + +- [ ] **Step 6: Restore threading in `internal/cliconfig/productmetrics.go`** + +Change: + +```go + // LoadOrCreateInstallIDDefault still runs (and persists its file) purely + // to detect first-run for the disclosure notice below — the returned id + // value itself is deliberately discarded, never threaded to NewProvider: + // see provider.go's doc comment on why a per-install identifier must + // never become a Prometheus-remote-write label. + _, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + }) +``` + +to: + +```go + // LoadOrCreateInstallIDDefault persists (or reads back) this process's + // local install-id file and reports firstRun for the disclosure notice + // below. Reinstated as a real, exported resource attribute (see + // provider.go's doc comment) after its cardinality cost was sized and + // accepted. + installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + InstallID: installID, + }) +``` + +Note: `BuildProductMetrics`'s signature does NOT change in this task — mecak8s's alternate provisioning (Task 7) overrides `installID` BEFORE calling `BuildProductMetrics` by having its OWN caller read the env-var-provided id and pass it through a new, distinct code path added in Task 7; this task only restores the DEFAULT (local-file) path all four binaries currently share. + +- [ ] **Step 7: Update `installid.go`'s doc comment** + +In `internal/adapter/productmetrics/installid.go`, remove the paragraph beginning "The returned id is deliberately never threaded into any exported metric attribute..." (added when `install.id` was removed) — replace with: + +```go +// The returned id IS threaded into an exported resource attribute (see +// provider.go) — this package makes no attempt to keep the id local-only; +// that was a prior, now-reverted design (see git history / the ADR's +// cost-analysis section for why it was reinstated). +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` and `cd ../../cliconfig && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/config.go internal/adapter/productmetrics/provider.go \ + internal/adapter/productmetrics/provider_test.go internal/adapter/productmetrics/installid.go \ + internal/cliconfig/productmetrics.go +git commit -m "feat(productmetrics): reinstate mecatl.install.id after sizing its cardinality cost" +``` + +--- + +### Task 3: Per-run tracking — `had_tool_call`, tool category + outcome + +**Files:** +- Modify: `internal/adapter/productmetrics/metrics.go` (unify per-run tracking; add `had_tool_call` attribute; add the `attrCategory`/`attrOutcome` keys) +- Modify: `internal/adapter/productmetrics/toolcall.go` (implement `ToolCallForRun`, category/outcome derivation, per-run tallying) +- Modify: `internal/adapter/productmetrics/metrics_test.go`, `toolcall_test.go` +- Modify: `internal/adapter/productmetrics/bounded_test.go` (extend the allowlist + drive `ToolCallForRun` with sensitive markers) + +**Interfaces:** +- Consumes: `port.RunAwareToolCallRecorder` (Task 1). +- Produces: `Recorder.ToolCallForRun(...)` (satisfies the new interface); `mecatl.product.runs_completed`'s new `had_tool_call` attribute; `mecatl.product.tool_calls`'s new `category`/`outcome` attributes. + +- [ ] **Step 1: Write the failing tests** + +```go +// In toolcall_test.go, alongside the existing TestRecorderToolCallCountsWithoutIdentity: + +func TestRecorderToolCallForRunCategorizesBuiltinsByName(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, "category", "Bash"); got != 1 { + t.Errorf("tool_calls{category=Bash} = %d, want 1", got) + } + if got := sumPoint(t, agg, "category", "Read"); got != 1 { + t.Errorf("tool_calls{category=Read} = %d, want 1", got) + } +} + +func TestRecorderToolCallForRunBucketsMCPToolsUnderOneCategory(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__github__list_issues"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__slack__post_message"}, session.ToolResult{}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, "category", "mcp"); got != 2 { + t.Errorf("tool_calls{category=mcp} = %d, want 2 (both MCP-server tools bucketed together)", got) + } + // The real server/tool names must never appear as an attribute value. + var rm metricdata.ResourceMetrics + _ = reader.Collect(context.Background(), &rm) + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + continue + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + if kv.Value.AsString() == "github" || kv.Value.AsString() == "list_issues" { + t.Fatalf("MCP server/tool name leaked as an attribute value: %s=%s", kv.Key, kv.Value.AsString()) + } + } + } + } + } +} + +func TestRecorderToolCallForRunRecordsOutcome(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, "outcome", "success"); got != 1 { + t.Errorf("tool_calls{outcome=success} = %d, want 1", got) + } + if got := sumPoint(t, agg, "outcome", "error"); got != 1 { + t.Errorf("tool_calls{outcome=error} = %d, want 1", got) + } +} +``` + +```go +// In metrics_test.go, alongside the existing runs_completed tests: + +func TestRecorderRunsCompletedHadToolCallTrueWhenASuccessfulToolCallOccurred(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1", got) + } +} + +func TestRecorderRunsCompletedHadToolCallFalseWithNoToolCall(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-2", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1", got) + } +} + +func TestRecorderRunsCompletedHadToolCallFalseWhenOnlyToolCallErrored(t *testing.T) { + // A tool call that ERRORED does not count toward had_tool_call — the + // product definition requires at least one SUCCESSFUL tool/action. + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-3", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-3", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1 (the only tool call errored)", got) + } +} + +func TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns(t *testing.T) { + // Two runs interleaved (a real possibility: Team/Parallel fan-out, or + // two concurrent client sessions on one process) must not leak state + // into each other. + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-a", session.SessionID("s1"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-b", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-a", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "false"); got != 1 { + t.Errorf("run-b's had_tool_call = %d points at false, want exactly 1 (run-a's tool call must not leak into run-b)", got) + } + if got := sumPoint(t, agg, "had_tool_call", "true"); got != 1 { + t.Errorf("run-a's had_tool_call = %d points at true, want exactly 1", got) + } +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd internal/adapter/productmetrics && go build ./...` +Expected: FAIL — `ToolCallForRun` undefined, `had_tool_call`/`category`/`outcome` attributes don't exist yet. + +- [ ] **Step 3: Unify per-run tracking in `metrics.go`** + +Replace the existing `runFamiliesUsed`/`usedFamilies` per-run tracking (added in the prior plan's review-fix commit) with a single, richer per-run record covering everything this task and Task 4 need: + +```go +// perRunState tracks, per LIVE run (keyed by session.Event.RunID / the same +// id ToolCallForRun receives), the bounded facts this package derives across +// the Emit/ToolCallForRun boundary. Cleared on EvResult so it stays bounded +// to concurrently-live runs, never growing across a process's lifetime. +type perRunState struct { + subagentSeen bool + teamSeen bool + hadToolCall bool + toolCallCount int64 + startedAt time.Time +} + +// runs guards concurrent access to the live-run map — the SAME discipline +// the prior subagent/team dedup fix already established, now extended to +// cover had_tool_call/tool_calls_per_run/run_duration too. +type runs struct { + mu sync.Mutex + byRun map[string]*perRunState +} + +func newRuns() *runs { return &runs{byRun: make(map[string]*perRunState)} } + +// get returns (creating if absent) the live perRunState for runID. An empty +// runID (no run context) returns a throwaway, never-shared state — matching +// the prior code's "always counts" fallback for the no-run-context case. +func (r *runs) get(runID string) *perRunState { + if runID == "" { + return &perRunState{} + } + r.mu.Lock() + defer r.mu.Unlock() + st, ok := r.byRun[runID] + if !ok { + st = &perRunState{} + r.byRun[runID] = st + } + return st +} + +// clear drops runID's live state at EvResult, returning the state that was +// there (or a zero-value one if none existed — e.g. a run with no tool +// calls and no delegation family use). +func (r *runs) clear(runID string) *perRunState { + if runID == "" { + return &perRunState{} + } + r.mu.Lock() + defer r.mu.Unlock() + st, ok := r.byRun[runID] + if !ok { + return &perRunState{} + } + delete(r.byRun, runID) + return st +} +``` + +Replace the `Recorder` struct's `mu sync.Mutex` + `runFamiliesUsed map[string]usedFamilies` fields with a single `perRun *runs` field, and update `NewRecorder` to initialize it: `r.perRun = newRuns()`. + +Update `firstInRun`/`clearRun` (the prior plan's helpers) to use `perRun.get(runID)`/`perRun.clear(runID)` instead of the old map directly — e.g.: + +```go +func (r *Recorder) firstInRun(runID string, family delegationFamily) bool { + st := r.perRun.get(runID) + r.perRun.mu.Lock() + defer r.perRun.mu.Unlock() + switch family { + case familySubagent: + if st.subagentSeen { + return false + } + st.subagentSeen = true + case familyTeam: + if st.teamSeen { + return false + } + st.teamSeen = true + } + return true +} +``` + +(Adjust exact lock placement so `runs.get`'s own internal lock and this method's use of the returned pointer don't double-lock or race — the simplest correct shape is for ALL mutation of a `*perRunState`'s fields to happen while holding `r.perRun.mu`, so restructure `get`/`clear` to return the state WITHOUT unlocking around field access, or have every state-mutating method take the lock itself around both the map lookup AND the field mutation as one critical section. Get this right and prove it with `-race` in Step 8 — this is the one place in this task worth extra care.) + +- [ ] **Step 4: Add the `had_tool_call` attribute to `recordResult`** + +```go +const attrHadToolCall = "had_tool_call" + +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, runID string) { + st := r.perRun.clear(runID) + hadToolCall := "false" + if st.hadToolCall { + hadToolCall = "true" + } + if res == nil { + r.runsCompleted.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrStop, string(session.StopNone)), + attribute.String(attrHadToolCall, hadToolCall))) + return + } + r.runsCompleted.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrStop, string(res.Stop)), + attribute.String(attrHadToolCall, hadToolCall))) + u := res.Usage + // ... existing tokens.Add(...) calls unchanged ... +} +``` + +Update `Emit`'s `case session.EvResult:` arm to pass `ev.RunID`: `r.recordResult(ctx, ev.Result, ev.RunID)`. + +- [ ] **Step 5: Implement `ToolCallForRun` in `toolcall.go`** + +```go +package productmetrics + +import ( + "context" + "strings" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +const ( + attrCategory = "category" + attrOutcome = "outcome" +) + +// mcpToolPrefix is the STRUCTURAL naming convention every client MCP tool is +// registered under (internal/adapter/mcp/tool.go: `"mcp__" + server + "__" + +// toolName`) — checking this prefix, rather than maintaining a built-in-tool +// allowlist, means (a) a new built-in tool is automatically and correctly +// categorized by its own real name with no allowlist to keep in sync, and +// (b) an MCP server/tool name can never leak, structurally, regardless of +// what any future MCP integration is named. +const mcpToolPrefix = "mcp__" + +// toolCategory derives the bounded category attribute for a tool call: the +// tool's own name for a built-in (never sensitive — mecatl's own fixed +// catalog), or the single literal "mcp" for anything MCP-server-provided +// (never the specific server/tool name). +func toolCategory(name string) string { + if strings.HasPrefix(name, mcpToolPrefix) { + return "mcp" + } + return name +} + +// ToolCall satisfies port.ToolCallRecorder for a caller that does not +// implement/use the richer port.RunAwareToolCallRecorder path — it records +// with no run correlation (runID ""), matching this package's pre-existing, +// always-counts behavior for the no-run-context case. +func (r *Recorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + r.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder. It records the +// bounded category/outcome attributes and tallies the per-run state Task 4's +// run_duration and this task's had_tool_call/tool_calls_per_run all read at +// EvResult time. It never reads call.Name/result.Content beyond the bounded +// category derivation above — no free text, no session id, no MCP +// server/tool name. +func (r *Recorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + ctx := context.Background() + outcome := "success" + if result.IsError { + outcome = "error" + } + r.toolCalls.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrCategory, toolCategory(call.Name)), + attribute.String(attrOutcome, outcome))) + + st := r.perRun.get(runID) + r.perRun.mu.Lock() + st.toolCallCount++ + if !result.IsError { + st.hadToolCall = true + } + r.perRun.mu.Unlock() +} + +// Compile-time interface checks. +var ( + _ port.ToolCallRecorder = (*Recorder)(nil) + _ port.RunAwareToolCallRecorder = (*Recorder)(nil) +) +``` + +(Remove the OLD `ToolCall` implementation this replaces — the prior plan's version that ignored every parameter and just bumped `r.toolCalls.Add(ctx, 1)` with no attributes.) + +- [ ] **Step 6: Update `metrics.go`'s `tool_calls` instrument description** + +The `mecatl.adoption.tool_calls`/`mecatl.product.tool_calls` counter's `metric.WithDescription(...)` string currently says "no tool identity attached" — update it: + +```go + if r.toolCalls, err = meter.Int64Counter("mecatl.product.tool_calls", + metric.WithDescription("Total tool calls executed, by bounded category (a built-in tool's own name, or the single value \"mcp\" for any MCP-server tool) and outcome.")); err != nil { +``` + +- [ ] **Step 7: Extend `bounded_test.go`** + +Add `attrCategory`, `attrOutcome`, `attrHadToolCall` to `allowedAttributeKeys`. Extend the existing `TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent` to ALSO drive `ToolCallForRun("run-x", ..., session.ToolCall{Name: "mcp__evilserver__leak_this_name"}, ...)` and assert `"evilserver"`/`"leak_this_name"` never appear as an attribute value anywhere in the collected output — this is the guard test's whole job, extend it rather than adding a separate one. + +- [ ] **Step 8: Run tests with `-race` to verify correctness and no data races** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` +Expected: PASS, all tests including the new ones, `-race` clean (this task adds concurrent map + struct-field access — `-race` is not optional here). + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/toolcall.go \ + internal/adapter/productmetrics/metrics_test.go internal/adapter/productmetrics/toolcall_test.go \ + internal/adapter/productmetrics/bounded_test.go +git commit -m "feat(productmetrics): had_tool_call, tool category/outcome via RunAwareToolCallRecorder" +``` + +--- + +### Task 4: `run_duration` histogram + +**Files:** +- Modify: `internal/adapter/productmetrics/metrics.go` +- Test: `internal/adapter/productmetrics/metrics_test.go` + +**Interfaces:** +- Consumes: `perRunState.startedAt` (Task 3). +- Produces: `mecatl.product.run_duration` histogram (seconds). + +- [ ] **Step 1: Write the failing test** + +```go +func TestRecorderRunDurationRecordedFromSessionInitToResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + time.Sleep(5 * time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.run_duration"] + if !ok { + t.Fatal("mecatl.product.run_duration missing") + } + hist, ok := agg.(metricdata.Histogram[float64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[float64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 { + t.Fatalf("expected exactly 1 recorded duration, got %+v", hist.DataPoints) + } + if hist.DataPoints[0].Sum <= 0 { + t.Errorf("recorded duration sum = %v, want > 0", hist.DataPoints[0].Sum) + } +} + +func TestRecorderRunDurationNotRecordedWithoutMatchingSessionInit(t *testing.T) { + // A run whose EvSessionInit this Recorder never observed (e.g. process + // restarted mid-run — an edge case, not a common path) must not record a + // bogus/negative duration. + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "orphan-run", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + if agg, ok := collect(t, reader)["mecatl.product.run_duration"]; ok { + if hist, ok := agg.(metricdata.Histogram[float64]); ok && len(hist.DataPoints) > 0 && hist.DataPoints[0].Count > 0 { + t.Errorf("recorded a duration for a run with no observed EvSessionInit: %+v", hist.DataPoints) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderRunDuration -v` +Expected: FAIL — instrument doesn't exist yet. + +- [ ] **Step 3: Add the instrument and start/stop bracketing** + +In `NewRecorder`, add: + +```go + if r.runDuration, err = meter.Float64Histogram("mecatl.product.run_duration", + metric.WithDescription("Wall-clock duration of a run, from session init to result, in seconds."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: run_duration histogram: %w", err) + } +``` + +Add `runDuration metric.Float64Histogram` to the `Recorder` struct. + +In `Emit`'s `case session.EvSessionInit:` arm, stamp the start time: + +```go + case session.EvSessionInit: + r.sessionsStarted.Add(ctx, 1) + if ev.RunID != "" { + st := r.perRun.get(ev.RunID) + r.perRun.mu.Lock() + st.startedAt = time.Now() + r.perRun.mu.Unlock() + } +``` + +In `recordResult` (or right after `st := r.perRun.clear(runID)` in `Emit`'s `EvResult` handling), record the duration only when `startedAt` was actually observed: + +```go + if !st.startedAt.IsZero() { + r.runDuration.Record(ctx, time.Since(st.startedAt).Seconds()) + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderRunDuration -v` +Expected: PASS + +- [ ] **Step 5: Run the full package suite with `-race`** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 6: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/metrics_test.go +git commit -m "feat(productmetrics): run_duration histogram" +``` + +--- + +### Task 5: `tool_calls_per_run` histogram + `time_to_first_value` + +**Files:** +- Modify: `internal/adapter/productmetrics/metrics.go` (the `tool_calls_per_run` histogram, recorded at `EvResult` from `perRunState.toolCallCount`) +- Create: `internal/adapter/productmetrics/firstvalue.go` (the one-time marker + histogram) +- Test: `internal/adapter/productmetrics/firstvalue_test.go` +- Modify: `internal/cliconfig/productmetrics.go` (thread the first-value check into `BuildProductMetrics`) + +**Interfaces:** +- Produces: `mecatl.product.tool_calls_per_run` histogram; `mecatl.product.time_to_first_value` histogram (recorded at most once per install); `func LoadOrCreateFirstValueMarker(...) (alreadyRecorded bool, err error)` mirroring `installid.go`'s shape. + +- [ ] **Step 1: `tool_calls_per_run` — write the failing test** + +```go +func TestRecorderToolCallsPerRunRecordedAtResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg := collect(t, reader)["mecatl.product.tool_calls_per_run"] + hist, ok := agg.(metricdata.Histogram[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[int64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Sum != 2 { + t.Fatalf("expected one data point summing to 2, got %+v", hist.DataPoints) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails, then implement** + +Add to `NewRecorder`: + +```go + if r.toolCallsPerRun, err = meter.Int64Histogram("mecatl.product.tool_calls_per_run", + metric.WithDescription("Total tool calls made within a single run.")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls_per_run histogram: %w", err) + } +``` + +In `recordResult`, alongside the existing `st := r.perRun.clear(runID)`: + +```go + r.toolCallsPerRun.Record(ctx, st.toolCallCount) +``` + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderToolCallsPerRun -v` — expect PASS after the change. + +- [ ] **Step 3: `time_to_first_value` — write the failing test** + +```go +package productmetrics + +import ( + "errors" + "os" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + written := map[string][]byte{} + readFile := func(p string) ([]byte, error) { + if d, ok := written[p]; ok { + return d, nil + } + return nil, os.ErrNotExist + } + writeFile := func(p string, d []byte, _ os.FileMode) error { written[p] = d; return nil } + mkdirAll := func(string, os.FileMode) error { return nil } + + already, err := LoadOrCreateFirstValueMarker(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) + } + if already { + t.Error("already = true on first call, want false") + } + + // Second call must report it as already recorded, and not error. + already2, err := LoadOrCreateFirstValueMarker(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateFirstValueMarker: %v", err) + } + if !already2 { + t.Error("already = false on second call, want true") + } +} + +func TestLoadOrCreateFirstValueMarkerFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + if _, err := LoadOrCreateFirstValueMarker(env, nil, nil, nil); err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } +} +``` + +- [ ] **Step 4: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestLoadOrCreateFirstValueMarker -v` +Expected: FAIL — `LoadOrCreateFirstValueMarker` undefined. + +- [ ] **Step 5: Write `firstvalue.go`** + +```go +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// firstValueMarkerRelPath is the state-dir-relative path to a bare marker +// file recording whether this install's first meaningful-and-successful run +// has already been observed — so mecatl.product.time_to_first_value is +// recorded at most once per install, ever, mirroring installid.go's +// first-run marker pattern exactly. +const firstValueMarkerRelPath = "mecatl/first-value-recorded" + +// LoadOrCreateFirstValueMarker reports whether this install's first-value +// moment was already recorded (already == true), creating the marker (and +// returning already == false) the first time it is called. Once created, it +// is never removed automatically — deleting it (like the install-id file) +// resets the install and lets time_to_first_value fire once more. +func LoadOrCreateFirstValueMarker( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (already bool, err error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return false, fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + path := filepath.Join(base, firstValueMarkerRelPath) + + if readFile != nil { + if _, rerr := readFile(path); rerr == nil { + return true, nil + } + } + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte("1"), 0o600); werr != nil { + return false, fmt.Errorf("productmetrics: write first-value marker: %w", werr) + } + } + return false, nil +} + +// LoadOrCreateFirstValueMarkerDefault binds LoadOrCreateFirstValueMarker to +// the real process environment and filesystem. +func LoadOrCreateFirstValueMarkerDefault() (already bool, err error) { + return LoadOrCreateFirstValueMarker(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} +``` + +- [ ] **Step 6: Add the `time_to_first_value` instrument and recording logic** + +Add to `NewRecorder`: + +```go + if r.timeToFirstValue, err = meter.Float64Histogram("mecatl.product.time_to_first_value", + metric.WithDescription("One-time-per-install duration from this install's first-seen moment to its first had_tool_call=true, stop=success run."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: time_to_first_value histogram: %w", err) + } +``` + +Add a field to `Recorder`: `firstSeenAt time.Time` (set once, at construction) and `firstValueRecorded *atomic.Bool` (or guard via the marker file check, done ONCE at `BuildProductMetrics` construction time rather than per-event — see Step 7, this is simpler than trying to gate it per-Emit-call). + +Actually, the simplest correct design: do the "has this already been recorded" check ONCE, in `internal/cliconfig.BuildProductMetrics` (Step 7 below), NOT inside `Recorder` itself — pass a plain `bool` (`trackFirstValue`) into `NewRecorder`/`Config`, and have `Recorder.recordResult` check `res.Stop == session.StopEndTurn && st.hadToolCall && !r.firstValueAlreadyRecorded` before recording once and flipping an in-memory flag (`sync.Once` or a guarded bool) — the FILE write (marking it recorded forever) happens in the CALLER once, when `BuildProductMetrics` first observes `already == false` at startup... but that's wrong too, since the marker needs to be written the MOMENT the qualifying run actually happens, not at process startup (a process might never have a qualifying run). Correct shape: `Recorder` itself owns a `sync.Once`-guarded write-through: on the FIRST qualifying `EvResult`, it (a) computes and records the duration, (b) calls a caller-injected `markFirstValueRecorded func() error` closure (wrapping `os.WriteFile` at the real path) exactly once. Thread this closure into `NewRecorder` (or a new `NewRecorderWithFirstValue(mp, alreadyRecorded bool, markRecorded func())` variant) rather than `Config`, to keep `NewRecorder`'s existing signature stable for the (many) existing call sites/tests that don't care about this feature. + +Given the added complexity, the pragmatic shape: + +```go +// Recorder field additions: + firstSeenAt time.Time + firstValueDone bool // true if already recorded (this run OR a prior one) + firstValueRecordFn func() error // writes the local marker file; nil disables recording entirely + firstValueMu sync.Mutex +``` + +`NewRecorder`'s signature stays unchanged (existing callers/tests untouched); add a new setter-style method used only by `BuildProductMetrics`: + +```go +// EnableFirstValueTracking arms mecatl.product.time_to_first_value tracking: +// firstSeenAt is this install's first-seen timestamp (from the SAME local +// install-id file's mtime, or "now" if unavailable — an approximation is +// fine, this metric's whole purpose is a coarse "how long did onboarding +// take" signal, not a billing-grade timer). alreadyRecorded, when true, +// permanently disables further recording for this Recorder's lifetime (this +// install already has its one sample). recordFn persists the marker so a +// LATER process invocation also stays disabled; it is called at most once. +func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecorded bool, recordFn func() error) { + r.firstValueMu.Lock() + defer r.firstValueMu.Unlock() + r.firstSeenAt = firstSeenAt + r.firstValueDone = alreadyRecorded + r.firstValueRecordFn = recordFn +} +``` + +In `recordResult`, after the existing token/had_tool_call recording: + +```go + if res != nil && res.Stop == session.StopEndTurn && st.hadToolCall { + r.firstValueMu.Lock() + if !r.firstValueDone && !r.firstSeenAt.IsZero() { + r.firstValueDone = true + r.timeToFirstValue.Record(ctx, time.Since(r.firstSeenAt).Seconds()) + if r.firstValueRecordFn != nil { + _ = r.firstValueRecordFn() // best-effort; a failed write just risks re-recording once on a later process, not a correctness bug + } + } + r.firstValueMu.Unlock() + } +``` + +- [ ] **Step 7: Wire this into `BuildProductMetrics`** + +In `internal/cliconfig/productmetrics.go`, after constructing `recorder` and before returning: + +```go + firstValueAlready, fvErr := productmetrics.LoadOrCreateFirstValueMarkerDefault() + // A failure here degrades to "track it anyway" (fvErr != nil implies + // firstValueAlready's zero value false) rather than disabling the whole + // pipeline — time_to_first_value is a nice-to-have signal, not + // load-bearing enough to fail product metrics setup entirely over. + firstSeenAt := time.Now() + if info, statErr := os.Stat(installIDFilePath(...)); statErr == nil { // see note below + firstSeenAt = info.ModTime() + } + recorder.EnableFirstValueTracking(firstSeenAt, fvErr == nil && firstValueAlready, func() error { + _, _, err := productmetrics.LoadOrCreateFirstValueMarkerDefault() + return err + }) +``` + +Note to implementer: `installIDFilePath(...)` is illustrative — `installid.go`'s `installIDRelPath` const plus `xdgconfig.UserStateDir` is how the REAL install-id file's path is computed; either export a small helper from `productmetrics` that returns this path (cleanest), or accept the simpler approximation of always using `time.Now()` as `firstSeenAt` when the install-id file's actual mtime isn't easily available at this call site — the metric's own doc comment already says an approximation is acceptable. Use your judgment on which is cleaner; either is acceptable, but document whichever you pick in the instrument's description string if it differs from what Step 6 already says. + +- [ ] **Step 8: Run the full package suite** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` and `cd ../../cliconfig && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/firstvalue.go \ + internal/adapter/productmetrics/firstvalue_test.go internal/cliconfig/productmetrics.go +git commit -m "feat(productmetrics): tool_calls_per_run + time_to_first_value" +``` + +--- + +### Task 6: Update `DryRunRecorder` to match + +**Files:** +- Modify: `internal/adapter/productmetrics/dryrun.go` +- Modify: `internal/adapter/productmetrics/dryrun_test.go` + +**Interfaces:** +- Produces: `DryRunRecorder` now also implements `port.RunAwareToolCallRecorder`, and logs the same new bounded fields (`had_tool_call`, `category`, `outcome`) the real `Recorder` would have recorded — the dry-run's whole purpose is showing EXACTLY what the real pipeline would send, so it must track every new field the real one does. + +- [ ] **Step 1: Write the failing test** + +```go +func TestDryRunRecorderLogsCategoryOutcomeAndHadToolCall(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__someserver__sensitive_tool"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + found := false + for _, args := range diag.allArgs() { // see note: capturingDiag needs a small extension to expose recorded args, not just messages, for this assertion — extend it minimally + for _, a := range args { + if s, ok := a.(string); ok && (s == "someserver" || s == "sensitive_tool") { + t.Fatalf("MCP server/tool name leaked in dry-run output: %q", s) + } + } + } + _ = found +} +``` + +Note to implementer: the existing `capturingDiag` fake (in `dryrun_test.go`) currently only captures `msg string`, not the `args ...any` — extend it minimally to also store `args` per call, since this test (and the privacy discipline this file exists to prove) needs to inspect them. + +- [ ] **Step 2: Run test to verify it fails, then implement** + +In `dryrun.go`, implement `ToolCallForRun` mirroring the real `Recorder`'s logic (category/outcome derivation), and update `ToolCall` to delegate to it with `runID=""`, same shape as Task 3's real `Recorder`: + +```go +func (d *DryRunRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + d.ToolCallForRun("", id, call, result, queued, took) +} + +func (d *DryRunRecorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + outcome := "success" + if result.IsError { + outcome = "error" + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1", + "category", toolCategory(call.Name), "outcome", outcome) +} + +var _ port.RunAwareToolCallRecorder = (*DryRunRecorder)(nil) +``` + +Update `Emit`'s `EvResult` case to also log `had_tool_call` (reuse whatever per-run tracking is simplest for the dry-run path — a lighter-weight version than the real `Recorder`'s is fine here, e.g. its own small `runs *runs`-shaped field, or simply omitting exact had_tool_call tracking in dry-run and logging `"had_tool_call", "unknown (dry-run does not track per-run state)"` if that's meaningfully simpler — use your judgment, document whichever you choose). + +- [ ] **Step 3: Run tests** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add internal/adapter/productmetrics/dryrun.go internal/adapter/productmetrics/dryrun_test.go +git commit -m "feat(productmetrics): DryRunRecorder mirrors the new category/outcome/had_tool_call fields" +``` + +--- + +### Task 7: mecak8s install-id ConfigMap + +**Files:** +- Create: `deploy/helm/mecak8s/templates/install-id-configmap.yaml` +- Modify: `deploy/helm/mecak8s/templates/deployment.yaml` (mount the id as an env var) +- Modify: `deploy/helm/mecak8s/chart_test.go` (or wherever this chart's existing tests live — extend, don't invent a new test file if one already renders/asserts this chart's templates) +- Modify: `cmd/mecak8s/observability.go` (read the env var instead of calling the local-file mechanism, when set) + +**Interfaces:** +- Produces: a `ConfigMap` named e.g. `{{ include "mecak8s.fullname" . }}-install-id` holding one key (`installId`), generated once via the `lookup`-based idiom and reused across `helm upgrade`; an env var `MECATL_PRODUCT_METRICS_INSTALL_ID` sourced from it, mounted into the `mecak8s` container. + +- [ ] **Step 1: Write the ConfigMap template** + +```yaml +# deploy/helm/mecak8s/templates/install-id-configmap.yaml +# +# Generates ONE stable install-id for this Helm release, reused across every +# replica and every `helm upgrade` — unlike a per-pod local file (which mecak8s +# cannot use at all: it runs storage-free, no PVC, per ADR 0048, and every pod +# restart would otherwise mint a fresh, never-reused id — the worst-case +# cardinality pattern for the product-metrics pipeline this feeds). The +# `lookup` guard is the standard Helm idiom for "generate once, keep stable on +# upgrade": if a ConfigMap of this name already exists in this release's +# namespace, its EXISTING value is reused verbatim; only a genuinely first +# `helm install` (or a deliberately deleted ConfigMap) mints a new one. +{{- $existing := lookup "v1" "ConfigMap" .Release.Namespace (printf "%s-install-id" (include "mecak8s.fullname" .)) }} +{{- $installID := "" }} +{{- if $existing }} +{{- $installID = index $existing.data "installId" }} +{{- else }} +{{- $installID = uuidv4 }} +{{- end }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "mecak8s.fullname" . }}-install-id + labels: + {{- include "mecak8s.labels" . | nindent 4 }} +data: + installId: {{ $installID | quote }} +``` + +(`mecak8s.fullname`/`mecak8s.labels` are illustrative — before writing this, check `deploy/helm/mecak8s/templates/_helpers.tpl` for the chart's REAL helper template names and use those, not invented ones.) + +- [ ] **Step 2: Mount it as an env var in `deployment.yaml`** + +In `deploy/helm/mecak8s/templates/deployment.yaml`, add to the container's `env:` list (find the existing `env:` block — confirmed at line 201 in the prior investigation): + +```yaml + - name: MECATL_PRODUCT_METRICS_INSTALL_ID + valueFrom: + configMapKeyRef: + name: {{ include "mecak8s.fullname" . }}-install-id + key: installId +``` + +- [ ] **Step 3: Read the env var in `cmd/mecak8s/observability.go`** + +Before the existing call to `cliconfig.BuildProductMetrics(...)`, add: + +```go + // mecak8s cannot use the local-file install-id mechanism the other three + // binaries share (storage-free, no PVC, per ADR 0048 — every pod restart + // would mint a fresh id, the worst-case cardinality pattern). Instead, a + // stable per-Helm-release id is provisioned via a ConfigMap (see + // deploy/helm/mecak8s/templates/install-id-configmap.yaml) and threaded + // through this env var. Empty (no chart-provisioned id, e.g. running the + // binary directly outside the chart) falls back to whatever + // BuildProductMetrics's own default local-file mechanism produces — + // which will still work, just without the "one stable id per k8s + // deployment" guarantee the chart provides. + installIDOverride := os.Getenv("MECATL_PRODUCT_METRICS_INSTALL_ID") +``` + +Thread `installIDOverride` into `cliconfig.BuildProductMetrics`'s call — this requires a SMALL signature addition to `BuildProductMetrics` (an optional `installIDOverride string` parameter, empty meaning "use the default local-file mechanism"): when non-empty, `BuildProductMetrics` skips `LoadOrCreateInstallIDDefault()` entirely and uses the override value directly as `Config.InstallID`. Update the OTHER three binaries' call sites to pass `""` (unaffected, unchanged behavior). + +- [ ] **Step 4: Update/extend the chart's existing test** + +Find the existing Helm chart test (`deploy/helm/mecak8s/chart_test.go`, confirmed to exist by the earlier `grep -rln readOnlyRootFilesystem` search) and add a case asserting the new `ConfigMap` template renders with a valid UUID in `data.installId`, and that the `Deployment` template's env var correctly references it via `configMapKeyRef`. + +- [ ] **Step 5: Run the chart tests and the `cmd/mecak8s`/`cliconfig` Go tests** + +Run: `cd deploy/helm/mecak8s && go test ./... -v` (if this is how chart_test.go is invoked — check its actual invocation mechanism, e.g. it may use the `helm` binary via `os/exec` or a Go Helm-templating library; follow whatever the EXISTING tests in this file already do) and `cd /Users/reyniero/work/mecatl/.claude/worktrees/product-metrics-otel/cmd/mecak8s && go test ./... -race -v` and `cd ../../internal/cliconfig && go test ./... -race -v`. +Expected: PASS + +- [ ] **Step 6: Run `task k8s:e2e` or equivalent if this repo has one (check Taskfile.yml for a k8s-specific e2e task) as an extra confidence check, given this touches the real Helm chart** + +If such a task exists, run it; if it requires a live kind cluster and is out of scope for a quick local check, note that in your report and rely on the chart_test.go coverage instead. + +- [ ] **Step 7: Commit** + +```bash +git add deploy/helm/mecak8s/templates/install-id-configmap.yaml deploy/helm/mecak8s/templates/deployment.yaml \ + deploy/helm/mecak8s/chart_test.go cmd/mecak8s/observability.go internal/cliconfig/productmetrics.go +git commit -m "feat(mecak8s): provision a stable per-release install-id via a Helm ConfigMap" +``` + +--- + +### Task 8: ADR + user-docs + PR description updates, final verification + +**Files:** +- Modify: `docs/adr/0319-product-metrics.md` +- Modify: `user-docs/building/what-you-get/observability.md` +- Modify: the open PR's description (via `gh pr edit`) + +**Interfaces:** none — documentation only, plus final verification. + +- [ ] **Step 1: Update the ADR's catalog table** + +Add rows for `mecatl.product.run_duration`, `mecatl.product.tool_calls_per_run`, `mecatl.product.time_to_first_value`; update `runs_completed`'s row to show its new `had_tool_call` attribute; update `tool_calls`'s row to show its new `category`/`outcome` attributes (removing the "no tool/MCP-server name label at all" claim, replacing it with an accurate description of the bounded category scheme). + +- [ ] **Step 2: Add a new ADR section documenting the reinstatement** + +Add a section (e.g. "## Reinstating `mecatl.install.id`, and the mecak8s ConfigMap") recording: why it was reinstated (sized, accepted cost — cite the actual $/month figures from this conversation), the `RunAwareToolCallRecorder` engine addition and why it's additive/non-breaking, and the mecak8s-specific ConfigMap mechanism and why the local-file approach cannot work there (storage-free, ADR 0048, pod churn). + +- [ ] **Step 3: Update `user-docs/building/what-you-get/observability.md`** + +Update the "What's collected" paragraph to include the new fields (tool category/outcome, had_tool_call, run duration, tool-calls-per-run, time-to-first-value) and to accurately state that an anonymous per-install identifier is now collected (reversing the prior "no ... identifier" framing) — be precise and honest here, this is the user-facing disclosure text's source of truth. + +- [ ] **Step 4: Update `internal/cliconfig/productmetrics.go`'s `ProductMetricsDisclosureNotice` string** + +This is the actual STARTUP disclosure text users see — it must also honestly reflect that a per-install identifier is now collected. Update its wording accordingly. + +- [ ] **Step 5: Run every gate** + +```bash +task lint +task test +task build +task docs +task site:build +``` +All must pass clean. + +- [ ] **Step 6: Update the PR description** + +Fetch the current PR body (`gh pr view --json body -q .body`), update the "Metrics catalog" section to reflect the new/changed instruments, add a short "Reinstating install.id" note explaining the reversal and its rationale (cost sizing, mecak8s ConfigMap mechanism), and push via `gh pr edit`. + +- [ ] **Step 7: Commit the doc changes** + +```bash +git add docs/adr/0319-product-metrics.md user-docs/building/what-you-get/observability.md internal/cliconfig/productmetrics.go +git commit -m "docs: document had_tool_call, tool category/outcome, and the install.id reinstatement" +git push +``` From a0cc1f99cb890ea6cefbf0270c9a2f7efd584e0f Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:02:24 -0400 Subject: [PATCH 27/47] feat(engine): add the optional RunAwareToolCallRecorder port capability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds port.RunAwareToolCallRecorder, an OPTIONAL ToolCallRecorder extension that additionally receives the calling run's RunID. The dispatcher's one ToolCallRecorder call site (engine/agent/dispatch.go execute) type-asserts for it and prefers ToolCallForRun when implemented, falling back to the plain ToolCall otherwise — purely additive, no existing ToolCallRecorder implementer is affected. Mirrors the HookApprovalLearner precedent in engine/port/hookrunner.go. Co-Authored-By: Claude Sonnet 5 --- engine/CHANGELOG.md | 5 + engine/agent/dispatch.go | 6 +- .../run_aware_tool_call_recorder_test.go | 110 ++++++++++++++++++ engine/api/port.txt | 2 + engine/port/log.go | 21 ++++ 5 files changed, 143 insertions(+), 1 deletion(-) create mode 100644 engine/agent/run_aware_tool_call_recorder_test.go diff --git a/engine/CHANGELOG.md b/engine/CHANGELOG.md index 3208644c6d..6116b9266e 100644 --- a/engine/CHANGELOG.md +++ b/engine/CHANGELOG.md @@ -28,6 +28,11 @@ The covered surface is the eight core packages (`session`, `governance`, `learni - **Session placement authority repair** — removes the orphan exported `session.PlacementSelector` protocol, adds persisted display-only `session.PlacementMetadata`, requires a valid `EnvironmentRef` at aggregate construction, and rejects direct engine runs whose live environment does not match the session identity. Changed (breaking, pre-v1 minor). +- `port.RunAwareToolCallRecorder`: an optional `ToolCallRecorder` extension + that additionally receives the calling run's `RunID`, letting a consumer + correlate a tool call to the run that made it. Purely additive — no + existing `ToolCallRecorder` implementer is affected. Added (minor). + - **Delegation artifact boundary (ADR 0288)** — adds the distinct `agent.ArtifactHandle` type, changes `agent.PreservedForkStore.Preserve` to key retained forks by that opaque handle rather than a physical root, and removes `Workspace`/`WinnerWorkspace` from `session.ParallelPayload`. Parallel results now expose an opaque preserved-artifact handle while physical fork roots remain private orchestration state. Changed (breaking, pre-v1 minor). - **Unified environment and placement identity** — adds `Revision` and `Valid` to `session.EnvironmentRef`, makes that exact `{Kind, ID, Revision}` value the runtime and durable placement identity, and removes the short-lived duplicate `session.PlacementRef`/`PlacementKind` types. Engine-created Subagent, Parallel, and Team child sessions now persist the identity carried by their `tool.Environment`; `port.ScheduleSpec` and `port.SessionDiscoveryMeta` replace workspace paths with the exact private environment identity, with schedules also retaining their trusted placement scope. Changed (breaking, pre-v1 minor). diff --git a/engine/agent/dispatch.go b/engine/agent/dispatch.go index 7baaae2347..819a543ee9 100644 --- a/engine/agent/dispatch.go +++ b/engine/agent/dispatch.go @@ -1290,7 +1290,11 @@ func (e *Engine) execute(ctx context.Context, r *Run, sess *session.Session, env res = session.RepairToolResult(res) if e.deps.ToolCallRecorder != nil { - e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + if aware, ok := e.deps.ToolCallRecorder.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(r.RunID(), sess.ID, c, res, queued, dur) + } else { + e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + } } e.emit(r, session.Event{Type: session.EvToolResult, Turn: turnIdx, ToolResult: ptr(res)}) diff --git a/engine/agent/run_aware_tool_call_recorder_test.go b/engine/agent/run_aware_tool_call_recorder_test.go new file mode 100644 index 0000000000..6b0db4dcac --- /dev/null +++ b/engine/agent/run_aware_tool_call_recorder_test.go @@ -0,0 +1,110 @@ +package agent_test + +import ( + "context" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/adapter/memfs" + "github.com/stacklok/mecatl/engine/adapter/mockllm" + "github.com/stacklok/mecatl/engine/agent" + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/engine/tool" +) + +// runAwareFakeRecorder implements BOTH port.ToolCallRecorder and the new +// port.RunAwareToolCallRecorder, recording which method the dispatcher chose. +type runAwareFakeRecorder struct { + plainCalls int + runAwareCalls int + lastRunID string +} + +func (f *runAwareFakeRecorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + f.plainCalls++ +} + +func (f *runAwareFakeRecorder) ToolCallForRun(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + f.runAwareCalls++ + f.lastRunID = runID +} + +// newToolCallingEngine builds an engine + session wired with a single Read tool +// call followed by a final text turn, mirroring TestFullCycle's mockllm script, +// with the given ToolCallRecorder injected. +func newToolCallingEngine(t *testing.T, rec interface { + ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) +}) (*agent.Engine, *session.Session) { + t.Helper() + read := &fakeTool{name: "Read", readOnly: true, + exec: func(_ context.Context, in session.ToolCall, _ tool.Workspace) (session.ToolResult, error) { + return session.NewToolResult(in.ID, "file contents"), nil + }} + cat := catalogWith(t, read) + + llm := mockllm.New( + mockllm.ChunksTurn( + mockllm.TextChunk("let me look"), + mockllm.ToolCallChunk(toolCall("c1", "Read", `{"path":"a.go"}`)), + mockllm.UsageChunk(session.Usage{InputTokens: 10, OutputTokens: 2}), + mockllm.DoneChunk(session.StopEndTurn), + ), + mockllm.ChunksTurn( + mockllm.TextChunk("all done"), + mockllm.UsageChunk(session.Usage{InputTokens: 5, OutputTokens: 3}), + mockllm.DoneChunk(session.StopEndTurn), + ), + ) + + clk := &fakeClock{t: time.Unix(0, 0)} + e := newEngine(agent.Deps{LLM: llm, Catalog: cat, Clock: clk, ToolCallRecorder: rec}) + sess := newSession(t, session.Limits{}) + return e, sess +} + +// driveOneToolCallingTurn drives the engine through the single tool-calling +// turn scripted by newToolCallingEngine and returns the enclosing Run's RunID. +func driveOneToolCallingTurn(t *testing.T, e *agent.Engine, sess *session.Session) string { + t.Helper() + ws := memfs.NewWorkspace("/ws") + r := e.Run(context.Background(), sess, agent.EnvForWS(ws, nil), agent.RunRequest{Text: "look at a.go"}) + runID := r.RunID() + drain(r) + return runID +} + +// TestExecutePrefersRunAwareToolCallRecorderWhenImplemented pins that the +// dispatcher, at its one ToolCallRecorder call site, calls ToolCallForRun +// (never both) when the injected recorder implements it, passing the SAME +// RunID the enclosing Run already carries — and falls back to the plain +// ToolCall for a recorder that does not implement the richer interface +// (every existing ToolCallRecorder implementer is unaffected). +func TestExecutePrefersRunAwareToolCallRecorderWhenImplemented(t *testing.T) { + rec := &runAwareFakeRecorder{} + e, sess := newToolCallingEngine(t, rec) + runID := driveOneToolCallingTurn(t, e, sess) + + if rec.plainCalls != 0 { + t.Errorf("plainCalls = %d, want 0 (RunAwareToolCallRecorder must be preferred)", rec.plainCalls) + } + if rec.runAwareCalls == 0 { + t.Fatal("runAwareCalls = 0, want at least 1") + } + if rec.lastRunID != runID { + t.Errorf("lastRunID = %q, want %q (the enclosing Run's own id)", rec.lastRunID, runID) + } +} + +// TestExecuteFallsBackToPlainToolCallRecorder is a regression guard: a +// recorder implementing ONLY port.ToolCallRecorder (not the richer +// RunAwareToolCallRecorder) must keep working exactly as before, using the +// package's existing plain recordingLogger fixture. +func TestExecuteFallsBackToPlainToolCallRecorder(t *testing.T) { + logger := &recordingLogger{} + e, sess := newToolCallingEngine(t, logger) + driveOneToolCallingTurn(t, e, sess) + + if logger.calls != 1 { + t.Fatalf("logger recorded %d tool calls, want 1", logger.calls) + } +} diff --git a/engine/api/port.txt b/engine/api/port.txt index 2e7c81c105..7a77f17637 100644 --- a/engine/api/port.txt +++ b/engine/api/port.txt @@ -31,6 +31,7 @@ ifacemethod func (PrunableStore).Delete(ctx context.Context, id session.SessionID) error ifacemethod func (PrunableStore).List(ctx context.Context) ([]StoredSession, error) ifacemethod func (RetryDispositionError).RetryDisposition() session.RetryDisposition + ifacemethod func (RunAwareToolCallRecorder).ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued time.Duration, took time.Duration) ifacemethod func (ScheduleCreator).Create(ctx context.Context, s Schedule) error ifacemethod func (ScheduleManager).CreateSchedule(ctx context.Context, spec ScheduleSpec) (Schedule, error) ifacemethod func (ScheduleManager).DeleteSchedule(ctx context.Context, name string) error @@ -189,6 +190,7 @@ type PrunableStore interface{Delete(ctx context.Context, id session.SessionID) e type ReadOptions struct{Limit int; Follow bool} type RetryDisposition = session.RetryDisposition type RetryDispositionError interface{RetryDisposition() session.RetryDisposition; error} +type RunAwareToolCallRecorder interface{ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued time.Duration, took time.Duration)} type Schedule struct{Spec ScheduleSpec; State ScheduleState} type ScheduleCreator interface{Create(ctx context.Context, s Schedule) error} type ScheduleFire struct{ID string; ScheduleName string; SessionID session.SessionID; FiredAt time.Time; StartedAt time.Time; ProgressAt time.Time; Deadline time.Time; Stop session.StopReason; Err string} diff --git a/engine/port/log.go b/engine/port/log.go index 881707408a..38784833d0 100644 --- a/engine/port/log.go +++ b/engine/port/log.go @@ -39,3 +39,24 @@ type ToolCallRecorder interface { // Clock is injected. ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) } + +// RunAwareToolCallRecorder is an OPTIONAL capability a ToolCallRecorder may +// ALSO implement to additionally receive the RunID of the run that made the +// call (the same opaque per-run correlation id carried on session.Event.RunID, +// ADR 0249) — the one thing ToolCall's signature cannot express, since a +// SessionID can span many sequential runs over a session's lifetime and +// ToolCall alone gives no way to tell which run a given call belongs to. +// +// The engine TYPE-ASSERTS this interface on Deps.ToolCallRecorder and calls +// ToolCallForRun INSTEAD OF ToolCall (never both) when implemented — so a +// recorder that implements only the base ToolCallRecorder is wholly +// unaffected (no method added to ToolCallRecorder: that would be a breaking +// change, mirroring the HookApprovalLearner precedent in hookrunner.go). +type RunAwareToolCallRecorder interface { + // ToolCallForRun is ToolCall's signature plus the leading runID — the + // same value the enclosing Run stamps onto every session.Event.RunID it + // emits. Consumers that need to correlate a tool call to the run that + // made it (e.g. "did this run have at least one successful tool call") + // use this instead of ToolCall. + ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) +} From ba9fa375c017aa32b5068db7e40818df4080f99b Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:08:01 -0400 Subject: [PATCH 28/47] feat(productmetrics): reinstate mecatl.install.id after sizing its cardinality cost Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/config.go | 6 +++++ internal/adapter/productmetrics/installid.go | 10 ++++---- internal/adapter/productmetrics/provider.go | 23 +++++++++++-------- .../adapter/productmetrics/provider_test.go | 5 ++-- internal/cliconfig/productmetrics.go | 17 +++++++------- 5 files changed, 35 insertions(+), 26 deletions(-) diff --git a/internal/adapter/productmetrics/config.go b/internal/adapter/productmetrics/config.go index 2a871b113e..db188ad9ff 100644 --- a/internal/adapter/productmetrics/config.go +++ b/internal/adapter/productmetrics/config.go @@ -87,4 +87,10 @@ type Config struct { Binary Binary // Version is the mecatl build version (resource attribute service.version). Version string + // InstallID is this process's persisted (or externally-provisioned, for + // mecak8s — see Task 7) anonymous install identifier. Reinstated as a + // resource attribute after being sized and accepted: ~$1,930/month at + // 100K installs under worst-case 24/7 uptime on the actual AMP pricing + // model (see the ADR's updated cost-analysis section, Task 8). + InstallID string } diff --git a/internal/adapter/productmetrics/installid.go b/internal/adapter/productmetrics/installid.go index f4e8282e7b..e98c8ed2ec 100644 --- a/internal/adapter/productmetrics/installid.go +++ b/internal/adapter/productmetrics/installid.go @@ -25,12 +25,10 @@ const installIDRelPath = "mecatl/telemetry-id" // one-time disclosure notice. readFile/writeFile/mkdirAll are injected for // testing; LoadOrCreateInstallIDDefault binds the real filesystem. // -// The returned id is deliberately never threaded into any exported metric -// attribute or resource (see provider.go): this pipeline's destination is a -// Prometheus-remote-write backend, where a per-install identifier would -// become a permanent, unbounded-cardinality label on every instrument. The -// file still exists purely as a local first-run marker for the disclosure -// notice — its actual UUID value has no other consumer. +// The returned id IS threaded into an exported resource attribute (see +// provider.go) — this package makes no attempt to keep the id local-only; +// that was a prior, now-reverted design (see git history / the ADR's +// cost-analysis section for why it was reinstated). func LoadOrCreateInstallID( env xdgconfig.ResolveEnv, readFile func(string) ([]byte, error), diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go index bf7ee2cf75..5afb2d4f80 100644 --- a/internal/adapter/productmetrics/provider.go +++ b/internal/adapter/productmetrics/provider.go @@ -53,15 +53,17 @@ func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { // or the exporter tries TLS against a plaintext listener and every // export fails. The real production endpoint is always https://. // - // Deliberately NOT included: any per-install identifier. This pipeline's - // destination is a Prometheus-remote-write backend (stacklok/infra#5604), - // where every resource attribute becomes a permanent label on EVERY - // instrument's time series — attaching a random per-install value here - // would multiply active-series count by (installs × instrument count), - // an unbounded-cardinality cost with no bound as adoption grows. Only - // mecatl.binary (a small closed enum) is attached; unique-install - // counting is approximated from heartbeat volume instead (see - // installid.go's doc comment). + // mecatl.install.id is a per-install random UUID, deliberately attached + // as a resource attribute (so it flattens onto every instrument this + // provider exports). This was removed once (see git history) over + // unbounded-cardinality concerns on the Prometheus-remote-write + // destination (stacklok/infra#5604), then reinstated after the actual + // cost was sized against real AMP pricing and accepted — see the ADR's + // cost-analysis section for the numbers. mecak8s provisions this value + // differently (a stable per-Helm-release ConfigMap, not this package's + // local install-id file — see internal/cliconfig's mecak8s wiring and + // deploy/helm/mecak8s/templates/install-id-configmap.yaml), since a + // pod-local file would mint a new id on every pod restart. composite, err := providers.NewCompositeProvider(ctx, providers.WithServiceName("mecatl"), providers.WithServiceVersion(cfg.Version), @@ -70,7 +72,8 @@ func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { providers.WithInsecure(strings.HasPrefix(endpoint, "http://")), providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), providers.WithCustomAttributes(map[string]string{ - "mecatl.binary": string(cfg.Binary), + "mecatl.install.id": cfg.InstallID, + "mecatl.binary": string(cfg.Binary), }), ) if err != nil { diff --git a/internal/adapter/productmetrics/provider_test.go b/internal/adapter/productmetrics/provider_test.go index e8c4062490..62c736f335 100644 --- a/internal/adapter/productmetrics/provider_test.go +++ b/internal/adapter/productmetrics/provider_test.go @@ -35,8 +35,9 @@ func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { defer func() { endpoint = origEndpoint }() p, err := NewProvider(context.Background(), Config{ - Binary: BinaryMecated, - Version: "test", + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", }) if err != nil { t.Fatalf("NewProvider: %v", err) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index b7d4ce2601..f8c32f3eb9 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -81,19 +81,20 @@ func BuildProductMetrics( return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil } - // LoadOrCreateInstallIDDefault still runs (and persists its file) purely - // to detect first-run for the disclosure notice below — the returned id - // value itself is deliberately discarded, never threaded to NewProvider: - // see provider.go's doc comment on why a per-install identifier must - // never become a Prometheus-remote-write label. - _, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + // LoadOrCreateInstallIDDefault persists (or reads back) this process's + // local install-id file and reports firstRun for the disclosure notice + // below. Reinstated as a real, exported resource attribute (see + // provider.go's doc comment) after its cardinality cost was sized and + // accepted. + installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() if err != nil { return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) } provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ - Binary: binary, - Version: version, + Binary: binary, + Version: version, + InstallID: installID, }) if err != nil { return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: provider: %w", err) From 722c7cf81c1ee50c1227a86acf46dd87cfaffc9d Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:24:06 -0400 Subject: [PATCH 29/47] feat(productmetrics): had_tool_call, tool category/outcome via RunAwareToolCallRecorder Consume the new optional port.RunAwareToolCallRecorder so this adapter can correlate a tool call to the run that made it, and derive two new bounded facts from that correlation: - runs_completed gains had_tool_call ("true"/"false"): whether the run made at least one SUCCESSFUL tool call. An errored-only run is false - the product definition is "this run took an action", not "attempted one". - tool_calls gains category + outcome. outcome is {success,error}; category is the closed-set projection in toolCategory. The prior subagent/team per-run dedup (runFamiliesUsed/usedFamilies/ firstInRun/clearRun) is unified into ONE mutex-guarded per-run record, perRunTracker/perRunState, which the later run_duration and tool_calls_per_run work extends. Its lock discipline is the point: every method does its map lookup AND its field mutation as a single critical section and no *perRunState escapes a locked region, because Emit (per event) and ToolCallForRun (per tool call) run concurrently on a fan-out run. finish() returns a value copy so the had_tool_call read at EvResult is race-free, and clears the entry so the map stays bounded to live runs. DEVIATION from the plan's stated approach, flagged for review: the plan specified deriving category from the structural mcp__ prefix check ALONE, "never a maintained allowlist". That design emits any non-mcp__ name verbatim, which fails this package's existing privacy-guard test (it drives ToolCall with Name: "secret-tool-name-marker" and asserts no marker reaches an attribute value) and would put an unbounded label on the counter - a catalog is not only built-ins plus MCP tools, since an agent def, a learned skill, or a future extension seam can register a name derived from operator or model input. toolCategory therefore keeps the mcp__ prefix bucket AND gates verbatim names through builtinToolCategories, falling back to the single literal "other". A new built-in showing up as "other" is the visible, harmless prompt to add a line; a leak is impossible either way. DryRunRecorder is brought into lockstep in the same commit (part of the plan's Task 6): an audit path that understates what the real Recorder sends defeats the purpose it exists for. Its remaining Task 6 work (the later histograms) is untouched. Co-Authored-By: Claude Sonnet 5 --- .../adapter/productmetrics/bounded_test.go | 80 ++++++- internal/adapter/productmetrics/dryrun.go | 74 ++++-- .../adapter/productmetrics/dryrun_test.go | 49 ++++ internal/adapter/productmetrics/metrics.go | 225 ++++++++++++------ .../adapter/productmetrics/metrics_test.go | 82 +++++++ internal/adapter/productmetrics/toolcall.go | 118 ++++++++- .../adapter/productmetrics/toolcall_test.go | 167 +++++++++++++ 7 files changed, 679 insertions(+), 116 deletions(-) diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go index f226f2f6c7..52884141d9 100644 --- a/internal/adapter/productmetrics/bounded_test.go +++ b/internal/adapter/productmetrics/bounded_test.go @@ -6,6 +6,7 @@ import ( "testing" "time" + "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/sdk/metric/metricdata" "github.com/stacklok/mecatl/engine/session" @@ -16,21 +17,30 @@ import ( // must add it here explicitly — the same "closed set is a reviewed // decision" discipline as internal/adapter/telemetry's attrRole. var allowedAttributeKeys = map[string]bool{ - attrStop: true, - attrKind: true, - attrFeature: true, - attrProvider: true, - attrMode: true, + attrStop: true, + attrKind: true, + attrFeature: true, + attrProvider: true, + attrMode: true, + attrHadToolCall: true, + attrCategory: true, + attrOutcome: true, } // sensitiveMarkers are strings injected into every field the Recorder must -// NEVER read. If any of these ever shows up in a collected metric name or -// attribute value, something started reading a field it shouldn't. +// NEVER read, or (for the tool-name markers) must only ever read through a +// closed-set projection. If any of these ever shows up in a collected metric +// name or attribute value, something started emitting a field it shouldn't. var sensitiveMarkers = []string{ "sensitive-session-id-marker", "secret-tool-name-marker", "secret-tool-content-marker", "secret-error-text-marker", + // The MCP server + remote tool names inside a namespaced mcp__ tool name: + // operator-chosen free text, which must be bucketed under the single + // literal "mcp" rather than emitted. + "evilserver", + "leak_this_name", } func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T) { @@ -55,6 +65,19 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T session.ToolResult{Content: "secret-tool-content-marker", IsError: true}, 10*time.Millisecond, 20*time.Millisecond, ) + // The run-aware path, with an MCP-namespaced name whose server and remote + // tool halves are both operator-chosen free text. + r.ToolCallForRun( + "run-x", + session.SessionID("sensitive-session-id-marker"), + session.ToolCall{Name: "mcp__evilserver__leak_this_name"}, + session.ToolResult{Content: "secret-tool-content-marker"}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-x", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) r.Heartbeat(FeatureSnapshot{ Memory: true, Guardrails: true, MCP: true, Scheduling: true, Provider: ProviderOther, Mode: ModeK8s, @@ -88,15 +111,29 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T if !allowedAttributeKeys[key] { t.Errorf("metric %s carries attribute key %q, not in allowedAttributeKeys", md.Name, key) } + // Every value must be a STRING: a non-string value would + // slip past the sensitive-substring walk below with an + // empty AsString(), making this guard vacuous for it. + if kv.Value.Type() != attribute.STRING { + t.Errorf("metric %s attribute %q has value type %v, want STRING", md.Name, key, kv.Value.Type()) + } assertNoSensitiveSubstring(t, kv.Value.AsString()) } } - // mecatl.product.tool_calls carries NO attributes at all — the - // strongest form of "no tool identity ever attaches." + // mecatl.product.tool_calls carries exactly the two bounded + // tool-call keys — the category value can only ever be a built-in + // tool's own name, "mcp", or "other" (see toolCategory), so no + // tool identity beyond mecatl's own fixed catalog can attach. if md.Name == "mecatl.product.tool_calls" { for _, dp := range sum.DataPoints { - if dp.Attributes.Len() != 0 { - t.Errorf("mecatl.product.tool_calls data point carries %d attributes, want 0: %v", + if _, ok := dp.Attributes.Value(attrCategory); !ok { + t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrCategory, dp.Attributes) + } + if _, ok := dp.Attributes.Value(attrOutcome); !ok { + t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrOutcome, dp.Attributes) + } + if dp.Attributes.Len() != 2 { + t.Errorf("mecatl.product.tool_calls data point carries %d attributes, want exactly 2: %v", dp.Attributes.Len(), dp.Attributes) } } @@ -105,6 +142,27 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T } } +// TestToolCategoryOnlyEmitsClosedSetValues is the direct unit-level guard on +// the one projection that reads a tool name: whatever it is fed, the output +// must be a member of builtinToolCategories ∪ {"mcp", "other"}. +func TestToolCategoryOnlyEmitsClosedSetValues(t *testing.T) { + inputs := []string{ + "", "Read", "Bash", "mcp__evilserver__leak_this_name", "mcp__", "mcp_", + "secret-tool-name-marker", "MCP__X__Y", "read", "Read ", + "agent-def-derived-name", strings.Repeat("x", 4096), + } + for _, in := range inputs { + got := toolCategory(in) + if got == categoryMCP || got == categoryOther { + continue + } + if !builtinToolCategories[got] { + t.Errorf("toolCategory(%q) = %q, which is outside the closed set (builtins ∪ {%q, %q})", + in, got, categoryMCP, categoryOther) + } + } +} + func assertNoSensitiveSubstring(t *testing.T, s string) { t.Helper() for _, marker := range sensitiveMarkers { diff --git a/internal/adapter/productmetrics/dryrun.go b/internal/adapter/productmetrics/dryrun.go index a5841c6d0f..c47117381c 100644 --- a/internal/adapter/productmetrics/dryrun.go +++ b/internal/adapter/productmetrics/dryrun.go @@ -8,54 +8,68 @@ import ( "github.com/stacklok/mecatl/engine/session" ) -// DryRunRecorder implements the same two ports as Recorder (port.EventSink + -// port.ToolCallRecorder) but logs every would-be observation via an injected -// port.Diagnostics instead of exporting it over OTLP — the -// --product-metrics-dry-run audit path, so a skeptical operator can see -// exactly what this pipeline would have sent without trusting the docs. It -// logs ONLY the same bounded fields Recorder ever reads (event type, stop -// reason, token counts by kind, feature/provider/mode enum values) — never a -// tool name, session id, or free-text content, mirroring Recorder's own +// DryRunRecorder implements the same ports as Recorder (port.EventSink + +// port.ToolCallRecorder + port.RunAwareToolCallRecorder) but logs every +// would-be observation via an injected port.Diagnostics instead of exporting +// it over OTLP — the --product-metrics-dry-run audit path, so a skeptical +// operator can see exactly what this pipeline would have sent without +// trusting the docs. It logs ONLY the same bounded fields Recorder ever reads +// (event type, stop reason, had_tool_call, the closed-set tool category and +// outcome, token counts by kind, feature/provider/mode enum values) — never a +// session id, a raw tool name, or free-text content, mirroring Recorder's own // privacy discipline exactly. +// +// The audit path must stay in LOCKSTEP with Recorder: an attribute Recorder +// attaches but DryRunRecorder omits makes this surface understate what is +// sent, which is the one thing it exists to rule out. type DryRunRecorder struct { - diag port.Diagnostics + diag port.Diagnostics + perRun *perRunTracker } // Compile-time interface checks. var ( - _ port.EventSink = (*DryRunRecorder)(nil) - _ port.ToolCallRecorder = (*DryRunRecorder)(nil) + _ port.EventSink = (*DryRunRecorder)(nil) + _ port.ToolCallRecorder = (*DryRunRecorder)(nil) + _ port.RunAwareToolCallRecorder = (*DryRunRecorder)(nil) ) // NewDryRunRecorder builds a DryRunRecorder over the given Diagnostics sink. func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder { - return &DryRunRecorder{diag: diag} + return &DryRunRecorder{diag: diag, perRun: newPerRunTracker()} } -// Emit logs the bounded event type (and, for EvResult, the stop reason and -// token counts by kind) — the exact same fields Recorder.Emit reads. +// Emit logs the bounded event type (and, for EvResult, the stop reason, +// had_tool_call, and token counts by kind) — the exact same fields +// Recorder.Emit reads. func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { switch ev.Type { case session.EvSessionInit: d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") case session.EvResult: - d.emitResult(ctx, ev.Result) + d.emitResult(ctx, ev.Result, d.perRun.finish(ev.RunID)) case session.EvSubagentStart: - d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") + if d.perRun.markFamilyUsed(ev.RunID, familySubagent) { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") + } case session.EvTeamStart: - d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record team_used+1") + if d.perRun.markFamilyUsed(ev.RunID, familyTeam) { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record team_used+1") + } } } -func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload) { +func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload, st perRunState) { if res == nil { d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed", - "stop", string(session.StopNone)) + "stop", string(session.StopNone), + attrHadToolCall, st.hadToolCall) return } u := res.Usage d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed + tokens", "stop", string(res.Stop), + attrHadToolCall, st.hadToolCall, "input_tokens", u.InputTokens, "output_tokens", u.OutputTokens, "cache_read_tokens", u.CacheReadTokens, @@ -63,10 +77,24 @@ func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayl "reasoning_tokens", u.ReasoningTokens) } -// ToolCall logs only that a call happened — no name, no session id, no -// content, no duration — matching Recorder.ToolCall's restraint exactly. -func (d *DryRunRecorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { - d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1") +// ToolCall logs the run-less form, matching Recorder.ToolCall's delegation. +func (d *DryRunRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + d.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun logs the two bounded tool-call attributes Recorder attaches — +// the closed-set category (never the raw name for anything outside mecatl's +// own catalog, never an MCP server/tool name) and the outcome — and tallies +// the same per-run state, matching Recorder.ToolCallForRun exactly. +func (d *DryRunRecorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + outcome := outcomeSuccess + if result.IsError { + outcome = outcomeError + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1", + attrCategory, toolCategory(call.Name), + attrOutcome, outcome) + d.perRun.markToolCall(runID, result.IsError) } // Heartbeat logs the closed-enum feature/provider/mode signal, matching diff --git a/internal/adapter/productmetrics/dryrun_test.go b/internal/adapter/productmetrics/dryrun_test.go index 1df9009765..be7fff9998 100644 --- a/internal/adapter/productmetrics/dryrun_test.go +++ b/internal/adapter/productmetrics/dryrun_test.go @@ -52,6 +52,55 @@ func TestDryRunRecorderImplementsPorts(_ *testing.T) { r := NewDryRunRecorder(diag) var _ port.EventSink = r var _ port.ToolCallRecorder = r + var _ port.RunAwareToolCallRecorder = r +} + +// TestDryRunRecorderMirrorsRecorderToolCallAttributes pins the lockstep +// contract: the audit path must log the SAME bounded attributes Recorder +// attaches (an audit surface that understates what is sent defeats its own +// purpose), and the category must still be the closed-set projection — never +// the raw name, never an MCP server/tool name. +func TestDryRunRecorderMirrorsRecorderToolCallAttributes(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.ToolCallForRun("run-1", session.SessionID("s"), + session.ToolCall{Name: "mcp__evilserver__leak_this_name"}, + session.ToolResult{IsError: true}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + if len(diag.args) != 2 { + t.Fatalf("got %d logged lines, want 2: %v", len(diag.lines), diag.lines) + } + if !hasArg(diag.args[0], attrCategory, categoryMCP) { + t.Errorf("tool_calls dry-run log args = %v, want %s=%s", diag.args[0], attrCategory, categoryMCP) + } + if !hasArg(diag.args[0], attrOutcome, outcomeError) { + t.Errorf("tool_calls dry-run log args = %v, want %s=%s", diag.args[0], attrOutcome, outcomeError) + } + if !hasArg(diag.args[1], attrHadToolCall, false) { + t.Errorf("runs_completed dry-run log args = %v, want %s=false (the only tool call errored)", diag.args[1], attrHadToolCall) + } + for _, args := range diag.args { + for _, a := range args { + if s, ok := a.(string); ok && (strings.Contains(s, "evilserver") || strings.Contains(s, "leak_this_name")) { + t.Errorf("dry-run log leaked an MCP server/tool name: %v", args) + } + } + } +} + +// hasArg reports whether a Diagnostics key/value arg slice carries key=want. +func hasArg(args []any, key string, want any) bool { + for i := 0; i+1 < len(args); i += 2 { + if k, ok := args[i].(string); ok && k == key && args[i+1] == want { + return true + } + } + return false } // TestDryRunRecorderResultNeverLeaksFreeText covers the EvResult branch (not diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index fb8dd39b2d..5c4b999df5 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -3,6 +3,7 @@ package productmetrics import ( "context" "fmt" + "strconv" "sync" "go.opentelemetry.io/otel/attribute" @@ -16,21 +17,26 @@ import ( const meterName = "github.com/stacklok/mecatl/internal/adapter/productmetrics" // Attribute keys. Every value ever attached under these keys is drawn from a -// bounded closed set (session.StopReason, the fixed token-kind strings, or -// this package's own Feature/ProviderFamily/DeploymentMode enums) — never a -// session id, model id, tool name, or free text. +// bounded closed set (session.StopReason, the fixed token-kind strings, this +// package's own Feature/ProviderFamily/DeploymentMode enums, or "true"/"false") +// — never a session id, model id, or free text. The tool_calls instrument's +// two keys live with their closed-set projection in toolcall.go. const ( - attrStop = "stop" - attrKind = "kind" - attrFeature = "feature" - attrProvider = "family" - attrMode = "mode" + attrStop = "stop" + attrKind = "kind" + attrFeature = "feature" + attrProvider = "family" + attrMode = "mode" + attrHadToolCall = "had_tool_call" ) // Recorder is the product-metrics adapter: it implements port.EventSink -// (this file) and port.ToolCallRecorder (toolcall.go), deriving ONLY the -// bounded counts in the design's catalog. It never reads a tool name, -// session id, model id, or any free-text field. +// (this file) and port.ToolCallRecorder + port.RunAwareToolCallRecorder +// (toolcall.go), deriving ONLY the bounded counts in the design's catalog. It +// never reads a session id, model id, result content, or any free-text field; +// the one thing it reads from a tool call is its NAME, and only to map it +// through the closed-set projection in toolcall.go (toolCategory), which can +// emit nothing but a built-in tool's own name, "mcp", or "other". type Recorder struct { heartbeat metric.Int64Counter featureEnabled metric.Int64Counter @@ -43,29 +49,127 @@ type Recorder struct { subagentUsed metric.Int64Counter teamUsed metric.Int64Counter - // runFamiliesUsed dedups subagentUsed/teamUsed to their documented + // perRun holds the bounded per-live-run facts this package derives across + // the Emit/ToolCallForRun boundary. See perRunTracker. + perRun *perRunTracker +} + +// perRunState is the bounded set of facts tracked for ONE live run, keyed by +// the loop-stamped session.Event.RunID (an opaque per-run correlation id, ADR +// 0249 — never a session id, tool name, or free-text field, so this package's +// no-PII invariant holds). Every field is a count or a boolean derived from a +// closed vocabulary; nothing here is ever attached as an attribute VALUE. +type perRunState struct { + // subagentSeen/teamSeen dedup subagentUsed/teamUsed to their documented // "at least once per run" semantics: a run's first EvSubagentStart or // EvTeamStart increments the counter, later ones in the SAME run (e.g. a - // fan-out of concurrent Subagent calls) do not. Keyed by the loop-stamped - // session.Event.RunID (opaque, not a session id) — never a tool name, - // session id, or free-text field, preserving this package's no-PII - // invariant. Bounded to concurrently-live runs: each run's entry is - // cleared on its EvResult. - mu sync.Mutex - runFamiliesUsed map[string]usedFamilies + // fan-out of concurrent Subagent calls) do not. + subagentSeen bool + teamSeen bool + + // hadToolCall records whether the run made at least one SUCCESSFUL tool + // call — the product definition of "this run took an action", read at + // EvResult time as the runs_completed had_tool_call attribute. + hadToolCall bool + + // toolCallCount is the run's total tool calls (successful or not). It is + // tallied here but not yet published as an instrument; the tool-calls-per-run + // distribution is a later task in this plan. + toolCallCount int64 +} + +// perRunTracker guards the live-run state map. Bounded to concurrently-live +// runs: a run's entry is created lazily on its first observed fact and dropped +// on its EvResult, so the map never grows across a process's lifetime. +// +// CONCURRENCY: every method below performs its map lookup AND its field +// mutation as ONE critical section under mu, and no *perRunState pointer ever +// escapes a locked region. Callers therefore cannot race on a state's fields: +// Emit (per event) and ToolCallForRun (per tool call) run concurrently on a +// fan-out run, and the only shape that is provably safe is "the lock covers +// both halves". +type perRunTracker struct { + mu sync.Mutex + states map[string]*perRunState +} + +func newPerRunTracker() *perRunTracker { + return &perRunTracker{states: make(map[string]*perRunState)} +} + +// stateLocked returns runID's live state, creating it if absent. The caller +// MUST hold t.mu, and must not retain the pointer past the critical section. +func (t *perRunTracker) stateLocked(runID string) *perRunState { + st, ok := t.states[runID] + if !ok { + st = &perRunState{} + t.states[runID] = st + } + return st +} + +// markFamilyUsed reports whether this is the first time, within the run +// identified by runID, that family has been observed — marking it seen as a +// side effect. An empty runID (no run context to dedup against) always counts, +// matching the pre-dedup behavior. +func (t *perRunTracker) markFamilyUsed(runID string, family delegationFamily) bool { + if runID == "" { + return true + } + t.mu.Lock() + defer t.mu.Unlock() + st := t.stateLocked(runID) + seen := &st.subagentSeen + if family == familyTeam { + seen = &st.teamSeen + } + if *seen { + return false + } + *seen = true + return true +} + +// markToolCall tallies one tool call against runID. An empty runID (a caller +// on the base port.ToolCallRecorder path, with no run to correlate against) is +// tracked nowhere — its call is still counted on the tool_calls instrument, +// but it can contribute to no run's had_tool_call. +func (t *perRunTracker) markToolCall(runID string, errored bool) { + if runID == "" { + return + } + t.mu.Lock() + defer t.mu.Unlock() + st := t.stateLocked(runID) + st.toolCallCount++ + if !errored { + st.hadToolCall = true + } } -// usedFamilies tracks, per live run, which delegation families have already -// been counted at least once. -type usedFamilies struct { - subagent bool - team bool +// finish drops runID's live state at EvResult and returns a COPY of what was +// there (the zero value if the run recorded nothing — e.g. a run with no tool +// calls and no delegation-family use). Returning a copy, not the pointer, +// keeps every field read outside the lock race-free. +func (t *perRunTracker) finish(runID string) perRunState { + if runID == "" { + return perRunState{} + } + t.mu.Lock() + defer t.mu.Unlock() + st, ok := t.states[runID] + if !ok { + return perRunState{} + } + delete(t.states, runID) + return *st } // Compile-time interface checks. var ( - _ port.EventSink = (*Recorder)(nil) - _ port.ToolCallRecorder = (*Recorder)(nil) + _ port.EventSink = (*Recorder)(nil) + _ port.ToolCallRecorder = (*Recorder)(nil) + _ port.RunAwareToolCallRecorder = (*Recorder)(nil) ) // NewRecorder constructs every instrument from the given MeterProvider. It @@ -73,7 +177,7 @@ var ( // is fallible. func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { meter := mp.Meter(meterName) - r := &Recorder{runFamiliesUsed: make(map[string]usedFamilies)} + r := &Recorder{perRun: newPerRunTracker()} var err error if r.heartbeat, err = meter.Int64Counter("mecatl.product.heartbeat", @@ -97,11 +201,11 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { return nil, fmt.Errorf("productmetrics: sessions_started counter: %w", err) } if r.runsCompleted, err = meter.Int64Counter("mecatl.product.runs_completed", - metric.WithDescription("Total runs completed, by bounded stop reason.")); err != nil { + metric.WithDescription("Total runs completed, by bounded stop reason and whether the run made at least one successful tool call.")); err != nil { return nil, fmt.Errorf("productmetrics: runs_completed counter: %w", err) } if r.toolCalls, err = meter.Int64Counter("mecatl.product.tool_calls", - metric.WithDescription("Total tool calls executed (no tool identity attached).")); err != nil { + metric.WithDescription(`Total tool calls executed, by bounded category (a built-in tool's own name, the single value "mcp" for any MCP-server tool, or "other") and outcome.`)); err != nil { return nil, fmt.Errorf("productmetrics: tool_calls counter: %w", err) } if r.tokens, err = meter.Int64Counter("mecatl.product.tokens", @@ -124,22 +228,23 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { // ONLY ev.Type, ev.RunID, ev.Result.Stop, and ev.Result.Usage — never a // session id, model id/alias, tool name, or any free-text field // (ev.Result.Text/Error are never touched). ev.RunID is an opaque per-run -// correlation id (ADR 0249), not a session id, and is used ONLY to dedup -// subagentUsed/teamUsed to one count per run (see firstInRun); it never -// becomes an attribute value. +// correlation id (ADR 0249), not a session id, and is used ONLY as the key of +// the per-run tracker (dedup'ing subagentUsed/teamUsed and resolving +// had_tool_call at EvResult); it never becomes an attribute value. func (r *Recorder) Emit(ctx context.Context, ev session.Event) { switch ev.Type { case session.EvSessionInit: r.sessionsStarted.Add(ctx, 1) case session.EvResult: - r.recordResult(ctx, ev.Result) - r.clearRun(ev.RunID) + // finish both reads and clears the run's state, so the had_tool_call + // resolution and the bounded-map cleanup are one step. + r.recordResult(ctx, ev.Result, r.perRun.finish(ev.RunID)) case session.EvSubagentStart: - if r.firstInRun(ev.RunID, familySubagent) { + if r.perRun.markFamilyUsed(ev.RunID, familySubagent) { r.subagentUsed.Add(ctx, 1) } case session.EvTeamStart: - if r.firstInRun(ev.RunID, familyTeam) { + if r.perRun.markFamilyUsed(ev.RunID, familyTeam) { r.teamUsed.Add(ctx, 1) } } @@ -153,49 +258,19 @@ const ( familyTeam ) -// firstInRun reports whether this is the first time, within the run -// identified by runID, that family has been observed — marking it seen as a -// side effect. An empty runID (no run context to dedup against) always -// counts, matching the pre-dedup behavior. -func (r *Recorder) firstInRun(runID string, family delegationFamily) bool { - if runID == "" { - return true - } - r.mu.Lock() - defer r.mu.Unlock() - u := r.runFamiliesUsed[runID] - var seen *bool - switch family { - case familySubagent: - seen = &u.subagent - case familyTeam: - seen = &u.team - } - if *seen { - return false - } - *seen = true - r.runFamiliesUsed[runID] = u - return true -} - -// clearRun drops the run's dedup entry once it has ended (EvResult), so -// runFamiliesUsed stays bounded to concurrently-live runs. -func (r *Recorder) clearRun(runID string) { - if runID == "" { - return +// recordResult counts the completed run against its bounded stop reason and +// the had_tool_call fact carried by the run's just-finished state. +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, st perRunState) { + stop := session.StopNone + if res != nil { + stop = res.Stop } - r.mu.Lock() - delete(r.runFamiliesUsed, runID) - r.mu.Unlock() -} - -func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload) { + r.runsCompleted.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrStop, string(stop)), + attribute.String(attrHadToolCall, strconv.FormatBool(st.hadToolCall)))) if res == nil { - r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(session.StopNone)))) return } - r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(res.Stop)))) u := res.Usage r.tokens.Add(ctx, int64(u.InputTokens), metric.WithAttributes(attribute.String(attrKind, "input"))) r.tokens.Add(ctx, int64(u.OutputTokens), metric.WithAttributes(attribute.String(attrKind, "output"))) diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go index ffad2aa238..c9cf5d2d37 100644 --- a/internal/adapter/productmetrics/metrics_test.go +++ b/internal/adapter/productmetrics/metrics_test.go @@ -3,6 +3,7 @@ package productmetrics import ( "context" "testing" + "time" "go.opentelemetry.io/otel/attribute" sdkmetric "go.opentelemetry.io/otel/sdk/metric" @@ -173,3 +174,84 @@ func TestRecorderSubagentUsedCountsEachDistinctRun(t *testing.T) { t.Errorf("subagent_used = %d, want 3 (run-1's dedup entry cleared on its EvResult)", got) } } + +func TestRecorderRunsCompletedHadToolCallTrueWhenASuccessfulToolCallOccurred(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1", got) + } +} + +func TestRecorderRunsCompletedHadToolCallFalseWithNoToolCall(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-2", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1", got) + } +} + +// TestRecorderRunsCompletedHadToolCallFalseWhenOnlyToolCallErrored pins the +// product definition: had_tool_call means the run took at least one +// SUCCESSFUL action, so a run whose only tool call errored is false. +func TestRecorderRunsCompletedHadToolCallFalseWhenOnlyToolCallErrored(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-3", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-3", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1 (the only tool call errored)", got) + } +} + +// TestRecorderRunsCompletedHadToolCallOnNilResult pins that a result-less +// EvResult still carries the attribute (the stop-reason arm already defaults +// to StopNone) rather than emitting an attribute-shape that differs from +// every other data point on the instrument. +func TestRecorderRunsCompletedHadToolCallOnNilResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-4", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-4"}) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1", got) + } + if got := sumPoint(t, agg, attrStop, string(session.StopNone)); got != 1 { + t.Errorf("runs_completed{stop=none} = %d, want 1", got) + } +} + +// TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns proves the per-run +// state is scoped to its run: two interleaved runs (a real possibility — +// Team/Parallel fan-out, or two concurrent client sessions on one process) +// must not leak their tool-call facts into each other. +func TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-a", session.SessionID("s1"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-b", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-a", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1 (run-a's tool call must not leak into run-b)", got) + } + if got := sumPoint(t, agg, attrHadToolCall, "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1 (run-a's own tool call)", got) + } +} diff --git a/internal/adapter/productmetrics/toolcall.go b/internal/adapter/productmetrics/toolcall.go index 1d7ade3217..cad4b4776e 100644 --- a/internal/adapter/productmetrics/toolcall.go +++ b/internal/adapter/productmetrics/toolcall.go @@ -2,16 +2,120 @@ package productmetrics import ( "context" + "strings" "time" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + "github.com/stacklok/mecatl/engine/session" ) -// ToolCall records ONLY that a tool call happened — no tool name, no -// session id, no result content, no duration. It satisfies -// port.ToolCallRecorder. The three typed parameters it ignores (id, call, -// result) are accepted only because the port's signature requires them; not -// one of their fields is ever read. -func (r *Recorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { - r.toolCalls.Add(context.Background(), 1) +// Attribute keys for the tool_calls instrument. Both draw from a closed set: +// attrCategory from builtinToolCategories ∪ {"mcp", "other"} (see +// toolCategory), attrOutcome from {"success", "error"}. +const ( + attrCategory = "category" + attrOutcome = "outcome" +) + +// The two bounded category values that are not a built-in tool's own name. +const ( + categoryMCP = "mcp" + categoryOther = "other" +) + +// mcpToolPrefix is the STRUCTURAL naming convention every MCP-server tool is +// registered under (internal/adapter/mcp/clientmcp.go: `"mcp__" + server + +// "__" + toolName`). Bucketing the whole prefix under one literal means an +// MCP server or remote tool name — operator-chosen free text, and the one +// genuinely unbounded slice of the catalog — can never reach an attribute +// value, structurally, whatever a future MCP integration is named. +const mcpToolPrefix = "mcp__" + +// builtinToolCategories is the CLOSED set of tool names this package may emit +// verbatim as a category: mecatl's own fixed catalog, where the name carries +// no operator or user content and the cardinality is fixed at compile time. +// +// It is deliberately an allowlist rather than a "not mcp__-prefixed ⇒ safe" +// inference: a catalog is not only built-ins plus MCP tools (an agent def, +// a learned skill, or a future extension seam can register a name derived +// from operator or model input), so the safe default for an unrecognised +// name is the single literal categoryOther — never the name itself. A new +// built-in showing up as "other" in the metric is the visible, harmless +// prompt to add a line here. +var builtinToolCategories = map[string]bool{ + // Filesystem + shell (engine/adapter/fstools, engine/agent). + "Read": true, "ListDir": true, "Edit": true, "Write": true, + "Copy": true, "Move": true, "Remove": true, "Grep": true, "Glob": true, + "Bash": true, "BashStatus": true, "BashSystemTemp": true, + // Outbound reads (engine/adapter/webfetch, engine/adapter/search). + "WebFetch": true, "WebSearch": true, + // MCP meta-tools — mecatl's OWN fixed names, distinct from the + // mcp__-prefixed server tools they operate over. + "ListMcpResources": true, "ReadMcpResource": true, + "CallMcpWithQuery": true, "FetchMcpResource": true, + // Delegation (engine/agent). + "Subagent": true, "SubagentStatus": true, "InspectSubagent": true, + "Parallel": true, "Team": true, "InspectMember": true, "SubmitResult": true, + // Skills. + "Skill": true, "SkillDraft": true, + // Project memory (internal/adapter/memory). + "Remember": true, "Recall": true, "SearchMemory": true, + "InspectMemory": true, "ForgetMemory": true, "UndoMemory": true, + // User model (internal/adapter/memory). + "RememberUser": true, "RecallUser": true, "SearchUserModel": true, + "InspectUserMemory": true, "ForgetUserMemory": true, "UndoUserMemory": true, + // Composition-owned + miscellaneous built-ins. + "PresentPlan": true, "Schedule": true, "ScheduleQuery": true, + "DiscoverModels": true, "ToolSearch": true, +} + +// The two bounded outcome values. +const ( + outcomeSuccess = "success" + outcomeError = "error" +) + +// toolCategory projects a tool name onto the bounded category attribute: the +// tool's own name for a recognised built-in, categoryMCP for anything +// MCP-server-provided, categoryOther for everything else. +func toolCategory(name string) string { + if strings.HasPrefix(name, mcpToolPrefix) { + return categoryMCP + } + if builtinToolCategories[name] { + return name + } + return categoryOther +} + +// ToolCall satisfies port.ToolCallRecorder. It is the fallback path for a +// caller that drives the base port without the run correlation — it records +// with no run id, so the call is counted on tool_calls but can contribute to +// no run's had_tool_call. The engine itself always prefers ToolCallForRun +// (it type-asserts port.RunAwareToolCallRecorder), so in production this arm +// serves only a non-loop caller. +func (r *Recorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + r.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder. It records the +// bounded category/outcome attributes and tallies the run's per-run state +// (had_tool_call, and the tool-call count a later task publishes). +// +// It reads exactly two things off its arguments: call.Name, only through the +// closed-set toolCategory projection, and result.IsError, a boolean. The +// session id, the queued/took durations, and every free-text field +// (result.Content, call arguments) are ignored — the ignored parameters are +// accepted only because the port's signature requires them. +func (r *Recorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + outcome := outcomeSuccess + if result.IsError { + outcome = outcomeError + } + r.toolCalls.Add(context.Background(), 1, metric.WithAttributes( + attribute.String(attrCategory, toolCategory(call.Name)), + attribute.String(attrOutcome, outcome))) + r.perRun.markToolCall(runID, result.IsError) } diff --git a/internal/adapter/productmetrics/toolcall_test.go b/internal/adapter/productmetrics/toolcall_test.go index 8a96fb2362..8e3cb61126 100644 --- a/internal/adapter/productmetrics/toolcall_test.go +++ b/internal/adapter/productmetrics/toolcall_test.go @@ -1,9 +1,13 @@ package productmetrics import ( + "context" + "sync" "testing" "time" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "github.com/stacklok/mecatl/engine/session" ) @@ -22,3 +26,166 @@ func TestRecorderToolCallCountsWithoutIdentity(t *testing.T) { t.Errorf("tool_calls = %d, want 2", got) } } + +func TestRecorderToolCallForRunCategorizesBuiltinsByName(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrCategory, "Bash"); got != 1 { + t.Errorf("tool_calls{category=Bash} = %d, want 1", got) + } + if got := sumPoint(t, agg, attrCategory, "Read"); got != 1 { + t.Errorf("tool_calls{category=Read} = %d, want 1", got) + } +} + +// TestRecorderToolCallForRunBucketsUnrecognisedNamesUnderOther pins the +// closed-set discipline: a name that is NOT in mecatl's own fixed catalog is +// not assumed safe to emit verbatim (an agent def, a learned skill, or a +// future extension seam could derive one from operator or model input), so it +// lands on the single literal "other". +func TestRecorderToolCallForRunBucketsUnrecognisedNamesUnderOther(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "some-operator-named-tool"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "another_unknown"}, session.ToolResult{}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrCategory, categoryOther); got != 2 { + t.Errorf("tool_calls{category=other} = %d, want 2 (unrecognised names must not be emitted verbatim)", got) + } +} + +func TestRecorderToolCallForRunBucketsMCPToolsUnderOneCategory(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__github__list_issues"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__slack__post_message"}, session.ToolResult{}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrCategory, categoryMCP); got != 2 { + t.Errorf("tool_calls{category=mcp} = %d, want 2 (both MCP-server tools bucketed together)", got) + } + + // The real server/tool names must never appear as an attribute value. + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + continue + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + for _, leak := range []string{"github", "list_issues", "slack", "post_message"} { + if kv.Value.AsString() == leak { + t.Fatalf("MCP server/tool name leaked as an attribute value: %s=%s", kv.Key, kv.Value.AsString()) + } + } + } + } + } + } +} + +func TestRecorderToolCallForRunRecordsOutcome(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrOutcome, outcomeSuccess); got != 1 { + t.Errorf("tool_calls{outcome=success} = %d, want 1", got) + } + if got := sumPoint(t, agg, attrOutcome, outcomeError); got != 1 { + t.Errorf("tool_calls{outcome=error} = %d, want 1", got) + } +} + +// TestRecorderPerRunTrackerIsRaceFreeUnderConcurrentUse drives the two +// concurrent entry points into the shared per-run map — ToolCallForRun (per +// tool call, from the dispatcher's read-parallel batches) and Emit (per +// event) — against overlapping run ids, so `-race` exercises the one +// lock-discipline invariant this task introduces. +// +// The live runs are finished SEQUENTIALLY afterwards, deliberately: a +// concurrent EvResult racing its own run's tool calls has no defined +// ordering, so the bounded-map assertion below would be a coin flip rather +// than an invariant. Runs finished concurrently use disjoint ids. +func TestRecorderPerRunTrackerIsRaceFreeUnderConcurrentUse(t *testing.T) { + r, _ := newTestRecorder(t) + live := []string{"run-a", "run-b", "run-c"} + finishing := []string{"run-d", "run-e", "run-f"} + + var wg sync.WaitGroup + for _, runID := range live { + for i := 0; i < 8; i++ { + wg.Add(2) + go func(runID string) { + defer wg.Done() + r.ToolCallForRun(runID, session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + }(runID) + go func(runID string) { + defer wg.Done() + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: runID}) + }(runID) + } + } + for _, runID := range finishing { + wg.Add(1) + go func(runID string) { + defer wg.Done() + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: runID, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + }(runID) + } + wg.Wait() + + for _, runID := range live { + st := r.perRun.finish(runID) + if st.toolCallCount != 8 { + t.Errorf("%s toolCallCount = %d, want 8", runID, st.toolCallCount) + } + if !st.hadToolCall || !st.subagentSeen { + t.Errorf("%s state = %+v, want hadToolCall and subagentSeen both true", runID, st) + } + } + + // Every run has now been finished, so no state may be left behind. + r.perRun.mu.Lock() + left := len(r.perRun.states) + r.perRun.mu.Unlock() + if left != 0 { + t.Errorf("perRun.states holds %d entries after every run's EvResult, want 0 (the map must stay bounded to live runs)", left) + } +} + +// TestRecorderToolCallForRunTalliesPerRunCount pins the per-run tool-call +// count the later tool-calls-per-run task reads. It is tallied but not yet +// published as an instrument, so it is asserted on the state directly. +func TestRecorderToolCallForRunTalliesPerRunCount(t *testing.T) { + r, _ := newTestRecorder(t) + for i := 0; i < 3; i++ { + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: i == 0}, 0, 0) + } + // A call with no run correlation must not land on any run. + r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + + st := r.perRun.finish("run-1") + if st.toolCallCount != 3 { + t.Errorf("toolCallCount = %d, want 3", st.toolCallCount) + } + if !st.hadToolCall { + t.Error("hadToolCall = false, want true (two of the three calls succeeded)") + } + if got := r.perRun.finish(""); got != (perRunState{}) { + t.Errorf("finish(\"\") = %+v, want the zero state", got) + } +} From 3b66f4a427331270fee4bd8ddab51ff4811e24a1 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:31:20 -0400 Subject: [PATCH 30/47] fix(productmetrics): complete the builtinToolCategories allowlist Adds the six team-coordination tools (SendMessage, AddTask, ClaimTask, CompleteTask, ListTasks, RecordFinding) and InspectSession, which were missing and collapsing into "other" -- losing exactly the delegation-adoption signal this metrics catalog cares about. Removes BashSystemTemp, a permission-gate name that is never a dispatched ToolCall.Name and so could never match. Fixes review findings from Task 3 of the activation-extension plan. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/toolcall.go | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/internal/adapter/productmetrics/toolcall.go b/internal/adapter/productmetrics/toolcall.go index cad4b4776e..027ced4b69 100644 --- a/internal/adapter/productmetrics/toolcall.go +++ b/internal/adapter/productmetrics/toolcall.go @@ -48,7 +48,7 @@ var builtinToolCategories = map[string]bool{ // Filesystem + shell (engine/adapter/fstools, engine/agent). "Read": true, "ListDir": true, "Edit": true, "Write": true, "Copy": true, "Move": true, "Remove": true, "Grep": true, "Glob": true, - "Bash": true, "BashStatus": true, "BashSystemTemp": true, + "Bash": true, "BashStatus": true, // Outbound reads (engine/adapter/webfetch, engine/adapter/search). "WebFetch": true, "WebSearch": true, // MCP meta-tools — mecatl's OWN fixed names, distinct from the @@ -58,6 +58,11 @@ var builtinToolCategories = map[string]bool{ // Delegation (engine/agent). "Subagent": true, "SubagentStatus": true, "InspectSubagent": true, "Parallel": true, "Team": true, "InspectMember": true, "SubmitResult": true, + // Team coordination (engine/agent/teamtools.go). + "SendMessage": true, "AddTask": true, "ClaimTask": true, + "CompleteTask": true, "ListTasks": true, "RecordFinding": true, + // Session debugging (internal/adapter/sessiondebug). + "InspectSession": true, // Skills. "Skill": true, "SkillDraft": true, // Project memory (internal/adapter/memory). From 5f827f8da965229ffdaa751804406a80b9722bcd Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:33:27 -0400 Subject: [PATCH 31/47] feat(productmetrics): run_duration histogram Add mecatl.product.run_duration, a wall-clock histogram from a run's EvSessionInit to its EvResult. perRunState gains a startedAt field stamped by a new markStarted tracker method (same lock discipline as markFamilyUsed/markToolCall), and recordResult records the duration only when startedAt was actually observed, so a run whose EvSessionInit this Recorder never saw does not record a bogus duration. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/metrics.go | 31 ++++++++++++++++ .../adapter/productmetrics/metrics_test.go | 36 +++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index 5c4b999df5..05c0d79fa9 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -5,6 +5,7 @@ import ( "fmt" "strconv" "sync" + "time" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/metric" @@ -48,6 +49,7 @@ type Recorder struct { tokens metric.Int64Counter subagentUsed metric.Int64Counter teamUsed metric.Int64Counter + runDuration metric.Float64Histogram // perRun holds the bounded per-live-run facts this package derives across // the Emit/ToolCallForRun boundary. See perRunTracker. @@ -76,6 +78,13 @@ type perRunState struct { // tallied here but not yet published as an instrument; the tool-calls-per-run // distribution is a later task in this plan. toolCallCount int64 + + // startedAt is the wall-clock time this run's EvSessionInit was observed, + // used at EvResult time to compute run_duration. It stays the zero Time + // for a run whose EvSessionInit this Recorder never saw (e.g. a process + // restart mid-run) — recordResult must never record a duration from a + // zero startedAt. + startedAt time.Time } // perRunTracker guards the live-run state map. Bounded to concurrently-live @@ -130,6 +139,19 @@ func (t *perRunTracker) markFamilyUsed(runID string, family delegationFamily) bo return true } +// markStarted stamps runID's start time as now. Called once, from Emit's +// EvSessionInit case; a run with no observed EvSessionInit never has this +// called and its startedAt stays the zero Time. +func (t *perRunTracker) markStarted(runID string) { + if runID == "" { + return + } + t.mu.Lock() + defer t.mu.Unlock() + st := t.stateLocked(runID) + st.startedAt = time.Now() +} + // markToolCall tallies one tool call against runID. An empty runID (a caller // on the base port.ToolCallRecorder path, with no run to correlate against) is // tracked nowhere — its call is still counted on the tool_calls instrument, @@ -221,6 +243,11 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { metric.WithDescription("Runs that used the Team delegation family at least once.")); err != nil { return nil, fmt.Errorf("productmetrics: team_used counter: %w", err) } + if r.runDuration, err = meter.Float64Histogram("mecatl.product.run_duration", + metric.WithDescription("Wall-clock duration of a run, from session init to result, in seconds."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: run_duration histogram: %w", err) + } return r, nil } @@ -235,6 +262,7 @@ func (r *Recorder) Emit(ctx context.Context, ev session.Event) { switch ev.Type { case session.EvSessionInit: r.sessionsStarted.Add(ctx, 1) + r.perRun.markStarted(ev.RunID) case session.EvResult: // finish both reads and clears the run's state, so the had_tool_call // resolution and the bounded-map cleanup are one step. @@ -268,6 +296,9 @@ func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, r.runsCompleted.Add(ctx, 1, metric.WithAttributes( attribute.String(attrStop, string(stop)), attribute.String(attrHadToolCall, strconv.FormatBool(st.hadToolCall)))) + if !st.startedAt.IsZero() { + r.runDuration.Record(ctx, time.Since(st.startedAt).Seconds()) + } if res == nil { return } diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go index c9cf5d2d37..3a0318e82e 100644 --- a/internal/adapter/productmetrics/metrics_test.go +++ b/internal/adapter/productmetrics/metrics_test.go @@ -255,3 +255,39 @@ func TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns(t *testing.T) { t.Errorf("runs_completed{had_tool_call=true} = %d, want 1 (run-a's own tool call)", got) } } + +func TestRecorderRunDurationRecordedFromSessionInitToResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + time.Sleep(5 * time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.run_duration"] + if !ok { + t.Fatal("mecatl.product.run_duration missing") + } + hist, ok := agg.(metricdata.Histogram[float64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[float64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 { + t.Fatalf("expected exactly 1 recorded duration, got %+v", hist.DataPoints) + } + if hist.DataPoints[0].Sum <= 0 { + t.Errorf("recorded duration sum = %v, want > 0", hist.DataPoints[0].Sum) + } +} + +func TestRecorderRunDurationNotRecordedWithoutMatchingSessionInit(t *testing.T) { + // A run whose EvSessionInit this Recorder never observed (e.g. process + // restarted mid-run — an edge case, not a common path) must not record a + // bogus/negative duration. + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "orphan-run", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + if agg, ok := collect(t, reader)["mecatl.product.run_duration"]; ok { + if hist, ok := agg.(metricdata.Histogram[float64]); ok && len(hist.DataPoints) > 0 && hist.DataPoints[0].Count > 0 { + t.Errorf("recorded a duration for a run with no observed EvSessionInit: %+v", hist.DataPoints) + } + } +} From 6764f06843a1c3dd57d852e668e2e21ad0caca5e Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:43:37 -0400 Subject: [PATCH 32/47] feat(productmetrics): add tool_calls_per_run and time_to_first_value Publish the per-run tool-call tally that perRunState already tracked as mecatl.product.tool_calls_per_run, an Int64 histogram recorded at every EvResult (a tool-less run contributes an honest 0, so the distribution keeps its left tail). Add mecatl.product.time_to_first_value: a Float64 histogram recorded at most ONCE per install, ever, measuring the duration from this install's first-seen moment to its first run that both made a successful tool call and ended on StopEndTurn. The once-ever contract has two halves: an in-memory firstValueTracker whose test-and-claim is one critical section (a fan-out deployment must not record two samples), and a local marker file under XDG_STATE_HOME mirroring installid.go's injected-filesystem shape, written at the instant the qualifying run is observed rather than at startup -- a process that never has a qualifying run must not burn the install's one sample. FirstValueRecorded is therefore a pure READ used at startup, distinct from LoadOrCreateFirstValueMarker (the write half used as the record callback). Recording is off unless armed: NewRecorder's signature is unchanged and an unarmed Recorder never touches the instrument, so every existing caller and test keeps its exact prior posture. BuildProductMetrics arms it with time.Now() as firstSeenAt -- deliberate approximation for a coarse onboarding signal, sound because the marker means the metric can only fire on an install that has not yet had a qualifying run. A marker-read failure degrades to "track it anyway" rather than failing the pipeline. The no-PII guard in bounded_test.go now walks histogram data points too, and fails hard on an aggregation shape it does not recognise, so a future instrument cannot silently fall outside it. Co-Authored-By: Claude Sonnet 5 --- .../adapter/productmetrics/bounded_test.go | 54 +++-- internal/adapter/productmetrics/firstvalue.go | 101 +++++++++ .../adapter/productmetrics/firstvalue_test.go | 202 ++++++++++++++++++ internal/adapter/productmetrics/metrics.go | 117 ++++++++-- .../adapter/productmetrics/metrics_test.go | 41 ++++ internal/cliconfig/productmetrics.go | 31 +++ 6 files changed, 519 insertions(+), 27 deletions(-) create mode 100644 internal/adapter/productmetrics/firstvalue.go create mode 100644 internal/adapter/productmetrics/firstvalue_test.go diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go index 52884141d9..ae4a074c5c 100644 --- a/internal/adapter/productmetrics/bounded_test.go +++ b/internal/adapter/productmetrics/bounded_test.go @@ -45,6 +45,10 @@ var sensitiveMarkers = []string{ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T) { r, reader := newTestRecorder(t) + // Arm time_to_first_value so its data points are collected too — an + // unarmed Recorder never records it, which would leave that instrument + // outside the walk below. + r.EnableFirstValueTracking(time.Now(), false, nil) // Drive every observation path with deliberately sensitive-looking data. r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) @@ -99,12 +103,8 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T for _, sm := range rm.ScopeMetrics { for _, md := range sm.Metrics { assertNoSensitiveSubstring(t, md.Name) - sum, ok := md.Data.(metricdata.Sum[int64]) - if !ok { - t.Fatalf("metric %s: aggregation is %T, want Sum[int64]", md.Name, md.Data) - } - for _, dp := range sum.DataPoints { - iter := dp.Attributes.Iter() + for _, attrs := range dataPointAttributes(t, md.Name, md.Data) { + iter := attrs.Iter() for iter.Next() { kv := iter.Attribute() key := string(kv.Key) @@ -125,16 +125,16 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T // tool's own name, "mcp", or "other" (see toolCategory), so no // tool identity beyond mecatl's own fixed catalog can attach. if md.Name == "mecatl.product.tool_calls" { - for _, dp := range sum.DataPoints { - if _, ok := dp.Attributes.Value(attrCategory); !ok { - t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrCategory, dp.Attributes) + for _, attrs := range dataPointAttributes(t, md.Name, md.Data) { + if _, ok := attrs.Value(attrCategory); !ok { + t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrCategory, attrs) } - if _, ok := dp.Attributes.Value(attrOutcome); !ok { - t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrOutcome, dp.Attributes) + if _, ok := attrs.Value(attrOutcome); !ok { + t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrOutcome, attrs) } - if dp.Attributes.Len() != 2 { + if attrs.Len() != 2 { t.Errorf("mecatl.product.tool_calls data point carries %d attributes, want exactly 2: %v", - dp.Attributes.Len(), dp.Attributes) + attrs.Len(), attrs) } } } @@ -142,6 +142,34 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T } } +// dataPointAttributes returns every collected data point's attribute set, +// whatever aggregation the instrument uses. The walk must cover histograms as +// well as sums: this package publishes run_duration, tool_calls_per_run and +// time_to_first_value as histograms, and an aggregation this helper did not +// know about would silently drop that instrument out of the no-PII guard — +// so an unrecognised shape is a hard failure, never a skip. +func dataPointAttributes(t *testing.T, name string, agg metricdata.Aggregation) []attribute.Set { + t.Helper() + var out []attribute.Set + switch data := agg.(type) { + case metricdata.Sum[int64]: + for _, dp := range data.DataPoints { + out = append(out, dp.Attributes) + } + case metricdata.Histogram[int64]: + for _, dp := range data.DataPoints { + out = append(out, dp.Attributes) + } + case metricdata.Histogram[float64]: + for _, dp := range data.DataPoints { + out = append(out, dp.Attributes) + } + default: + t.Fatalf("metric %s: unhandled aggregation %T — add it to dataPointAttributes so the no-PII walk keeps covering it", name, agg) + } + return out +} + // TestToolCategoryOnlyEmitsClosedSetValues is the direct unit-level guard on // the one projection that reads a tool name: whatever it is fed, the output // must be a member of builtinToolCategories ∪ {"mcp", "other"}. diff --git a/internal/adapter/productmetrics/firstvalue.go b/internal/adapter/productmetrics/firstvalue.go new file mode 100644 index 0000000000..fb71ff3f61 --- /dev/null +++ b/internal/adapter/productmetrics/firstvalue.go @@ -0,0 +1,101 @@ +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// firstValueMarkerRelPath is the state-dir-relative path to a bare marker file +// recording whether this install's first meaningful-and-successful run has +// already been observed — so mecatl.product.time_to_first_value is recorded at +// most once per install, ever. It lives beside the install-id file under +// XDG_STATE_HOME (machine-written runtime state, not human config) and mirrors +// installid.go's injected-filesystem shape exactly. +const firstValueMarkerRelPath = "mecatl/first-value-recorded" + +// firstValueMarkerPath resolves the marker's absolute path, failing closed when +// no state directory can be resolved at all. +func firstValueMarkerPath(env xdgconfig.ResolveEnv) (string, error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return "", fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + return filepath.Join(base, firstValueMarkerRelPath), nil +} + +// FirstValueRecorded reports whether this install's time_to_first_value sample +// was already recorded by some EARLIER process. It is a pure READ: it never +// creates the marker, because the marker means "the qualifying run happened", +// and process startup is not that moment (a process may exit without ever +// having one). The marker is written later, by LoadOrCreateFirstValueMarker, +// at the instant the qualifying run is observed. +// +// readFile is injected for testing; FirstValueRecordedDefault binds the real +// filesystem. +func FirstValueRecorded(env xdgconfig.ResolveEnv, readFile func(string) ([]byte, error)) (bool, error) { + path, err := firstValueMarkerPath(env) + if err != nil { + return false, err + } + if readFile == nil { + return false, nil + } + if _, rerr := readFile(path); rerr != nil { + return false, nil + } + return true, nil +} + +// FirstValueRecordedDefault binds FirstValueRecorded to the real process +// environment and filesystem. +func FirstValueRecordedDefault() (bool, error) { + return FirstValueRecorded(xdgconfig.OSEnv, os.ReadFile) +} + +// LoadOrCreateFirstValueMarker reports whether this install's first-value +// moment was already marked (already == true), creating the marker (and +// returning already == false) the first time it is called. It is the WRITE +// half, called at the moment a qualifying run is observed — not at startup. +// Once created, the marker is never removed automatically; deleting it (like +// the install-id file) resets the install and lets time_to_first_value fire +// once more. +// +// readFile/writeFile/mkdirAll are injected for testing; +// LoadOrCreateFirstValueMarkerDefault binds the real filesystem. +func LoadOrCreateFirstValueMarker( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (already bool, err error) { + path, err := firstValueMarkerPath(env) + if err != nil { + return false, err + } + + if readFile != nil { + if _, rerr := readFile(path); rerr == nil { + return true, nil + } + } + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte("1"), 0o600); werr != nil { + return false, fmt.Errorf("productmetrics: write first-value marker: %w", werr) + } + } + return false, nil +} + +// LoadOrCreateFirstValueMarkerDefault binds LoadOrCreateFirstValueMarker to the +// real process environment and filesystem. +func LoadOrCreateFirstValueMarkerDefault() (already bool, err error) { + return LoadOrCreateFirstValueMarker(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} diff --git a/internal/adapter/productmetrics/firstvalue_test.go b/internal/adapter/productmetrics/firstvalue_test.go new file mode 100644 index 0000000000..22acbf630d --- /dev/null +++ b/internal/adapter/productmetrics/firstvalue_test.go @@ -0,0 +1,202 @@ +package productmetrics + +import ( + "context" + "errors" + "os" + "testing" + "time" + + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func testResolveEnv() xdgconfig.ResolveEnv { + return xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } +} + +// fakeMarkerFS is the injected filesystem the marker helpers are driven over. +type fakeMarkerFS struct { + written map[string][]byte +} + +func newFakeMarkerFS() *fakeMarkerFS { return &fakeMarkerFS{written: map[string][]byte{}} } + +func (f *fakeMarkerFS) readFile(p string) ([]byte, error) { + if d, ok := f.written[p]; ok { + return d, nil + } + return nil, os.ErrNotExist +} + +func (f *fakeMarkerFS) writeFile(p string, d []byte, _ os.FileMode) error { + f.written[p] = d + return nil +} + +func (*fakeMarkerFS) mkdirAll(string, os.FileMode) error { return nil } + +func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T) { + env, fs := testResolveEnv(), newFakeMarkerFS() + + already, err := LoadOrCreateFirstValueMarker(env, fs.readFile, fs.writeFile, fs.mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) + } + if already { + t.Error("already = true on first call, want false") + } + + already2, err := LoadOrCreateFirstValueMarker(env, fs.readFile, fs.writeFile, fs.mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateFirstValueMarker: %v", err) + } + if !already2 { + t.Error("already = false on second call, want true") + } +} + +func TestLoadOrCreateFirstValueMarkerFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + if _, err := LoadOrCreateFirstValueMarker(env, nil, nil, nil); err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } + if _, err := FirstValueRecorded(env, nil); err == nil { + t.Fatal("expected FirstValueRecorded to error when no state dir can be resolved, got nil") + } +} + +// FirstValueRecorded is a pure read: a startup check must NOT create the +// marker, or an install whose first process never has a qualifying run would +// lose its one sample forever. +func TestFirstValueRecordedNeverCreatesTheMarker(t *testing.T) { + env, fs := testResolveEnv(), newFakeMarkerFS() + + already, err := FirstValueRecorded(env, fs.readFile) + if err != nil { + t.Fatalf("FirstValueRecorded: %v", err) + } + if already { + t.Error("already = true with no marker present, want false") + } + if len(fs.written) != 0 { + t.Fatalf("FirstValueRecorded wrote %v, want no writes", fs.written) + } + + if _, err := LoadOrCreateFirstValueMarker(env, fs.readFile, fs.writeFile, fs.mkdirAll); err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) + } + already, err = FirstValueRecorded(env, fs.readFile) + if err != nil { + t.Fatalf("FirstValueRecorded after marking: %v", err) + } + if !already { + t.Error("already = false after the marker was created, want true") + } +} + +// firstValueHistogram returns the collected time_to_first_value data points, +// or nil when the instrument recorded nothing at all. +func firstValueHistogram(t *testing.T, agg metricdata.Aggregation, present bool) []metricdata.HistogramDataPoint[float64] { + t.Helper() + if !present { + return nil + } + hist, ok := agg.(metricdata.Histogram[float64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[float64]", agg) + } + return hist.DataPoints +} + +func qualifyingRun(t *testing.T, r *Recorder, runID string) { + t.Helper() + r.ToolCallForRun(runID, session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: runID, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) +} + +func TestRecorderRecordsTimeToFirstValueOnceOnly(t *testing.T) { + r, reader := newTestRecorder(t) + var marks int + r.EnableFirstValueTracking(time.Now().Add(-90*time.Second), false, func() error { marks++; return nil }) + + qualifyingRun(t, r, "run-1") + qualifyingRun(t, r, "run-2") + + agg, present := collect(t, reader)["mecatl.product.time_to_first_value"] + dps := firstValueHistogram(t, agg, present) + if len(dps) != 1 || dps[0].Count != 1 { + t.Fatalf("expected exactly one sample, got %+v", dps) + } + if dps[0].Sum < 90 { + t.Errorf("recorded duration %v s, want at least the 90 s since firstSeenAt", dps[0].Sum) + } + if marks != 1 { + t.Errorf("marker persisted %d times, want exactly 1", marks) + } +} + +func TestRecorderSkipsTimeToFirstValueWhenAlreadyRecorded(t *testing.T) { + r, reader := newTestRecorder(t) + var marks int + r.EnableFirstValueTracking(time.Now().Add(-time.Second), true, func() error { marks++; return nil }) + + qualifyingRun(t, r, "run-1") + + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded despite alreadyRecorded=true: %+v", agg) + } + if marks != 0 { + t.Errorf("marker persisted %d times, want 0", marks) + } +} + +func TestRecorderSkipsTimeToFirstValueWhenNotArmed(t *testing.T) { + r, reader := newTestRecorder(t) + qualifyingRun(t, r, "run-1") + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded without EnableFirstValueTracking: %+v", agg) + } +} + +// A run that ended cleanly but took no action, and a run that acted but did not +// end cleanly, are both non-qualifying: the metric measures time to the first +// run that BOTH acted and succeeded. +func TestRecorderTimeToFirstValueRequiresToolCallAndCleanStop(t *testing.T) { + r, reader := newTestRecorder(t) + r.EnableFirstValueTracking(time.Now().Add(-time.Second), false, nil) + + // Clean stop, no tool call. + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + // Tool call, non-clean stop. + r.ToolCallForRun("run-2", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-2", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded for a non-qualifying run: %+v", agg) + } + + // The genuinely qualifying run does record. + qualifyingRun(t, r, "run-3") + agg, present := collect(t, reader)["mecatl.product.time_to_first_value"] + if dps := firstValueHistogram(t, agg, present); len(dps) != 1 || dps[0].Count != 1 { + t.Fatalf("expected one sample after the qualifying run, got %+v", dps) + } +} diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index 05c0d79fa9..ce11ab95e8 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -39,21 +39,92 @@ const ( // through the closed-set projection in toolcall.go (toolCategory), which can // emit nothing but a built-in tool's own name, "mcp", or "other". type Recorder struct { - heartbeat metric.Int64Counter - featureEnabled metric.Int64Counter - providerConfig metric.Int64Counter - deploymentMode metric.Int64Counter - sessionsStarted metric.Int64Counter - runsCompleted metric.Int64Counter - toolCalls metric.Int64Counter - tokens metric.Int64Counter - subagentUsed metric.Int64Counter - teamUsed metric.Int64Counter - runDuration metric.Float64Histogram + heartbeat metric.Int64Counter + featureEnabled metric.Int64Counter + providerConfig metric.Int64Counter + deploymentMode metric.Int64Counter + sessionsStarted metric.Int64Counter + runsCompleted metric.Int64Counter + toolCalls metric.Int64Counter + tokens metric.Int64Counter + subagentUsed metric.Int64Counter + teamUsed metric.Int64Counter + runDuration metric.Float64Histogram + toolCallsPerRun metric.Int64Histogram + timeToFirstValue metric.Float64Histogram // perRun holds the bounded per-live-run facts this package derives across // the Emit/ToolCallForRun boundary. See perRunTracker. perRun *perRunTracker + + // firstValue holds the once-per-install time_to_first_value state. It is + // deliberately SEPARATE from perRun (which is per-live-run, keyed by run id + // and dropped at EvResult): this is install-scoped, single-slot state whose + // whole lifetime is the process. + firstValue firstValueTracker +} + +// firstValueTracker guards the once-ever time_to_first_value state. Armed by +// EnableFirstValueTracking (composition), read and flipped at most once by +// recordResult. +// +// CONCURRENCY: as with perRunTracker, the "is it done" test and the "mark it +// done" write are ONE critical section — concurrent EvResult observations on a +// fan-out deployment would otherwise both pass the test and record two samples +// for a metric whose entire contract is "at most one, ever". +type firstValueTracker struct { + mu sync.Mutex + // armed is false until EnableFirstValueTracking is called; an unarmed + // Recorder never records the metric at all (the default, byte-identical to + // the pre-feature posture for every existing caller of NewRecorder). + armed bool + // firstSeenAt is this install's first-seen moment; the recorded duration is + // measured from it. + firstSeenAt time.Time + // done is true once the sample exists — either recorded by THIS process, or + // (per the persisted marker) by an earlier one. + done bool + // recordFn persists the marker so a LATER process invocation also stays + // disabled. Called at most once, best-effort. + recordFn func() error +} + +// EnableFirstValueTracking arms mecatl.product.time_to_first_value recording. +// firstSeenAt is this install's first-seen timestamp; alreadyRecorded, when +// true, permanently disables recording for this Recorder's lifetime (this +// install already has its one sample). recordFn persists the local marker so a +// later process invocation also stays disabled; it is called at most once, and +// may be nil (in-memory-only tracking). +// +// It is a separate arming step rather than a NewRecorder parameter so that +// NewRecorder's signature — and every existing caller and test of it — stays +// unchanged; an unarmed Recorder simply never records this instrument. +func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecorded bool, recordFn func() error) { + r.firstValue.mu.Lock() + defer r.firstValue.mu.Unlock() + r.firstValue.armed = true + r.firstValue.firstSeenAt = firstSeenAt + r.firstValue.done = alreadyRecorded + r.firstValue.recordFn = recordFn +} + +// claim reports whether THIS observation is the install's first-value moment, marking it claimed and persisting the marker as one atomic step. It +// returns the firstSeenAt to measure from; a false claim means the metric must +// not be recorded (unarmed, already recorded, or no usable firstSeenAt). +func (t *firstValueTracker) claim() (time.Time, bool) { + t.mu.Lock() + defer t.mu.Unlock() + if !t.armed || t.done || t.firstSeenAt.IsZero() { + return time.Time{}, false + } + t.done = true + if t.recordFn != nil { + // Best-effort: a failed write risks re-recording once on a later + // process, which is a fidelity wobble in a coarse onboarding signal — + // not a correctness bug worth failing anything over. + _ = t.recordFn() + } + return t.firstSeenAt, true } // perRunState is the bounded set of facts tracked for ONE live run, keyed by @@ -74,9 +145,8 @@ type perRunState struct { // EvResult time as the runs_completed had_tool_call attribute. hadToolCall bool - // toolCallCount is the run's total tool calls (successful or not). It is - // tallied here but not yet published as an instrument; the tool-calls-per-run - // distribution is a later task in this plan. + // toolCallCount is the run's total tool calls (successful or not), + // published at EvResult as the tool_calls_per_run distribution. toolCallCount int64 // startedAt is the wall-clock time this run's EvSessionInit was observed, @@ -248,6 +318,16 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { metric.WithUnit("s")); err != nil { return nil, fmt.Errorf("productmetrics: run_duration histogram: %w", err) } + if r.toolCallsPerRun, err = meter.Int64Histogram("mecatl.product.tool_calls_per_run", + metric.WithDescription("Total tool calls made within a single run."), + metric.WithUnit("{tool_call}")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls_per_run histogram: %w", err) + } + if r.timeToFirstValue, err = meter.Float64Histogram("mecatl.product.time_to_first_value", + metric.WithDescription("One-time-per-install duration, in seconds, from this install's first-seen moment (approximated by the first product-metrics startup that observes no marker) to its first run that both made a successful tool call and ended cleanly."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: time_to_first_value histogram: %w", err) + } return r, nil } @@ -296,9 +376,18 @@ func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, r.runsCompleted.Add(ctx, 1, metric.WithAttributes( attribute.String(attrStop, string(stop)), attribute.String(attrHadToolCall, strconv.FormatBool(st.hadToolCall)))) + r.toolCallsPerRun.Record(ctx, st.toolCallCount) if !st.startedAt.IsZero() { r.runDuration.Record(ctx, time.Since(st.startedAt).Seconds()) } + // The install's first-value moment: the first run that BOTH took an action + // (a successful tool call) and ended cleanly. Recorded at most once ever, + // across process restarts — see firstValueTracker. + if stop == session.StopEndTurn && st.hadToolCall { + if firstSeenAt, claimed := r.firstValue.claim(); claimed { + r.timeToFirstValue.Record(ctx, time.Since(firstSeenAt).Seconds()) + } + } if res == nil { return } diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go index 3a0318e82e..bbea855f3b 100644 --- a/internal/adapter/productmetrics/metrics_test.go +++ b/internal/adapter/productmetrics/metrics_test.go @@ -291,3 +291,44 @@ func TestRecorderRunDurationNotRecordedWithoutMatchingSessionInit(t *testing.T) } } } + +func TestRecorderToolCallsPerRunRecordedAtResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{}, 0, time.Millisecond) + // An ERRORED call still counts toward the run's tool-call total (only + // had_tool_call is success-gated). + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Grep"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"] + if !ok { + t.Fatal("mecatl.product.tool_calls_per_run missing") + } + hist, ok := agg.(metricdata.Histogram[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[int64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 || hist.DataPoints[0].Sum != 3 { + t.Fatalf("expected one data point with count 1 summing to 3, got %+v", hist.DataPoints) + } +} + +func TestRecorderToolCallsPerRunRecordsZeroForAToollessRun(t *testing.T) { + // A run that called no tool still contributes a 0 sample — otherwise the + // distribution silently over-reports by omitting its whole left tail. + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"] + if !ok { + t.Fatal("mecatl.product.tool_calls_per_run missing") + } + hist, ok := agg.(metricdata.Histogram[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[int64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 || hist.DataPoints[0].Sum != 0 { + t.Fatalf("expected one data point with count 1 summing to 0, got %+v", hist.DataPoints) + } +} diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index f8c32f3eb9..c60b398bd8 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -108,6 +108,8 @@ func BuildProductMetrics( go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) + armFirstValueTracking(ctx, recorder, diag) + return ProductMetricsHandles{ Sink: recorder, ToolCallRecorder: recorder, @@ -115,3 +117,32 @@ func BuildProductMetrics( FirstRun: firstRun, }, nil } + +// armFirstValueTracking enables mecatl.product.time_to_first_value on recorder. +// +// firstSeenAt is time.Now(): this process's start, not the install-id file's +// mtime. The approximation is deliberate and sound for the signal's purpose (a +// coarse "how long did onboarding take", not a billing-grade timer) — the +// marker read below means the metric can only ever fire on an install that has +// not yet had a qualifying run, and for a genuinely new install this process IS +// the first one, so "now" is that install's first-seen moment to within the +// process's own startup. It also keeps the install-id file's path private to +// the productmetrics package. +// +// A marker-read failure degrades to "track it anyway" rather than disabling +// anything: time_to_first_value is a nice-to-have signal, not load-bearing +// enough to fail the whole product-metrics pipeline over. The worst case is one +// duplicate sample from a later process. +func armFirstValueTracking(ctx context.Context, recorder *productmetrics.Recorder, diag port.Diagnostics) { + already, err := productmetrics.FirstValueRecordedDefault() + if err != nil && diag != nil { + diag.Log(ctx, port.LevelDebug, + "product metrics: could not read the first-value marker; time_to_first_value may be re-recorded once", + "error", err) + already = false + } + recorder.EnableFirstValueTracking(time.Now(), already, func() error { + _, werr := productmetrics.LoadOrCreateFirstValueMarkerDefault() + return werr + }) +} From 148673e4d2e73c0120867bfbff697038f3df290c Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:52:53 -0400 Subject: [PATCH 33/47] fix(productmetrics): correct time_to_first_value's misleading description The instrument description claimed it approximates "install first-seen moment", but the value actually recorded is the START OF THE PROCESS that reaches the qualifying run -- the LAST startup with no marker, not the first. A user who runs mecatl repeatedly over several days and only reaches a qualifying run on day three would see minutes, not three days, which is not what the old description promised a dashboard consumer. Fixes a review finding from Task 5 of the activation-extension plan. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/metrics.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index ce11ab95e8..d947a57ac5 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -324,7 +324,7 @@ func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { return nil, fmt.Errorf("productmetrics: tool_calls_per_run histogram: %w", err) } if r.timeToFirstValue, err = meter.Float64Histogram("mecatl.product.time_to_first_value", - metric.WithDescription("One-time-per-install duration, in seconds, from this install's first-seen moment (approximated by the first product-metrics startup that observes no marker) to its first run that both made a successful tool call and ended cleanly."), + metric.WithDescription("One-time-per-install duration, in seconds, from the start of the process that observed this install's first run that both made a successful tool call and ended cleanly (an approximation of onboarding time, not install age: a process started days after install and reaching that run in minutes reports minutes, not days)."), metric.WithUnit("s")); err != nil { return nil, fmt.Errorf("productmetrics: time_to_first_value histogram: %w", err) } From 177e94940f61dc58621c9c900c836d825eb15458 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 18:56:06 -0400 Subject: [PATCH 34/47] feat(productmetrics): DryRunRecorder mirrors run_duration/tool_calls_per_run/time_to_first_value MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DryRunRecorder already logged category/outcome/had_tool_call (from an earlier merged task); it did not yet mirror Recorder's run_duration, tool_calls_per_run, or time_to_first_value fields on runs_completed. Emit's EvSessionInit case now marks the run started via the existing perRunTracker, and emitResult logs tool_calls_per_run and (when this process observed the run's start) run_duration_seconds alongside the existing fields. time_to_first_value diverges from Recorder by design: dry-run stays stateless (no XDG_STATE_HOME reads/writes), so instead of the real once-ever, persisted-marker sample it logs a would-be time_to_first_value observation on every qualifying run (StopEndTurn + at least one successful tool call) — more verbose than the real pipeline, but the honest choice for a "show what could be sent" audit surface. Documented on DryRunRecorder's doc comment. Co-Authored-By: Claude Sonnet 5 --- internal/adapter/productmetrics/dryrun.go | 76 ++++++++--- .../adapter/productmetrics/dryrun_test.go | 120 ++++++++++++++++++ 2 files changed, 177 insertions(+), 19 deletions(-) diff --git a/internal/adapter/productmetrics/dryrun.go b/internal/adapter/productmetrics/dryrun.go index c47117381c..9acc44dcef 100644 --- a/internal/adapter/productmetrics/dryrun.go +++ b/internal/adapter/productmetrics/dryrun.go @@ -14,14 +14,27 @@ import ( // it over OTLP — the --product-metrics-dry-run audit path, so a skeptical // operator can see exactly what this pipeline would have sent without // trusting the docs. It logs ONLY the same bounded fields Recorder ever reads -// (event type, stop reason, had_tool_call, the closed-set tool category and -// outcome, token counts by kind, feature/provider/mode enum values) — never a -// session id, a raw tool name, or free-text content, mirroring Recorder's own -// privacy discipline exactly. +// (event type, stop reason, had_tool_call, run_duration, tool_calls_per_run, +// the closed-set tool category and outcome, token counts by kind, +// feature/provider/mode enum values) — never a session id, a raw tool name, +// or free-text content, mirroring Recorder's own privacy discipline exactly. // // The audit path must stay in LOCKSTEP with Recorder: an attribute Recorder // attaches but DryRunRecorder omits makes this surface understate what is // sent, which is the one thing it exists to rule out. +// +// time_to_first_value is the ONE deliberate divergence, by design: Recorder's +// version is a once-ever, install-scoped, persisted-marker sample (armed via +// EnableFirstValueTracking, composition-only). DryRunRecorder has — and must +// have — NO persistent state (no XDG_STATE_HOME reads/writes; it stays a +// stateless, one-shot audit tool per the ADR), so it cannot reproduce "at +// most once, ever" across process restarts. Instead it logs a would-be +// time_to_first_value observation on EVERY run that qualifies (StopEndTurn + +// at least one successful tool call), not just the first one this process +// happens to see. That is MORE verbose than what Recorder would actually +// send, but it is the honest, simpler choice for a diagnostic surface whose +// job is "show what COULD be sent" — an operator sees every candidate moment +// rather than only whichever one a real install's persisted marker allowed. type DryRunRecorder struct { diag port.Diagnostics perRun *perRunTracker @@ -40,12 +53,14 @@ func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder { } // Emit logs the bounded event type (and, for EvResult, the stop reason, -// had_tool_call, and token counts by kind) — the exact same fields -// Recorder.Emit reads. +// had_tool_call, run_duration_seconds, tool_calls_per_run, token counts by +// kind, and a separate would-be time_to_first_value line when the run +// qualifies) — the exact same underlying facts Recorder.Emit reads. func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { switch ev.Type { case session.EvSessionInit: d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") + d.perRun.markStarted(ev.RunID) case session.EvResult: d.emitResult(ctx, ev.Result, d.perRun.finish(ev.RunID)) case session.EvSubagentStart: @@ -60,21 +75,44 @@ func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { } func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload, st perRunState) { - if res == nil { - d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed", - "stop", string(session.StopNone), - attrHadToolCall, st.hadToolCall) - return + stop := session.StopNone + if res != nil { + stop = res.Stop } - u := res.Usage - d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed + tokens", - "stop", string(res.Stop), + + fields := []any{ + "stop", string(stop), attrHadToolCall, st.hadToolCall, - "input_tokens", u.InputTokens, - "output_tokens", u.OutputTokens, - "cache_read_tokens", u.CacheReadTokens, - "cache_write_tokens", u.CacheWriteTokens, - "reasoning_tokens", u.ReasoningTokens) + "tool_calls_per_run", st.toolCallCount, + } + // run_duration is only meaningful when this recorder actually observed the + // run's EvSessionInit (mirroring Recorder.recordResult's zero-startedAt + // guard) — a run whose start this process missed reports no duration + // rather than a nonsense one measured from time.Time{}. + if !st.startedAt.IsZero() { + fields = append(fields, "run_duration_seconds", time.Since(st.startedAt).Seconds()) + } + + msg := "product metrics (dry-run): would record runs_completed" + if res != nil { + u := res.Usage + msg = "product metrics (dry-run): would record runs_completed + tokens" + fields = append(fields, + "input_tokens", u.InputTokens, + "output_tokens", u.OutputTokens, + "cache_read_tokens", u.CacheReadTokens, + "cache_write_tokens", u.CacheWriteTokens, + "reasoning_tokens", u.ReasoningTokens) + } + d.diag.Log(ctx, port.LevelInfo, msg, fields...) + + // See the DryRunRecorder doc comment: unlike Recorder's once-ever, + // persisted-marker time_to_first_value, dry-run logs this on EVERY + // qualifying run (no state to track "first" against) — the honest + // simplification for a stateless audit surface. + if stop == session.StopEndTurn && st.hadToolCall { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record time_to_first_value") + } } // ToolCall logs the run-less form, matching Recorder.ToolCall's delegation. diff --git a/internal/adapter/productmetrics/dryrun_test.go b/internal/adapter/productmetrics/dryrun_test.go index be7fff9998..36fce7fd2a 100644 --- a/internal/adapter/productmetrics/dryrun_test.go +++ b/internal/adapter/productmetrics/dryrun_test.go @@ -150,3 +150,123 @@ func TestDryRunRecorderHeartbeatLogsOnlyEnums(t *testing.T) { t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) } } + +// TestDryRunRecorderLogsRunDurationAndToolCallsPerRun pins the two +// runs_completed fields this task adds: run_duration_seconds (present only +// when this recorder observed the run's EvSessionInit) and +// tool_calls_per_run (the run's total tool-call count, successful or not). +func TestDryRunRecorderLogsRunDurationAndToolCallsPerRun(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + // diag.args: [0]=sessions_started, [1]=Read tool_calls, [2]=Bash tool_calls, [3]=runs_completed. + if len(diag.args) != 4 { + t.Fatalf("got %d logged lines, want 4: %v", len(diag.lines), diag.lines) + } + runsCompleted := diag.args[3] + if !hasArg(runsCompleted, "tool_calls_per_run", int64(2)) { + t.Errorf("runs_completed dry-run log args = %v, want tool_calls_per_run=2", runsCompleted) + } + if !hasArg(runsCompleted, attrHadToolCall, true) { + t.Errorf("runs_completed dry-run log args = %v, want %s=true (one successful call)", runsCompleted, attrHadToolCall) + } + found := false + for i := 0; i+1 < len(runsCompleted); i += 2 { + if k, ok := runsCompleted[i].(string); ok && k == "run_duration_seconds" { + found = true + if _, ok := runsCompleted[i+1].(float64); !ok { + t.Errorf("run_duration_seconds arg = %v (%T), want float64", runsCompleted[i+1], runsCompleted[i+1]) + } + } + } + if !found { + t.Errorf("runs_completed dry-run log args = %v, want a run_duration_seconds field (EvSessionInit was observed)", runsCompleted) + } +} + +// TestDryRunRecorderOmitsRunDurationWhenSessionInitUnseen covers a run whose +// EvSessionInit this recorder never observed (e.g. it started before this +// process attached) — run_duration_seconds must be omitted rather than +// reporting a bogus duration measured from a zero time. +func TestDryRunRecorderOmitsRunDurationWhenSessionInitUnseen(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-never-started", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + if len(diag.args) != 1 { + t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) + } + for i := 0; i+1 < len(diag.args[0]); i += 2 { + if k, ok := diag.args[0][i].(string); ok && k == "run_duration_seconds" { + t.Errorf("run_duration_seconds present for a run whose EvSessionInit was never observed: %v", diag.args[0]) + } + } +} + +// TestDryRunRecorderLogsTimeToFirstValueEveryQualifyingRun pins this task's +// documented design choice: unlike Recorder's once-ever, persisted-marker +// time_to_first_value, the stateless dry-run path logs a would-be +// time_to_first_value observation on EVERY run that qualifies (StopEndTurn + +// at least one successful tool call) — proven here across TWO separate +// qualifying runs, both logging it. +func TestDryRunRecorderLogsTimeToFirstValueEveryQualifyingRun(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + qualify := func(runID string) { + r.ToolCallForRun(runID, session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: runID, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + } + qualify("run-a") + qualify("run-b") + + count := 0 + for _, line := range diag.lines { + if line == "product metrics (dry-run): would record time_to_first_value" { + count++ + } + } + if count != 2 { + t.Errorf("got %d time_to_first_value lines across two qualifying runs, want 2 (dry-run logs every qualifying run, not once-ever): %v", count, diag.lines) + } +} + +// TestDryRunRecorderOmitsTimeToFirstValueWhenNotQualifying covers the two +// non-qualifying shapes: no successful tool call, and a non-StopEndTurn stop. +func TestDryRunRecorderOmitsTimeToFirstValueWhenNotQualifying(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + // No tool call at all: StopEndTurn but hadToolCall stays false. + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-no-tools", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + // A successful tool call but a non-EndTurn stop. + r.ToolCallForRun("run-error-stop", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-error-stop", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + for _, line := range diag.lines { + if line == "product metrics (dry-run): would record time_to_first_value" { + t.Errorf("time_to_first_value logged for a non-qualifying run: %v", diag.lines) + } + } +} From 08a5b6c6e1fe18164f9231a85f81064c6181f51e Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 19:10:42 -0400 Subject: [PATCH 35/47] feat(mecak8s): provision a stable per-release install-id via a Helm ConfigMap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mecak8s runs storage-free with no PVC (ADR 0048), so it cannot use the local-file product-metrics install-id mechanism the other three binaries share: every pod restart would mint a fresh, never-reused id — the worst-case cardinality pattern for that pipeline. The chart now provisions ONE id per release into a `-install-id` ConfigMap, using the standard `lookup` idiom so a `helm upgrade` reuses the existing value verbatim and only a first install mints a new one, and mounts it into the container via `MECATL_PRODUCT_METRICS_INSTALL_ID`. `BuildProductMetrics` gains an `installIDOverride` parameter: non-empty skips `LoadOrCreateInstallIDDefault` entirely (and never reports FirstRun — the chart, not this process, owns the id's lifecycle). The other three call sites pass "" and are behaviourally unchanged. Because the env var is unconditional, the Deployment now always renders an `env:` block; the chart tests' env-shape assertions run through a new `appEnv` helper so they keep asserting exactly what they did before, and the one render-equality test masks the per-render uuidv4. Co-Authored-By: Claude Sonnet 5 --- cmd/mecak8s/observability.go | 13 +- cmd/mecated/main.go | 2 +- cmd/mecatequi/observability.go | 3 +- cmd/mecatui/main.go | 2 +- deploy/helm/mecak8s/chart_test.go | 140 ++++++++++++++---- deploy/helm/mecak8s/templates/deployment.yaml | 14 +- .../templates/install-id-configmap.yaml | 29 ++++ internal/cliconfig/productmetrics.go | 24 ++- internal/cliconfig/productmetrics_test.go | 68 ++++++++- .../building/what-you-get/observability.md | 2 + 10 files changed, 252 insertions(+), 45 deletions(-) create mode 100644 deploy/helm/mecak8s/templates/install-id-configmap.yaml diff --git a/cmd/mecak8s/observability.go b/cmd/mecak8s/observability.go index 8244e4a934..1a688f6a80 100644 --- a/cmd/mecak8s/observability.go +++ b/cmd/mecak8s/observability.go @@ -87,9 +87,20 @@ func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) FlagValue: cfg.productMetrics, SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), }) + // mecak8s cannot use the local-file install-id mechanism the other three + // binaries share: it runs storage-free with no PVC (ADR 0048), so every + // pod restart would mint a fresh, never-reused id — the worst-case + // cardinality pattern for this pipeline. The Helm chart instead provisions + // ONE stable id per release in a ConfigMap (see + // deploy/helm/mecak8s/templates/install-id-configmap.yaml) and threads it + // in through this env var. Empty (the binary run directly, outside the + // chart) falls back to BuildProductMetrics's own local-file default — + // still functional, just without the "one stable id per k8s deployment" + // guarantee the chart provides. + installIDOverride := os.Getenv("MECATL_PRODUCT_METRICS_INSTALL_ID") pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, cfg.productMetricsDryRun, productmetrics.BinaryMecak8s, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productmetrics.FeatureSnapshot{Mode: productmetrics.ModeK8s}, diag) + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeK8s}, installIDOverride, diag) if pmErr != nil { // Mirror the existing telemetry-setup-failure posture: a warning, never // a fatal error — product metrics are best-effort and must not block diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index 2a52f72ffb..6c6c61d17a 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -1145,7 +1145,7 @@ func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productMetricsSnapshot(cfg), diag) + productMetricsSnapshot(cfg), "" /* no install-id override: local-file mechanism */, diag) if err != nil { slog.Warn("product metrics disabled: setup failed", "err", err) } diff --git a/cmd/mecatequi/observability.go b/cmd/mecatequi/observability.go index 8507930525..32bd40e93c 100644 --- a/cmd/mecatequi/observability.go +++ b/cmd/mecatequi/observability.go @@ -74,7 +74,8 @@ func buildObservability(ctx context.Context, f flags, diag port.Diagnostics) (ob }) pm, pmErr := cliconfig.BuildProductMetrics(ctx, context.Background(), enabled, f.productMetricsDryRun, productmetrics.BinaryMecatequi, buildinfo.BuildID, 0, /* single fire, short-lived */ - productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}, diag) + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}, + "" /* no install-id override: local-file mechanism */, diag) if pmErr != nil { // Mirror the existing telemetry-setup-failure posture: a warning, never // a fatal error — product metrics are best-effort and must not block a diff --git a/cmd/mecatui/main.go b/cmd/mecatui/main.go index a268acccfa..2c4b39d321 100644 --- a/cmd/mecatui/main.go +++ b/cmd/mecatui/main.go @@ -1038,7 +1038,7 @@ func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, productmetrics.BinaryMecatui, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productMetricsSnapshot(), diag) + productMetricsSnapshot(), "" /* no install-id override: local-file mechanism */, diag) if err != nil { diag.Log(ctx, port.LevelWarn, "mecatui: product metrics disabled: setup failed", "err", err.Error()) } diff --git a/deploy/helm/mecak8s/chart_test.go b/deploy/helm/mecak8s/chart_test.go index a5f8bc1f06..976e329374 100644 --- a/deploy/helm/mecak8s/chart_test.go +++ b/deploy/helm/mecak8s/chart_test.go @@ -101,6 +101,30 @@ func deploymentFromRender(t *testing.T, rendered string) *appsv1.Deployment { return nil } +// installIDEnvVar is the chart-provisioned product-metrics install id +// (install-id-configmap.yaml). It is UNCONDITIONAL — mecak8s is storage-free +// (ADR 0048), so the local-file mechanism the other binaries use would mint a +// fresh id on every pod restart — which means every env-shape assertion below +// is about the OTHER, opt-in variables. appEnv drops it so those assertions +// keep saying exactly what they said before it existed. +const installIDEnvVar = "MECATL_PRODUCT_METRICS_INSTALL_ID" + +// maskInstallID neutralises the per-render uuidv4 so two renders can be +// compared for equality everywhere ELSE. See install-id-configmap.yaml. +func maskInstallID(rendered string) string { + return regexp.MustCompile(`(?m)^ installId: .*$`).ReplaceAllString(rendered, " installId: MASKED") +} + +func appEnv(env []corev1.EnvVar) []corev1.EnvVar { + out := make([]corev1.EnvVar, 0, len(env)) + for _, e := range env { + if e.Name != installIDEnvVar { + out = append(out, e) + } + } + return out +} + func pdbFromRender(t *testing.T, rendered string) *policyv1.PodDisruptionBudget { t.Helper() for _, document := range strings.Split(rendered, "\n---") { @@ -528,8 +552,8 @@ func TestMecak8sHelmChart_LearningStore(t *testing.T) { t.Fatalf("default render unexpectedly contains %q", forbidden) } } - if len(container.Env) != 0 { - t.Fatalf("default learning store environment = %#v, want none", container.Env) + if env := appEnv(container.Env); len(env) != 0 { + t.Fatalf("default learning store environment = %#v, want none", env) } for _, tc := range []struct { @@ -560,8 +584,8 @@ func TestMecak8sHelmChart_LearningStore(t *testing.T) { if got := slices.Contains(container.Args, "--driver-tls"); got != tc.wantTLS { t.Fatalf("anonymous learning store --driver-tls = %t, want %t: %q", got, tc.wantTLS, container.Args) } - if len(container.Env) != 0 || strings.Contains(rendered, "MECATL_DRIVER_AUTH_TOKEN") { - t.Fatalf("anonymous learning store environment = %#v, want no driver token", container.Env) + if env := appEnv(container.Env); len(env) != 0 || strings.Contains(rendered, "MECATL_DRIVER_AUTH_TOKEN") { + t.Fatalf("anonymous learning store environment = %#v, want no driver token", env) } }) } @@ -598,8 +622,8 @@ func TestMecak8sHelmChart_LearningStore(t *testing.T) { if slices.ContainsFunc(container.Args, func(arg string) bool { return strings.HasPrefix(arg, "--driver-auth-token") }) { t.Fatalf("driver token was rendered as an argument: %q", container.Args) } - if len(container.Env) != 1 || container.Env[0].Name != "MECATL_DRIVER_AUTH_TOKEN" || container.Env[0].Value != "" || container.Env[0].ValueFrom == nil || container.Env[0].ValueFrom.SecretKeyRef == nil || container.Env[0].ValueFrom.SecretKeyRef.Name != "learning-driver-credentials" || container.Env[0].ValueFrom.SecretKeyRef.Key != "bearer-token" { - t.Fatalf("learning driver environment = %#v", container.Env) + if env := appEnv(container.Env); len(env) != 1 || env[0].Name != "MECATL_DRIVER_AUTH_TOKEN" || env[0].Value != "" || env[0].ValueFrom == nil || env[0].ValueFrom.SecretKeyRef == nil || env[0].ValueFrom.SecretKeyRef.Name != "learning-driver-credentials" || env[0].ValueFrom.SecretKeyRef.Key != "bearer-token" { + t.Fatalf("learning driver environment = %#v", env) } for _, name := range []string{"learning-store-ca", "learning-store-mtls"} { if !slices.ContainsFunc(container.VolumeMounts, func(mount corev1.VolumeMount) bool { return mount.Name == name && mount.ReadOnly }) { @@ -736,10 +760,11 @@ func TestMecak8sHelmChart_ProductionFixturesReferenceProviderCredential(t *testi if !slices.Contains(container.Args, "--default-provider=openrouter") || !slices.Contains(container.Args, "--model=anthropic/claude-sonnet-4-6") { t.Fatalf("production provider selection = %q", container.Args) } - if len(container.Env) != 1 { - t.Fatalf("provider environment = %#v, want one SecretKeyRef", container.Env) + providerEnv := appEnv(container.Env) + if len(providerEnv) != 1 { + t.Fatalf("provider environment = %#v, want one SecretKeyRef", providerEnv) } - env := container.Env[0] + env := providerEnv[0] if env.Name != "OPENROUTER_API_KEY" || env.ValueFrom == nil || env.ValueFrom.SecretKeyRef == nil || env.ValueFrom.SecretKeyRef.Name != "provider-credentials" || env.ValueFrom.SecretKeyRef.Key != "openrouter-api-key" { t.Fatalf("provider environment = %#v", env) } @@ -1344,7 +1369,10 @@ func TestMecak8sHelmChart_ServerTLS(t *testing.T) { if err != nil { t.Fatalf("render tls.enabled=false production values: %v", err) } - if defaultRender != falseRender { + // The chart-owned install id is a fresh uuidv4 on every client-side render + // (lookup finds no cluster ConfigMap), so it differs between two otherwise + // identical renders by design — mask it rather than comparing it. + if maskInstallID(defaultRender) != maskInstallID(falseRender) { t.Fatal("tls.enabled=false changed the default production render") } for _, forbidden := range []string{"--tls-cert", "--tls-key", "/var/run/secrets/tls", "name: tls", "scheme: HTTPS"} { @@ -1535,8 +1563,63 @@ func TestMecak8sHelmChart_KindLiveProviderDisablesMock(t *testing.T) { if slices.Contains(container.Args, "--mock") { t.Fatal("Kind live-provider profile retained --mock") } - if len(container.Env) != 1 || container.Env[0].Name != "OPENROUTER_API_KEY" || container.Env[0].ValueFrom == nil || container.Env[0].ValueFrom.SecretKeyRef == nil || container.Env[0].ValueFrom.SecretKeyRef.Name != "mecak8s-live-provider" { - t.Fatalf("Kind live-provider environment = %#v", container.Env) + if env := appEnv(container.Env); len(env) != 1 || env[0].Name != "OPENROUTER_API_KEY" || env[0].ValueFrom == nil || env[0].ValueFrom.SecretKeyRef == nil || env[0].ValueFrom.SecretKeyRef.Name != "mecak8s-live-provider" { + t.Fatalf("Kind live-provider environment = %#v", env) + } +} + +// TestMecak8sHelmChart_ProductMetricsInstallID pins the storage-free install-id +// contract (ADR 0048): mecak8s keeps no local state, so a per-pod install-id +// file would mint a fresh, never-reused id on every restart. The chart instead +// provisions ONE id per release in a ConfigMap and mounts it as an env var, so +// both halves — the generated value and the reference to it — must render +// unconditionally, on the bare default values as well as a production fixture. +func TestMecak8sHelmChart_ProductMetricsInstallID(t *testing.T) { + // A bare v4 UUID: nothing machine- or user-derived may appear here. + uuidV4 := regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`) + + for _, tc := range []struct { + name, release string + args []string + }{ + {name: "production", release: "production", args: productionArgs()}, + {name: "kind", release: "kind", args: kindFixtureArgs()}, + } { + t.Run(tc.name, func(t *testing.T) { + rendered, err := helm(t, tc.args...) + if err != nil { + t.Fatalf("render: %v", err) + } + + name := tc.release + "-mecak8s-install-id" + cm := configMapFromRender(t, rendered, name) + id := cm.Data["installId"] + if !uuidV4.MatchString(id) { + t.Fatalf("ConfigMap %q installId = %q, want a v4 UUID", name, id) + } + if cm.Labels["app.kubernetes.io/name"] != "mecak8s" { + t.Fatalf("ConfigMap %q labels = %#v, want the chart's standard labels", name, cm.Labels) + } + + container := deploymentFromRender(t, rendered).Spec.Template.Spec.Containers[0] + var got *corev1.EnvVar + for i := range container.Env { + if container.Env[i].Name == installIDEnvVar { + got = &container.Env[i] + } + } + if got == nil { + t.Fatalf("container environment = %#v, want %s", container.Env, installIDEnvVar) + } + // Referenced, never inlined: the id must come from the ConfigMap at + // pod start, so a `helm upgrade` that reuses the existing ConfigMap + // cannot be undone by a stale literal baked into the Deployment. + ref := got.ValueFrom + if got.Value != "" || ref == nil || ref.ConfigMapKeyRef == nil || + ref.ConfigMapKeyRef.Name != name || ref.ConfigMapKeyRef.Key != "installId" { + t.Fatalf("%s = %#v, want a configMapKeyRef to %q/installId", installIDEnvVar, *got, name) + } + }) } } @@ -1545,8 +1628,8 @@ func TestMecak8sHelmChart_ExtraEnv(t *testing.T) { if err != nil { t.Fatalf("render production values: %v", err) } - if strings.Contains(rendered, "\n env:") { - t.Fatal("default render (extraEnv unset) unexpectedly contains an env: block") + if env := appEnv(deploymentFromRender(t, rendered).Spec.Template.Spec.Containers[0].Env); len(env) != 0 { + t.Fatalf("default render (extraEnv unset) environment = %#v, want only the chart-owned install id", env) } args := append(productionArgs(), @@ -1801,19 +1884,21 @@ mcp: t.Fatalf("MCP args missing %q: %#v", want, container.Args) } } - if len(container.Env) != 1 || container.Env[0].Name != "MCP_INTERNAL_API_TOKEN" || - container.Env[0].Value != "" || container.Env[0].ValueFrom == nil || - container.Env[0].ValueFrom.SecretKeyRef == nil || - container.Env[0].ValueFrom.SecretKeyRef.Name != "internal-mcp" || - container.Env[0].ValueFrom.SecretKeyRef.Key != "bearer-token" { - t.Fatalf("static bearer environment = %#v", container.Env) + if env := appEnv(container.Env); len(env) != 1 || env[0].Name != "MCP_INTERNAL_API_TOKEN" || + env[0].Value != "" || env[0].ValueFrom == nil || + env[0].ValueFrom.SecretKeyRef == nil || + env[0].ValueFrom.SecretKeyRef.Name != "internal-mcp" || + env[0].ValueFrom.SecretKeyRef.Key != "bearer-token" { + t.Fatalf("static bearer environment = %#v", appEnv(container.Env)) } - if strings.Contains(rendered, "kind: ConfigMap") || strings.Contains(rendered, "--permission-config=/etc/mecatl-mcp/settings.yaml") { + // The install-id ConfigMap is unconditional, so the OAuth-profile probe + // names the MCP settings ConfigMap specifically rather than any ConfigMap. + if strings.Contains(rendered, "-mecak8s-mcp") || strings.Contains(rendered, "--permission-config=/etc/mecatl-mcp/settings.yaml") { t.Fatal("static/no-auth MCP render unexpectedly created an OAuth profile") } } -func TestMecak8sHelmChart_MCPNoAuthDoesNotRenderEnv(t *testing.T) { +func TestMecak8sHelmChart_MCPNoAuthRendersNoTokenEnv(t *testing.T) { rendered, err := renderMCPValues(t, ` mcp: servers: @@ -1825,11 +1910,8 @@ mcp: t.Fatalf("render no-auth MCP values: %v", err) } container := deploymentFromRender(t, rendered).Spec.Template.Spec.Containers[0] - if len(container.Env) != 0 { - t.Fatalf("no-auth MCP environment = %#v, want empty", container.Env) - } - if strings.Contains(rendered, "\n env:") { - t.Fatal("no-auth MCP render unexpectedly contains an env block") + if env := appEnv(container.Env); len(env) != 0 { + t.Fatalf("no-auth MCP environment = %#v, want empty", env) } } @@ -1997,8 +2079,8 @@ mcp: if slices.ContainsFunc(container.Args, func(arg string) bool { return strings.HasPrefix(arg, "--mcp-server=") }) { t.Fatal("broker routes must not be duplicated as legacy global flags") } - if len(container.Env) != 1 || container.Env[0].Name != "MECATL_MCP_OAUTH_REGISTERED_CLIENT_SECRET" || container.Env[0].ValueFrom == nil || container.Env[0].ValueFrom.SecretKeyRef == nil || container.Env[0].ValueFrom.SecretKeyRef.Name != "oauth-registered" || container.Env[0].ValueFrom.SecretKeyRef.Key != "client-secret" { - t.Fatalf("OAuth environment is not SecretKeyRef-only: %#v", container.Env) + if env := appEnv(container.Env); len(env) != 1 || env[0].Name != "MECATL_MCP_OAUTH_REGISTERED_CLIENT_SECRET" || env[0].ValueFrom == nil || env[0].ValueFrom.SecretKeyRef == nil || env[0].ValueFrom.SecretKeyRef.Name != "oauth-registered" || env[0].ValueFrom.SecretKeyRef.Key != "client-secret" { + t.Fatalf("OAuth environment is not SecretKeyRef-only: %#v", env) } cm := configMapFromRender(t, rendered, "production-mecak8s-mcp") profile := cm.Data["settings.yaml"] diff --git a/deploy/helm/mecak8s/templates/deployment.yaml b/deploy/helm/mecak8s/templates/deployment.yaml index ff3517e5a7..fce175fc8d 100644 --- a/deploy/helm/mecak8s/templates/deployment.yaml +++ b/deploy/helm/mecak8s/templates/deployment.yaml @@ -195,10 +195,17 @@ spec: {{- if $mcpSettings }} - --permission-config=/etc/mecatl-mcp/settings.yaml {{- end }} - {{- $mcpEnvServers := false }} - {{- range .Values.mcp.servers }}{{- if ne .auth.mode "none" }}{{- $mcpEnvServers = true }}{{- end }}{{- end }} - {{- if or .Values.extraEnv $mcpEnvServers $learningStoreTokenConfigured }} env: + {{- /* The chart-provisioned, stable-per-release product-metrics + install id. Always present: mecak8s is storage-free (ADR + 0048), so the local-file mechanism the other binaries use + would mint a fresh id on every pod restart. See + install-id-configmap.yaml. */}} + - name: MECATL_PRODUCT_METRICS_INSTALL_ID + valueFrom: + configMapKeyRef: + name: {{ include "mecak8s.fullname" . }}-install-id + key: installId {{- if $learningStoreTokenConfigured }} - name: MECATL_DRIVER_AUTH_TOKEN valueFrom: @@ -222,7 +229,6 @@ spec: {{- with .Values.extraEnv }} {{- toYaml . | nindent 12 }} {{- end }} - {{- end }} ports: - {name: grpc, containerPort: 8080} - {name: http, containerPort: 8081} diff --git a/deploy/helm/mecak8s/templates/install-id-configmap.yaml b/deploy/helm/mecak8s/templates/install-id-configmap.yaml new file mode 100644 index 0000000000..67813206e3 --- /dev/null +++ b/deploy/helm/mecak8s/templates/install-id-configmap.yaml @@ -0,0 +1,29 @@ +{{/* +Generates ONE stable install-id for this Helm release, shared by every replica +and preserved across every `helm upgrade` — unlike a per-pod local file, which +mecak8s cannot use at all: it runs storage-free, no PVC (ADR 0048), so every pod +restart would otherwise mint a fresh, never-reused id — the worst-case +cardinality pattern for the product-metrics pipeline this feeds. + +The `lookup` guard is the standard Helm idiom for "generate once, keep stable on +upgrade": when a ConfigMap of this name already exists in the release namespace +its EXISTING value is reused verbatim, so only a genuinely first `helm install` +(or a deliberately deleted ConfigMap) mints a new id. `lookup` returns an empty +map for a client-side render (`helm template`, `--dry-run`), so those renders +show a throwaway id rather than the cluster's — expected, and never written. +*/}} +{{- $existing := lookup "v1" "ConfigMap" .Release.Namespace (printf "%s-install-id" (include "mecak8s.fullname" .)) }} +{{- $installID := "" }} +{{- if and $existing $existing.data }} +{{- $installID = index $existing.data "installId" }} +{{- end }} +{{- if not $installID }} +{{- $installID = uuidv4 }} +{{- end }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "mecak8s.fullname" . }}-install-id + labels: {{- include "mecak8s.labels" . | nindent 4 }} +data: + installId: {{ $installID | quote }} diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index c60b398bd8..658f524a5a 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -62,6 +62,16 @@ type ProductMetricsHandles struct { // short-lived process like mecatequi can exit before an unawaited goroutine // ever runs) and returns handles wrapping the DryRunRecorder as both Sink // and ToolCallRecorder. +// +// installIDOverride, when non-empty, is used verbatim as the install id and +// the local-file mechanism (LoadOrCreateInstallIDDefault) is skipped entirely. +// It exists for mecak8s, which runs storage-free with no PVC (ADR 0048): its +// Helm chart provisions ONE stable id per release in a ConfigMap and threads +// it in via MECATL_PRODUCT_METRICS_INSTALL_ID, because a local file would mint +// a fresh, never-reused id on every pod restart. An override never reports +// FirstRun (nothing was minted here, and the chart — not this process — owns +// the id's lifecycle). Every other binary passes "" and keeps the local-file +// behaviour unchanged. func BuildProductMetrics( ctx, heartbeatCtx context.Context, enabled, dryRun bool, @@ -69,6 +79,7 @@ func BuildProductMetrics( version string, heartbeatInterval time.Duration, snap productmetrics.FeatureSnapshot, + installIDOverride string, diag port.Diagnostics, ) (ProductMetricsHandles, error) { noop := func(context.Context) error { return nil } @@ -85,10 +96,15 @@ func BuildProductMetrics( // local install-id file and reports firstRun for the disclosure notice // below. Reinstated as a real, exported resource attribute (see // provider.go's doc comment) after its cardinality cost was sized and - // accepted. - installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() - if err != nil { - return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + // accepted. An externally provisioned id (see installIDOverride) bypasses + // it: there is no file to read, write, or report a first run from. + installID, firstRun := installIDOverride, false + if installID == "" { + var err error + installID, firstRun, err = productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } } provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go index 57a5ef8de9..87125ac262 100644 --- a/internal/cliconfig/productmetrics_test.go +++ b/internal/cliconfig/productmetrics_test.go @@ -2,6 +2,7 @@ package cliconfig import ( "context" + "strings" "testing" "github.com/stacklok/mecatl/engine/port" @@ -10,7 +11,7 @@ import ( func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), false, false, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) if err != nil { t.Fatalf("BuildProductMetrics(enabled=false): %v", err) } @@ -30,7 +31,7 @@ func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { // surface the error rather than silently disabling, so a caller notices // its release build is missing the ldflag. _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) if err == nil { t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") } @@ -43,7 +44,7 @@ func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { // case (above) fails closed with no baked key. func TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), true, true, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) if err != nil { t.Fatalf("BuildProductMetrics(enabled=true, dryRun=true): %v", err) } @@ -66,7 +67,7 @@ func TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider(t *testing // regardless of the dry-run flag's value. func TestBuildProductMetricsDisabledDryRunStillNoop(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), false, true, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) if err != nil { t.Fatalf("BuildProductMetrics(enabled=false, dryRun=true): %v", err) } @@ -74,3 +75,62 @@ func TestBuildProductMetricsDisabledDryRunStillNoop(t *testing.T) { t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder even with dryRun=true: %+v", h) } } + +// TestBuildProductMetricsInstallIDOverrideSkipsTheLocalFile is the mecak8s +// contract (storage-free, no PVC, ADR 0048): a chart-provisioned install id +// must bypass LoadOrCreateInstallIDDefault ENTIRELY, not merely take +// precedence over whatever it returns. +// +// The oracle is which failure surfaces. With no resolvable state directory the +// local-file mechanism cannot even mint an id, so the no-override call fails at +// the install-id step; an override must get PAST that step and fail later, at +// provider construction (no baked ingest key in any non-release build). If the +// override were applied after the file read, both calls would report the same +// install-id error. +func TestBuildProductMetricsInstallIDOverrideSkipsTheLocalFile(t *testing.T) { + // No XDG_STATE_HOME and no home dir => productmetrics.UserStateDir yields + // "" and LoadOrCreateInstallIDDefault fails closed. + t.Setenv("XDG_STATE_HOME", "") + t.Setenv("HOME", "") + + _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, + "", port.NopDiagnostics{}) + if err == nil || !strings.Contains(err.Error(), "install id") { + t.Fatalf("no override with an unresolvable state dir: err = %v, want an install-id failure", err) + } + + _, err = BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, + "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}) + if err == nil || strings.Contains(err.Error(), "install id") { + t.Fatalf("override with an unresolvable state dir: err = %v, want the install-id step skipped", err) + } +} + +// TestBuildProductMetricsInstallIDOverrideNeverReportsFirstRun pins the +// disclosure-notice half: an override mints nothing locally, so this process +// has no first run to announce — the chart owns the id's lifecycle. Asserted +// on the disabled and dry-run paths, the only two that return handles without +// a baked ingest key. +func TestBuildProductMetricsInstallIDOverrideNeverReportsFirstRun(t *testing.T) { + for _, tc := range []struct { + name string + enabled, dryRun bool + }{ + {name: "disabled", enabled: false}, + {name: "dry run", enabled: true, dryRun: true}, + } { + t.Run(tc.name, func(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), tc.enabled, tc.dryRun, + productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, + "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}) + if err != nil { + t.Fatalf("BuildProductMetrics with an install-id override: %v", err) + } + if h.FirstRun { + t.Error("an externally provisioned install id must never report FirstRun") + } + }) + } +} diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index a46f642281..64183e3884 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -245,6 +245,8 @@ The four channels above are all **operator-facing**: they help you observe your **Self-verify before trusting it.** `--product-metrics-dry-run` prints every observation this pipeline would have sent to stderr instead of exporting it, so you can check the "no PII" claim yourself rather than take the docs' word for it. +**On Kubernetes.** The reported install identifier is an anonymous random UUID. `mecated`, `mecatui`, and `mecatequi` keep it in a local state file, but `mecak8s` is storage-free (no PVC), so a per-pod file would produce a brand-new id on every restart. The Helm chart therefore provisions the id once, into a `-mecak8s-install-id` ConfigMap that is reused across every `helm upgrade`, and mounts it into the container as `MECATL_PRODUCT_METRICS_INSTALL_ID`. Delete that ConfigMap to reset the id, or opt out entirely with any of the switches above — the env var only decides *which* id is used when reporting is on. + --- ## What's next From 28da4ddf71f9b52be4c08fd9f276e6fe172773ee Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 19:23:56 -0400 Subject: [PATCH 36/47] docs: document had_tool_call, tool category/outcome, and the install.id reinstatement Updates ADR 0319's catalog table and adds two new sections covering this round's changes: the RunAwareToolCallRecorder engine port extension (why it exists, how the had_tool_call/category/outcome derivation works, and why category needs a maintained allowlist alongside the mcp__ prefix rule) and mecak8s's Helm ConfigMap install-id mechanism (why the local-file approach can't work there, how the lookup/uuidv4 idiom keeps it stable across upgrades). Updates the startup disclosure notice and the user-facing observability doc to honestly reflect that an anonymous per-install identifier is now collected, reversing the prior "no ... identifier" framing now that its reintroduction has been sized and accepted. Co-Authored-By: Claude Sonnet 5 --- docs/adr/0319-product-metrics.md | 135 +++++++++++++++--- internal/cliconfig/productmetrics.go | 7 +- .../building/what-you-get/observability.md | 2 +- 3 files changed, 122 insertions(+), 22 deletions(-) diff --git a/docs/adr/0319-product-metrics.md b/docs/adr/0319-product-metrics.md index 701b2e269e..bbd4ae52b7 100644 --- a/docs/adr/0319-product-metrics.md +++ b/docs/adr/0319-product-metrics.md @@ -2,7 +2,7 @@ - Status: Accepted - Date: 2026-09-09 -- Scope: `internal/adapter/productmetrics` (new), `internal/cliconfig`, `internal/adapter/permconfig` (new `telemetry:` operator section), `cmd/mecated`, `cmd/mecatui`, `cmd/mecatequi`, `cmd/mecak8s` +- Scope: `internal/adapter/productmetrics` (new), `internal/cliconfig`, `internal/adapter/permconfig` (new `telemetry:` operator section), `engine/port` (new `RunAwareToolCallRecorder`), `engine/agent` (dispatch wiring), `cmd/mecated`, `cmd/mecatui`, `cmd/mecatequi`, `cmd/mecak8s`, `deploy/helm/mecak8s` - Supersedes: — - Superseded by: — @@ -97,20 +97,32 @@ looking at either series family can mistake one for the other. **Resource attributes** (set once per process via `productmetrics.Config`, not per-metric labels): `service.name=mecatl`, `service.version`, +`mecatl.install.id` (a per-install random UUID — see below), `mecatl.binary` (one of the closed `Binary` set: `mecated`/`mecatui`/`mecatequi`/`mecak8s`). -**No per-install identifier is attached, deliberately.** This pipeline's -destination is a Prometheus-remote-write backend (`stacklok/infra#5604`), -where every resource attribute becomes a permanent label on *every* -instrument's time series. A random per-install UUID would multiply active -series by (installs × instrument count) with no bound as adoption grows — -an unbounded-cardinality cost for a precision (exact unique-install counts) -the design never actually required. `installid.go` still persists a local -UUID, but purely as a first-run marker for the disclosure notice (§ below) -— its value is never read back by `NewProvider` or attached to anything -exported. Unique-install counts are approximated from heartbeat volume/ -cadence instead of counted exactly. +**A per-install identifier IS attached, after being sized and accepted.** +This pipeline's destination is a Prometheus-remote-write backend +(`stacklok/infra#5604`), where every resource attribute becomes a permanent +label on *every* instrument's time series. A random per-install UUID was +initially left out over exactly this concern — it multiplies active series +by (installs × instrument count), an unbounded-cardinality cost that grows +with adoption. It was reinstated after sizing that cost against the actual +backend (Amazon Managed Service for Prometheus, per `stacklok/infra#5604`): +roughly $1,930/month at 100,000 installs under a worst-case assumption (every +instrument in this catalog, 24/7 uptime), and roughly $650/month under a +more realistic assumption (an interactive CLI tool, ~8h/day active use) — +both well within what the adoption signal an exact per-install breakdown +enables (activation rate, time-to-first-value, weekly retention) is worth to +the product team, and a defensible ceiling rather than a runaway. +`installid.go` still persists a local UUID; its value is now threaded into +`Config.InstallID` and attached as the `mecatl.install.id` resource +attribute. `mecak8s` — storage-free, no PVC (ADR 0048) — cannot use that +same local-file mechanism (every pod restart would mint a fresh, never-reused +id, the worst-case cardinality pattern this pipeline could hit); its Helm +chart instead provisions one stable id per release via a Kubernetes +ConfigMap (see "mecak8s's install-id: a Helm ConfigMap, not a local file" +below). **Heartbeat** (`metrics.go`/`heartbeat.go` — on start, then every ~24h for long-running processes; single fire for `mecatequi`): @@ -129,11 +141,15 @@ normal periodic-reader cadence since these are cumulative counters): | Instrument | Kind | Attributes | Fires on | |---|---|---|---| | `mecatl.product.sessions_started` | counter | none | `EvSessionInit` | -| `mecatl.product.runs_completed` | counter | `stop` (reuses `session.StopReason`) | `EvResult` | -| `mecatl.product.tool_calls` | counter | none — **no tool/MCP-server name label at all** | every `ToolCallRecorder.ToolCall` | +| `mecatl.product.runs_completed` | counter | `stop` (reuses `session.StopReason`), `had_tool_call` (`true`/`false`: at least one SUCCESSFUL tool call in the run — an errored-only run is `false`) | `EvResult` | +| `mecatl.product.tool_calls` | counter | `category` (a built-in tool's own name from a closed allowlist, `mcp` for anything MCP-server-provided via the structural `mcp__` name prefix, or `other` for anything unrecognized — **never a raw MCP server/tool name**), `outcome` (`success`/`error`) | every `ToolCallRecorder.ToolCall`/`RunAwareToolCallRecorder.ToolCallForRun` | | `mecatl.product.tokens` | counter | `kind` (`input`/`output`/`cache_read`/`cache_write`/`reasoning`) | `EvResult`'s `Usage` | | `mecatl.product.subagent_used` | counter | none | `EvSubagentStart` | | `mecatl.product.team_used` | counter | none | `EvTeamStart` | +| `mecatl.product.run_duration` | histogram (seconds) | none | `EvResult`, when this process observed the run's `EvSessionInit` | +| `mecatl.product.tool_calls_per_run` | histogram (count) | none | every `EvResult` (a tool-less run contributes an honest 0) | +| `mecatl.product.time_to_first_value` | histogram (seconds) | none | at most once per install, ever, on the first run with `had_tool_call=true` and `stop=StopEndTurn` | + **Deliberate scope refinement from the original design spec.** The spec's illustrative heartbeat catalog listed `teams`/`subagents`/`learning` as @@ -255,6 +271,85 @@ one and the balance shifts back toward requiring opt-in: trusting `--product-metrics` for real, rather than trusting either the docs or the code review that produced them. +### Correlating a tool call to its run: `port.RunAwareToolCallRecorder` + +`had_tool_call`, `tool_calls_per_run`, and `time_to_first_value` all need to +know which *run* a given tool call belongs to. The existing +`port.ToolCallRecorder.ToolCall(id session.SessionID, ...)` only carries a +`SessionID` — a session can span many sequential runs over its lifetime, so a +`SessionID` alone cannot answer "did this run make a tool call". The engine's +other event-sourced callback, `port.EventSink.Emit(ctx, session.Event)`, has +the opposite problem: `session.Event.RunID` is present, but `Emit` never sees +a tool call at all. + +The fix is a new, standalone, OPTIONAL port interface — +`port.RunAwareToolCallRecorder` — added to `engine/port/log.go`, mirroring the +existing `HookApprovalLearner` precedent (`engine/port/hookrunner.go`) exactly: + +```go +type RunAwareToolCallRecorder interface { + ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, + result session.ToolResult, queued, took time.Duration) +} +``` + +`engine/agent/dispatch.go`'s one `ToolCallRecorder` call site type-asserts for +this richer interface and prefers it (passing the enclosing `Run`'s own +`RunID()`, already in scope — no new parameter threading needed anywhere in +the call chain) when a recorder implements it, falling back to the plain +`ToolCall` otherwise. This is purely additive: no existing `ToolCallRecorder` +implementer (the operator telemetry pipeline, `jsonlstore`, `redisstore`) is +affected, and `engine/CHANGELOG.md` records it as `Added` (minor) per +`engine/COMPATIBILITY.md`. + +`internal/adapter/productmetrics.Recorder` implements +`RunAwareToolCallRecorder` (`toolcall.go`): `ToolCallForRun` records the +bounded `category`/`outcome` attributes on `tool_calls` and tallies a +per-run state (`hadToolCall`, `toolCallCount`, `startedAt`) keyed by `RunID` +in a mutex-guarded `perRunTracker`, cleared at the run's `EvResult`. The +category derivation is a closed-set projection, `toolCategory` +(`toolcall.go`): the `mcp__` structural name prefix (`internal/adapter/mcp`'s +`"mcp__" + server + "__" + toolName` construction) buckets every MCP-server +tool under the single literal `"mcp"` with no allowlist needed; every other +name is checked against a maintained `builtinToolCategories` allowlist +(mecatl's own fixed tool catalog), falling back to the literal `"other"` for +anything unrecognized. A hand-maintained allowlist for non-MCP tools was not +the original design (a pure structural-prefix rule was) — it became necessary +because this package's existing privacy-guard test (`bounded_test.go`) +correctly rejects any tool name reaching an attribute value verbatim, and a +prefix-only rule would let an agent-def name, a learned-skill name, or a +future extension-seam name (all potentially operator- or model-derived free +text) leak straight onto an exported counter. The allowlist trades "zero +maintenance for new built-ins" for "structurally impossible to leak" — a new +built-in tool shows up as `"other"` until a line is added, which is visible +and harmless, never a leak. + +### mecak8s's install-id: a Helm ConfigMap, not a local file + +`mecak8s` runs storage-free with no PVC (ADR 0048) — the local-file mechanism +`installid.go` uses for the other three binaries would mint a fresh, +never-reused install id on every pod restart, the worst-case cardinality +pattern this pipeline could hit (every replica of every deployment counted as +a distinct, ever-churning "install"). Its Helm chart +(`deploy/helm/mecak8s/templates/install-id-configmap.yaml`) instead +provisions ONE stable id per release into a `-mecak8s-install-id` +ConfigMap, using Helm's standard `lookup`+`uuidv4` "generate once, keep stable +on upgrade" idiom: a `lookup` against the release namespace for an existing +ConfigMap of that name reuses its `installId` value verbatim on every `helm +upgrade` (and safely re-mints a fresh id if the lookup finds no usable value — +a nil `.data` map or a missing key degrade to "mint a new one", never an +error or an empty string); only a genuinely first `helm install` mints a new +id. `lookup` runs with the Helm client's own credentials at render time, not +the pod's ServiceAccount at runtime, so no RBAC grant was needed. The id is +threaded into the container via a `MECATL_PRODUCT_METRICS_INSTALL_ID` +environment variable (`configMapKeyRef`), which `cmd/mecak8s/observability.go` +reads and passes to `internal/cliconfig.BuildProductMetrics`'s new +`installIDOverride` parameter — non-empty skips `LoadOrCreateInstallIDDefault` +entirely and never reports `FirstRun` (the chart, not the process, owns the +id's lifecycle). The other three binaries pass `""` and keep the local-file +behavior unchanged. Deleting the ConfigMap resets the id, the same as +deleting the local file does for the other three binaries. + ## Consequences - A new direct dependency surface: `github.com/stacklok/toolhive-core/telemetry/providers` @@ -271,10 +366,14 @@ one and the balance shifts back toward requiring opt-in: disabled-pipeline posture when a non-release build carries no baked ingest key (`bakedKey == ""` refuses `NewProvider` outright) — a local `go build`/ `go test`/CI build can never phone home regardless of flag state. -- A new small persisted file per install - (`$XDG_STATE_HOME/mecatl/telemetry-id`, a bare random v4 UUID) — trivially - reset by deleting it, carrying no machine or user information, and never - read back for export: it exists purely as a local first-run marker. +- Two small persisted files per install, both under `$XDG_STATE_HOME/mecatl/`: + `telemetry-id` (a bare random v4 UUID, now attached as the + `mecatl.install.id` resource attribute — see the reinstatement rationale + above) and `first-value-recorded` (a bare marker recording whether + `time_to_first_value` has already been sampled). Both are trivially reset + by deleting them, carry no machine or user information, and (on + `mecak8s`) are replaced entirely by the Helm-provisioned ConfigMap — see + "mecak8s's install-id" above. - ADR-0027 List-1 (resource inventory) is NOT extended: the heartbeat ticker's lifetime matches the process (owned by the caller's `heartbeatCtx`, cancelled on shutdown alongside the rest of composition's diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 658f524a5a..ef962ad1d9 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -22,9 +22,10 @@ import ( // telemetry without a visible disclosure is the pattern that burns // community trust; this is the whole of that disclosure. const ProductMetricsDisclosureNotice = `mecatl reports anonymous product-adoption metrics (version, OS/arch, which -major features you have enabled, and coarse session/run/tool-call counts — -never a prompt, file path, tool name, or model id) to help Stacklok understand -community adoption. This is on by default. To opt out: pass +major features you have enabled, an anonymous per-install identifier, and +coarse session/run/tool-call counts by bounded category — never a prompt, +file path, raw tool name, or model id) to help Stacklok understand community +adoption. This is on by default. To opt out: pass --product-metrics=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: see docs/adr/0319-product-metrics.md. diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 64183e3884..c167e3f7cb 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -235,7 +235,7 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. -**What's collected:** version, OS/arch, which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason), tool calls executed (no tool name), token counts by kind, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, tool name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0319](https://github.com/stacklok/mecatl/blob/main/docs/adr/0319-product-metrics.md). +**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0319](https://github.com/stacklok/mecatl/blob/main/docs/adr/0319-product-metrics.md). **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: From 3526a63290057a4d68e6512c29391c53bb9d205f Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 19:53:36 -0400 Subject: [PATCH 37/47] fix(productmetrics): forward RunAwareToolCallRecorder through both composition decorators The final whole-branch review of the had_tool_call/tool_calls_per_run/ time_to_first_value work found that neither decorator between productmetrics.Recorder and Deps.ToolCallRecorder implemented the new port.RunAwareToolCallRecorder interface: cliconfig.multiToolCallRecorder (TeeToolCallRecorder's fan-out) and internal/adapter/server's capabilityToolCallRecorder (SessionMutationCapability.GuardToolCallRecorder) both had only ToolCall. Since dispatch.go's type-assertion is on the composed Deps.ToolCallRecorder value itself, it always failed and fell back to plain ToolCall -- silently making had_tool_call, tool_calls_per_run, and time_to_first_value inert in every real binary, despite every per-task test and gate passing (nothing exercised the recorder the composition layer actually installs). Both decorators now implement ToolCallForRun, forwarding to each wrapped recorder's richer form when available and falling back to ToolCall otherwise -- capabilityToolCallRecorder under the same capability.allows(id) gate ToolCall already uses. Adds a regression test in each package asserting the composed/guarded value still satisfies port.RunAwareToolCallRecorder, driven the same way dispatch.go drives it (type-assert, then call). Co-Authored-By: Claude Sonnet 5 --- .../adapter/server/mutation_capability.go | 24 ++++++ .../mutation_capability_toolcall_test.go | 82 +++++++++++++++++++ internal/cliconfig/productmetrics_config.go | 25 ++++++ .../cliconfig/productmetrics_config_test.go | 57 +++++++++++++ 4 files changed, 188 insertions(+) create mode 100644 internal/adapter/server/mutation_capability_toolcall_test.go diff --git a/internal/adapter/server/mutation_capability.go b/internal/adapter/server/mutation_capability.go index 3df558bbcf..922b9cd6b4 100644 --- a/internal/adapter/server/mutation_capability.go +++ b/internal/adapter/server/mutation_capability.go @@ -117,3 +117,27 @@ func (r capabilityToolCallRecorder) ToolCall(id session.SessionID, call session. r.next.ToolCall(id, call, result, queued, took) } } + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder, forwarding to +// r.next's richer form under the SAME capability.allows(id) gate ToolCall +// uses. Without this method, wrapping a RunAwareToolCallRecorder-capable +// recorder (e.g. productmetrics.Recorder) behind this capability guard would +// ERASE the optional capability: the engine's dispatch.go type-assertion is +// on Deps.ToolCallRecorder itself, so a wrapper implementing only the base +// interface fails that assertion regardless of what it wraps, silently +// disabling had_tool_call/tool_calls_per_run/time_to_first_value. +func (r capabilityToolCallRecorder) ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + if !r.capability.allows(id) { + return + } + if aware, ok := r.next.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(runID, id, call, result, queued, took) + return + } + r.next.ToolCall(id, call, result, queued, took) +} + +// Compile-time interface check: capabilityToolCallRecorder must keep +// forwarding port.RunAwareToolCallRecorder, or the same capability silently +// goes inert whenever a productmetrics.Recorder is guarded by it. +var _ port.RunAwareToolCallRecorder = capabilityToolCallRecorder{} diff --git a/internal/adapter/server/mutation_capability_toolcall_test.go b/internal/adapter/server/mutation_capability_toolcall_test.go new file mode 100644 index 0000000000..61397725c7 --- /dev/null +++ b/internal/adapter/server/mutation_capability_toolcall_test.go @@ -0,0 +1,82 @@ +package server + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// runAwareRecorderFunc implements BOTH port.ToolCallRecorder and +// port.RunAwareToolCallRecorder, recording which method was actually called. +type runAwareRecorderFunc struct { + plainCalls *int + runAwareCalls *[]string +} + +func (f runAwareRecorderFunc) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + *f.plainCalls++ +} + +func (f runAwareRecorderFunc) ToolCallForRun(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + *f.runAwareCalls = append(*f.runAwareCalls, runID) +} + +// TestCapabilityToolCallRecorderForwardsRunAwareCapability pins the bug found +// in the final whole-branch review of the had_tool_call/tool_calls_per_run/ +// time_to_first_value work: GuardToolCallRecorder's returned value is what +// lands in Deps.ToolCallRecorder, so if it implemented only the base +// port.ToolCallRecorder, the engine's dispatch.go type-assertion for +// port.RunAwareToolCallRecorder would ALWAYS fail — silently making +// had_tool_call/tool_calls_per_run/time_to_first_value inert in every real +// binary, despite the wrapped recorder correctly implementing the richer +// interface. This test drives the guarded value exactly the way dispatch.go +// does: type-assert, then call ToolCallForRun if it succeeds. +func TestCapabilityToolCallRecorderForwardsRunAwareCapability(t *testing.T) { + plainCalls := 0 + var runAwareCalls []string + next := runAwareRecorderFunc{plainCalls: &plainCalls, runAwareCalls: &runAwareCalls} + + mc := NewSessionMutationCapability(false) // disabled gate: allows(id) always true + guarded := mc.GuardToolCallRecorder(next) + + aware, ok := guarded.(port.RunAwareToolCallRecorder) + if !ok { + t.Fatal("GuardToolCallRecorder's result does not implement port.RunAwareToolCallRecorder — had_tool_call/tool_calls_per_run/time_to_first_value would be inert in production") + } + aware.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if plainCalls != 0 { + t.Errorf("plainCalls = %d, want 0 (the run-aware form must be preferred)", plainCalls) + } + if len(runAwareCalls) != 1 || runAwareCalls[0] != "run-1" { + t.Errorf("runAwareCalls = %v, want [run-1]", runAwareCalls) + } +} + +// TestCapabilityToolCallRecorderGatesRunAwareOnAllows confirms +// ToolCallForRun respects the SAME capability.allows(id) gate ToolCall uses: +// once a session's mutation capability is invalidated, neither method should +// reach the wrapped recorder. +func TestCapabilityToolCallRecorderGatesRunAwareOnAllows(t *testing.T) { + plainCalls := 0 + var runAwareCalls []string + next := runAwareRecorderFunc{plainCalls: &plainCalls, runAwareCalls: &runAwareCalls} + + mc := NewSessionMutationCapability(true) + id := session.SessionID("s") + mc.Grant(id) + mc.Invalidate(id) + + guarded := mc.GuardToolCallRecorder(next) + aware, ok := guarded.(port.RunAwareToolCallRecorder) + if !ok { + t.Fatal("GuardToolCallRecorder's result does not implement port.RunAwareToolCallRecorder") + } + aware.ToolCallForRun("run-1", id, session.ToolCall{}, session.ToolResult{}, 0, 0) + + if plainCalls != 0 || len(runAwareCalls) != 0 { + t.Errorf("plainCalls=%d runAwareCalls=%v, want both empty (invalidated capability must block ToolCallForRun same as ToolCall)", plainCalls, runAwareCalls) + } +} diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go index 93986ffad1..7a02c009cc 100644 --- a/internal/cliconfig/productmetrics_config.go +++ b/internal/cliconfig/productmetrics_config.go @@ -81,7 +81,32 @@ func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecord type multiToolCallRecorder []port.ToolCallRecorder func (m multiToolCallRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + m.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder: it forwards runID to +// any element that implements the richer interface, and falls back to that +// element's plain ToolCall otherwise. Without this method, wrapping a +// RunAwareToolCallRecorder-capable recorder (e.g. productmetrics.Recorder) in +// a multiToolCallRecorder would ERASE the optional capability — the engine's +// dispatch.go type-assertion is on Deps.ToolCallRecorder itself, and a +// composed value that only implements the base interface fails that +// assertion regardless of what it wraps. This is the general hazard with +// decorating an optional-capability interface: every decorator in the chain +// must forward it, or the capability silently stops reaching the type that +// actually needs it. +func (m multiToolCallRecorder) ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { for _, r := range m { + if aware, ok := r.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(runID, id, call, result, queued, took) + continue + } r.ToolCall(id, call, result, queued, took) } } + +// Compile-time interface check: multiToolCallRecorder must keep forwarding +// port.RunAwareToolCallRecorder, or had_tool_call/tool_calls_per_run/ +// time_to_first_value silently go inert in every binary that tees a +// productmetrics.Recorder through this helper. +var _ port.RunAwareToolCallRecorder = multiToolCallRecorder(nil) diff --git a/internal/cliconfig/productmetrics_config_test.go b/internal/cliconfig/productmetrics_config_test.go index 3d0f753f21..b0adbbd1ef 100644 --- a/internal/cliconfig/productmetrics_config_test.go +++ b/internal/cliconfig/productmetrics_config_test.go @@ -58,3 +58,60 @@ type recorderFunc func(session.SessionID, session.ToolCall, session.ToolResult, func (f recorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { f(id, call, result, queued, took) } + +// runAwareRecorderFunc additionally implements port.RunAwareToolCallRecorder, +// so tests can distinguish which method a caller actually invoked. +type runAwareRecorderFunc struct { + plain func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) + runAware func(string, session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) +} + +func (f runAwareRecorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f.plain(id, call, result, queued, took) +} + +func (f runAwareRecorderFunc) ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f.runAware(runID, id, call, result, queued, took) +} + +// TestTeeToolCallRecorderForwardsRunAwareCapability pins the bug found in the +// final whole-branch review of the had_tool_call/tool_calls_per_run/ +// time_to_first_value work: TeeToolCallRecorder's returned value is what +// lands in Deps.ToolCallRecorder, so if it implemented only the base +// port.ToolCallRecorder, the engine's dispatch.go type-assertion for +// port.RunAwareToolCallRecorder would ALWAYS fail — silently making +// had_tool_call/tool_calls_per_run/time_to_first_value inert in every real +// binary, despite productmetrics.Recorder itself correctly implementing the +// richer interface. This test drives the composed value exactly the way +// dispatch.go does: type-assert, then call ToolCallForRun if it succeeds. +func TestTeeToolCallRecorderForwardsRunAwareCapability(t *testing.T) { + var runAwareCalls []string + var plainCalls []string + + runAware := runAwareRecorderFunc{ + plain: func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + plainCalls = append(plainCalls, "runaware-recorder-plain") + }, + runAware: func(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + runAwareCalls = append(runAwareCalls, runID) + }, + } + baseOnly := recorderFunc(func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + plainCalls = append(plainCalls, "base-only-recorder") + }) + + tee := TeeToolCallRecorder(baseOnly, runAware) + + aware, ok := tee.(port.RunAwareToolCallRecorder) + if !ok { + t.Fatal("TeeToolCallRecorder's result does not implement port.RunAwareToolCallRecorder — had_tool_call/tool_calls_per_run/time_to_first_value would be inert in production") + } + aware.ToolCallForRun("run-1", session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if len(runAwareCalls) != 1 || runAwareCalls[0] != "run-1" { + t.Errorf("runAwareCalls = %v, want the run-aware element to receive ToolCallForRun with runID %q", runAwareCalls, "run-1") + } + if len(plainCalls) != 1 || plainCalls[0] != "base-only-recorder" { + t.Errorf("plainCalls = %v, want the base-only element to fall back to ToolCall exactly once", plainCalls) + } +} From 1d03b142a726b3443ab3c4d1bd4550cd5640e78a Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 19:54:00 -0400 Subject: [PATCH 38/47] fix(productmetrics): mecak8s time_to_first_value durability, untracked-run bias, doc drift Three more findings from the final whole-branch review: - mecak8s has no durable local marker for time_to_first_value's once-ever contract (the same storage-free problem, ADR 0048, install-id solves via a Helm ConfigMap -- time_to_first_value never got the same treatment). armFirstValueTracking now no-ops entirely when installIDOverride is non-empty, so mecak8s simply doesn't report this metric rather than silently emitting one sample per pod restart under a "once per install, ever" label. - perRunTracker.finish now returns (perRunState, tracked bool): tracked is false for an empty RunID or a RunID this Recorder never saw an EvSessionInit/tool call for (e.g. RetryFailedStep's RunRequest{}, which mints no RunID). recordResult and DryRunRecorder now skip tool_calls_per_run entirely for an untracked run instead of recording a fabricated 0 -- a 0 there was indistinguishable from a genuine zero-tool-call run and would have biased the distribution downward for a retried run that may have made many real tool calls. - ADR 0319 doc drift: a stale "deliberately never exported" line two sections above the section that reverses it, a privacy paragraph that both said "never a tool name" three rows below a table showing tool_calls' category attribute IS one (bounded, allowlisted, but a name) and cited the wrong file for the closed set, and a stale "a later task publishes" comment in toolcall.go now that task has landed. The bounded_test.go guard's vacuity floor is tightened from a "< 10" floor to an exact 13 (NewRecorder's real instrument count), and its driving code now exercises run_duration too (it previously never observed an EvSessionInit for its run-aware scenario). Co-Authored-By: Claude Sonnet 5 --- docs/adr/0319-product-metrics.md | 12 +++-- .../adapter/productmetrics/bounded_test.go | 17 +++++-- internal/adapter/productmetrics/dryrun.go | 12 +++-- internal/adapter/productmetrics/metrics.go | 38 ++++++++++----- .../adapter/productmetrics/metrics_test.go | 31 +++++++++++- internal/adapter/productmetrics/toolcall.go | 2 +- .../adapter/productmetrics/toolcall_test.go | 14 ++++-- internal/cliconfig/productmetrics.go | 28 +++++++++-- internal/cliconfig/productmetrics_test.go | 48 +++++++++++++++++++ 9 files changed, 171 insertions(+), 31 deletions(-) diff --git a/docs/adr/0319-product-metrics.md b/docs/adr/0319-product-metrics.md index bbd4ae52b7..5dda955c79 100644 --- a/docs/adr/0319-product-metrics.md +++ b/docs/adr/0319-product-metrics.md @@ -67,7 +67,8 @@ Zero import relationship with `internal/adapter/telemetry`. It owns: mirroring the OTLP-push-with-flush precedent ADR 0098 already established for that binary's shape. - Its own local install-identity file (`installid.go`) — a first-run marker - only, deliberately never exported (see the cardinality note below). + whose UUID value is now also attached as the `mecatl.install.id` resource + attribute (see "A per-install identifier IS attached" below). Composition combines the two independent sinks with a trivial fan-out helper in `internal/cliconfig` (`BuildProductMetrics`, `TeeToolCallRecorder`) — the @@ -169,9 +170,12 @@ signal than a static capability flag. This ADR records the *shipped* set; readers should treat the design spec (`docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md`) as historical context, not the current catalog. -Nothing here is free text, a session/run/model identifier, a tool or MCP -server name, a file path, a prompt, or an output. Every label value is drawn -from a closed Go-level enum already defined in `internal/adapter/productmetrics/config.go` +Nothing here is free text, a session/run/model identifier, an MCP server/tool +name, a file path, a prompt, or an output. `category` is the one attribute +that emits a name verbatim — but only a built-in tool's own name, drawn from +a maintained closed allowlist in `toolcall.go` (`builtinToolCategories`), with +`"mcp"`/`"other"` as safe catch-alls; every other label value is drawn from a +closed Go-level enum defined in `internal/adapter/productmetrics/config.go` or reused from `engine/session` (`StopReason`). ### Opt-out precedence and the operator-tier-only settings gate diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go index ae4a074c5c..9c96ea3960 100644 --- a/internal/adapter/productmetrics/bounded_test.go +++ b/internal/adapter/productmetrics/bounded_test.go @@ -70,7 +70,10 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T 10*time.Millisecond, 20*time.Millisecond, ) // The run-aware path, with an MCP-namespaced name whose server and remote - // tool halves are both operator-chosen free text. + // tool halves are both operator-chosen free text. EvSessionInit is driven + // FIRST with the SAME RunID so this run is genuinely tracked — exercising + // run_duration and tool_calls_per_run too, not just category/outcome. + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-x"}) r.ToolCallForRun( "run-x", session.SessionID("sensitive-session-id-marker"), @@ -96,8 +99,16 @@ func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T for _, sm := range rm.ScopeMetrics { totalMetrics += len(sm.Metrics) } - if totalMetrics < 10 { - t.Fatalf("collected only %d metrics, want at least 10 (the full mecatl.product.* instrument set) — the walk below would otherwise pass vacuously", totalMetrics) + // wantInstrumentCount is NewRecorder's exact registered instrument count + // (heartbeat, feature_enabled, provider_configured, deployment_mode, + // sessions_started, runs_completed, tool_calls, tokens, subagent_used, + // team_used, run_duration, tool_calls_per_run, time_to_first_value). + // Asserting the EXACT count, not a floor, means a future instrument this + // test's driving code doesn't happen to exercise fails loudly here rather + // than silently passing the walk below vacuously. + const wantInstrumentCount = 13 + if totalMetrics != wantInstrumentCount { + t.Fatalf("collected %d metrics, want exactly %d (the full mecatl.product.* instrument set) — the walk below would otherwise pass vacuously on a new, unexercised instrument", totalMetrics, wantInstrumentCount) } for _, sm := range rm.ScopeMetrics { diff --git a/internal/adapter/productmetrics/dryrun.go b/internal/adapter/productmetrics/dryrun.go index 9acc44dcef..a2a010e47d 100644 --- a/internal/adapter/productmetrics/dryrun.go +++ b/internal/adapter/productmetrics/dryrun.go @@ -62,7 +62,8 @@ func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") d.perRun.markStarted(ev.RunID) case session.EvResult: - d.emitResult(ctx, ev.Result, d.perRun.finish(ev.RunID)) + st, tracked := d.perRun.finish(ev.RunID) + d.emitResult(ctx, ev.Result, st, tracked) case session.EvSubagentStart: if d.perRun.markFamilyUsed(ev.RunID, familySubagent) { d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") @@ -74,7 +75,7 @@ func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { } } -func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload, st perRunState) { +func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload, st perRunState, tracked bool) { stop := session.StopNone if res != nil { stop = res.Stop @@ -83,7 +84,12 @@ func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayl fields := []any{ "stop", string(stop), attrHadToolCall, st.hadToolCall, - "tool_calls_per_run", st.toolCallCount, + } + // tool_calls_per_run mirrors Recorder.recordResult's tracked guard: an + // untracked run (no RunID, or a RunID this recorder never saw an + // EvSessionInit/tool call for) would otherwise report a fabricated 0. + if tracked { + fields = append(fields, "tool_calls_per_run", st.toolCallCount) } // run_duration is only meaningful when this recorder actually observed the // run's EvSessionInit (mirroring Recorder.recordResult's zero-startedAt diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index d947a57ac5..48eac56e95 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -241,20 +241,30 @@ func (t *perRunTracker) markToolCall(runID string, errored bool) { // finish drops runID's live state at EvResult and returns a COPY of what was // there (the zero value if the run recorded nothing — e.g. a run with no tool -// calls and no delegation-family use). Returning a copy, not the pointer, -// keeps every field read outside the lock race-free. -func (t *perRunTracker) finish(runID string) perRunState { +// calls and no delegation-family use), plus tracked reporting whether this +// run ever had an entry in the map at all. tracked is false both for an empty +// runID (no run correlation available — the base ToolCallRecorder.ToolCall +// path, or a caller like RetryFailedStep's RunRequest{} that mints no RunID) +// and for a nonempty runID this Recorder never observed an EvSessionInit or +// tool call for (e.g. a process restart mid-run). Distinguishing "genuinely +// tracked, made zero tool calls" from "never tracked at all" matters: without +// it, tool_calls_per_run would record a FABRICATED 0 for an untracked run +// that may have made many real tool calls this Recorder simply never +// correlated to it — recordResult uses tracked to skip that histogram +// sample entirely rather than report a number that isn't true. Returning a +// copy, not the pointer, keeps every field read outside the lock race-free. +func (t *perRunTracker) finish(runID string) (st perRunState, tracked bool) { if runID == "" { - return perRunState{} + return perRunState{}, false } t.mu.Lock() defer t.mu.Unlock() - st, ok := t.states[runID] + existing, ok := t.states[runID] if !ok { - return perRunState{} + return perRunState{}, false } delete(t.states, runID) - return *st + return *existing, true } // Compile-time interface checks. @@ -346,7 +356,8 @@ func (r *Recorder) Emit(ctx context.Context, ev session.Event) { case session.EvResult: // finish both reads and clears the run's state, so the had_tool_call // resolution and the bounded-map cleanup are one step. - r.recordResult(ctx, ev.Result, r.perRun.finish(ev.RunID)) + st, tracked := r.perRun.finish(ev.RunID) + r.recordResult(ctx, ev.Result, st, tracked) case session.EvSubagentStart: if r.perRun.markFamilyUsed(ev.RunID, familySubagent) { r.subagentUsed.Add(ctx, 1) @@ -367,8 +378,11 @@ const ( ) // recordResult counts the completed run against its bounded stop reason and -// the had_tool_call fact carried by the run's just-finished state. -func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, st perRunState) { +// the had_tool_call fact carried by the run's just-finished state. tracked +// (from perRunTracker.finish) gates tool_calls_per_run: an untracked run +// (empty RunID, or a RunID this Recorder never observed an EvSessionInit or +// tool call for) must not report a fabricated 0 — see finish's doc comment. +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, st perRunState, tracked bool) { stop := session.StopNone if res != nil { stop = res.Stop @@ -376,7 +390,9 @@ func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, r.runsCompleted.Add(ctx, 1, metric.WithAttributes( attribute.String(attrStop, string(stop)), attribute.String(attrHadToolCall, strconv.FormatBool(st.hadToolCall)))) - r.toolCallsPerRun.Record(ctx, st.toolCallCount) + if tracked { + r.toolCallsPerRun.Record(ctx, st.toolCallCount) + } if !st.startedAt.IsZero() { r.runDuration.Record(ctx, time.Since(st.startedAt).Seconds()) } diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go index bbea855f3b..e877ad7d57 100644 --- a/internal/adapter/productmetrics/metrics_test.go +++ b/internal/adapter/productmetrics/metrics_test.go @@ -315,9 +315,15 @@ func TestRecorderToolCallsPerRunRecordedAtResult(t *testing.T) { } func TestRecorderToolCallsPerRunRecordsZeroForAToollessRun(t *testing.T) { - // A run that called no tool still contributes a 0 sample — otherwise the - // distribution silently over-reports by omitting its whole left tail. + // A TRACKED run (its EvSessionInit was observed) that called no tool still + // contributes a 0 sample — otherwise the distribution silently + // over-reports by omitting its whole left tail. This is distinct from an + // UNTRACKED run (no EvSessionInit ever observed, e.g. a retry path that + // mints no RunID, or a process restart mid-run): see + // TestRecorderToolCallsPerRunSkippedForAnUntrackedRun below — recording a + // 0 there would be a FABRICATED sample, not an honest one. r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"] @@ -332,3 +338,24 @@ func TestRecorderToolCallsPerRunRecordsZeroForAToollessRun(t *testing.T) { t.Fatalf("expected one data point with count 1 summing to 0, got %+v", hist.DataPoints) } } + +// TestRecorderToolCallsPerRunSkippedForAnUntrackedRun pins the fix for the +// finding in the final whole-branch review: a run whose EvSessionInit this +// Recorder never observed (e.g. RetryFailedStep's RunRequest{}, which mints +// no RunID, or a process restart mid-run) must not report a 0 on +// tool_calls_per_run — that 0 would be indistinguishable from a genuine +// zero-tool-call run, silently biasing the distribution downward for a run +// that may have made many real tool calls this Recorder simply never +// correlated. An untracked run must record NOTHING on this instrument. +func TestRecorderToolCallsPerRunSkippedForAnUntrackedRun(t *testing.T) { + r, reader := newTestRecorder(t) + // No EvSessionInit for "run-1" — this run is genuinely untracked, even + // though it carries a real, nonempty RunID. + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + if agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"]; ok { + if hist, ok := agg.(metricdata.Histogram[int64]); ok && len(hist.DataPoints) > 0 { + t.Fatalf("tool_calls_per_run recorded %+v for an untracked run, want no data point at all", hist.DataPoints) + } + } +} diff --git a/internal/adapter/productmetrics/toolcall.go b/internal/adapter/productmetrics/toolcall.go index 027ced4b69..89e8ed8f55 100644 --- a/internal/adapter/productmetrics/toolcall.go +++ b/internal/adapter/productmetrics/toolcall.go @@ -107,7 +107,7 @@ func (r *Recorder) ToolCall(id session.SessionID, call session.ToolCall, result // ToolCallForRun satisfies port.RunAwareToolCallRecorder. It records the // bounded category/outcome attributes and tallies the run's per-run state -// (had_tool_call, and the tool-call count a later task publishes). +// (had_tool_call, and the tool-call count published as tool_calls_per_run). // // It reads exactly two things off its arguments: call.Name, only through the // closed-set toolCategory projection, and result.IsError, a boolean. The diff --git a/internal/adapter/productmetrics/toolcall_test.go b/internal/adapter/productmetrics/toolcall_test.go index 8e3cb61126..181a2cb616 100644 --- a/internal/adapter/productmetrics/toolcall_test.go +++ b/internal/adapter/productmetrics/toolcall_test.go @@ -149,7 +149,10 @@ func TestRecorderPerRunTrackerIsRaceFreeUnderConcurrentUse(t *testing.T) { wg.Wait() for _, runID := range live { - st := r.perRun.finish(runID) + st, tracked := r.perRun.finish(runID) + if !tracked { + t.Errorf("%s: finish reported untracked, want tracked (tool calls were made)", runID) + } if st.toolCallCount != 8 { t.Errorf("%s toolCallCount = %d, want 8", runID, st.toolCallCount) } @@ -178,14 +181,17 @@ func TestRecorderToolCallForRunTalliesPerRunCount(t *testing.T) { // A call with no run correlation must not land on any run. r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) - st := r.perRun.finish("run-1") + st, tracked := r.perRun.finish("run-1") + if !tracked { + t.Error("finish(\"run-1\") reported untracked, want tracked") + } if st.toolCallCount != 3 { t.Errorf("toolCallCount = %d, want 3", st.toolCallCount) } if !st.hadToolCall { t.Error("hadToolCall = false, want true (two of the three calls succeeded)") } - if got := r.perRun.finish(""); got != (perRunState{}) { - t.Errorf("finish(\"\") = %+v, want the zero state", got) + if got, gotTracked := r.perRun.finish(""); got != (perRunState{}) || gotTracked { + t.Errorf("finish(\"\") = (%+v, tracked=%v), want the zero state and tracked=false", got, gotTracked) } } diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index ef962ad1d9..d34fc42b64 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -125,7 +125,7 @@ func BuildProductMetrics( go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) - armFirstValueTracking(ctx, recorder, diag) + armFirstValueTracking(ctx, recorder, installIDOverride, diag) return ProductMetricsHandles{ Sink: recorder, @@ -135,7 +135,26 @@ func BuildProductMetrics( }, nil } -// armFirstValueTracking enables mecatl.product.time_to_first_value on recorder. +// armFirstValueTracking enables mecatl.product.time_to_first_value on +// recorder — EXCEPT when installIDOverride is non-empty (mecak8s), where it +// deliberately does nothing. +// +// mecak8s runs storage-free with no PVC (ADR 0048) — the SAME reason its +// install-id comes from a Helm ConfigMap rather than a local file (see +// provider.go's doc comment). The once-ever contract of time_to_first_value +// depends on the SAME kind of durable local marker +// (LoadOrCreateFirstValueMarkerDefault, under $XDG_STATE_HOME) the install-id +// mechanism does, and mecak8s's Helm chart provisions no equivalent for it. +// Arming anyway would make every pod restart/replica rearm with +// alreadyRecorded=false, so a continuously-rolling deployment would emit a +// steady stream of "time to first value" samples that are really +// "time from this pod's start to its first qualifying run" — indistinguishable +// in the backend, under one stable mecatl.install.id, from a stream of +// brand-new installs onboarding continuously. Silently shipping that under a +// "once per install, ever" label would be worse than not shipping the metric +// at all for this one binary; a future durable marker (the ConfigMap, or +// Redis, since mecak8s already depends on it — ADR 0048) can lift this +// restriction later. // // firstSeenAt is time.Now(): this process's start, not the install-id file's // mtime. The approximation is deliberate and sound for the signal's purpose (a @@ -150,7 +169,10 @@ func BuildProductMetrics( // anything: time_to_first_value is a nice-to-have signal, not load-bearing // enough to fail the whole product-metrics pipeline over. The worst case is one // duplicate sample from a later process. -func armFirstValueTracking(ctx context.Context, recorder *productmetrics.Recorder, diag port.Diagnostics) { +func armFirstValueTracking(ctx context.Context, recorder *productmetrics.Recorder, installIDOverride string, diag port.Diagnostics) { + if installIDOverride != "" { + return + } already, err := productmetrics.FirstValueRecordedDefault() if err != nil && diag != nil { diag.Log(ctx, port.LevelDebug, diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go index 87125ac262..4ea5b97be5 100644 --- a/internal/cliconfig/productmetrics_test.go +++ b/internal/cliconfig/productmetrics_test.go @@ -5,7 +5,11 @@ import ( "strings" "testing" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" "github.com/stacklok/mecatl/internal/adapter/productmetrics" ) @@ -134,3 +138,47 @@ func TestBuildProductMetricsInstallIDOverrideNeverReportsFirstRun(t *testing.T) }) } } + +// TestArmFirstValueTrackingSkipsWhenInstallIDIsOverridden pins the fix for the +// finding in the final whole-branch review: mecak8s (which passes a non-empty +// installIDOverride) has no durable local marker for time_to_first_value's +// once-ever contract, the same storage-free problem (ADR 0048) install-id +// solves via a Helm ConfigMap. Arming anyway would make every pod +// restart/replica rearm with alreadyRecorded=false, turning "once per +// install, ever" into "once per pod start" — a silent correctness bug in the +// metric's own contract. armFirstValueTracking must therefore no-op entirely +// when installIDOverride is non-empty: EnableFirstValueTracking must never be +// called, so a subsequent qualifying EvResult records nothing. +func TestArmFirstValueTrackingSkipsWhenInstallIDIsOverridden(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + recorder, err := productmetrics.NewRecorder(mp) + if err != nil { + t.Fatalf("NewRecorder: %v", err) + } + + armFirstValueTracking(context.Background(), recorder, "11111111-2222-3333-[REDACTED]", port.NopDiagnostics{}) + + // Drive a qualifying run: a successful tool call followed by a clean-ended + // result. If tracking were (incorrectly) armed, this would record a + // time_to_first_value sample. + recorder.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + recorder.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name == "mecatl.product.time_to_first_value" { + if hist, ok := m.Data.(metricdata.Histogram[float64]); ok && len(hist.DataPoints) > 0 { + t.Fatalf("time_to_first_value recorded %d data point(s) despite an install-id override — armFirstValueTracking must skip arming entirely for mecak8s", len(hist.DataPoints)) + } + } + } + } +} From 9d34fa6d14912a0b688f58c36dfff9166d0c58ff Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Thu, 10 Sep 2026 20:48:24 -0400 Subject: [PATCH 39/47] fix(docs): regenerate the config reference at its post-reorg canonical location My merge-conflict resolution for docs/configuration-reference.md mistakenly regenerated the OLD full-content target -- ADR 0321's docs reorg had already landed on main, replacing that file with a short redirect stub and repointing the generator (per Taskfile.yml's docs:configref task) at user-docs/reference/configuration.md instead. Restores the stub and regenerates the correct target, which also picks up this branch's telemetry.productMetrics config documentation for the first time. Co-Authored-By: Claude Sonnet 5 --- docs/configuration-reference.md | 337 +-------------------------- user-docs/reference/configuration.md | 11 + 2 files changed, 14 insertions(+), 334 deletions(-) diff --git a/docs/configuration-reference.md b/docs/configuration-reference.md index bf8a7b4d41..940f653f11 100644 --- a/docs/configuration-reference.md +++ b/docs/configuration-reference.md @@ -1,335 +1,4 @@ ---- -title: Configuration reference -description: Look up every operator settings key, type, default, and allowed tier. -sidebar_position: 1 ---- +# Configuration reference moved -{/* GENERATED by 'task docs:configref' from the permconfig YAML schema. Do not hand-edit; run the task. */} -# Configuration reference - -This page is the **exhaustive, auto-generated reference** for the operator -`settings.yaml` surface. It is generated from the permconfig YAML schema -(`internal/adapter/permconfig/schema.go`) — the same structs that strictly -parse the file at startup — so it cannot drift from the code. - -For configuration workflows and examples, see [Configure Mecatl](/building/deployment/settings.md). -To scaffold a complete commented file, run -`mecated config init` (or `mecated config init --print` to print it). -Validate the conventional file with `mecated config validate`, or select -one with `mecated config validate --file PATH`; validation is offline, -read-only, and never prints settings values. - -The settings file lives at `/mecatl/settings.yaml` -(default `~/.config/mecatl/settings.yaml`). The **tier** column records -which configuration tier honours a subtree: `operator`-tier subtrees are -honoured ONLY from this user-global file + CLI (a project `.mecatl/settings.yaml` -copy is ignored with a WARN — honouring it would be a security downgrade); -`operator + project` subtrees may also be set per-project (within the -operator's cap / trust gate). - -The **Default** column describes the `settings.yaml` schema/resolver fallback when -an applicable key is ABSENT (`(empty)` for an unset string, `(absent)` for an unset -map/list/sub-block). It is not a universal process-runtime default: command roots -and modes can supply their own defaults, disable a feature, or reject a setting. -The example values in the `config init` skeleton are ILLUSTRATIVE, not defaults. -For the configuration planes and intentional per-mode differences, see -[Configure Mecatl](/building/deployment/settings.md). - -## `permissions` - -Tier: **operator + project** - -Allow/ask/deny rule-spec lists. Each entry is "Tool(pattern)" or a bare "Tool". Deny is deny-dominant and binds children too; allow/ask bind the main engine; the subagent block binds child engines. A project allow is trust-gated. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `permissions.allow` | `[]string` | `(absent)` | Allow lists rule specs that GRANT a tool call (effect Allow) on the MAIN engine. Under an untrusted project these are DROPPED by the trust gate (see Resolver). | -| `permissions.ask` | `[]string` | `(absent)` | Ask lists rule specs that REQUIRE approval (effect Ask) on the MAIN engine. Always honoured. | -| `permissions.deny` | `[]string` | `(absent)` | Deny lists rule specs that BLOCK a tool call (effect Deny) EVERYWHERE — main engine and subagents (a deny only tightens). Always honoured. | -| `permissions.subagent` | `subagentpermissions` | `(absent)` | Subagent holds the child-scoped rule-spec lists (issue #32): rules that bind ONLY subagent/member/branch engines, resolved through the child-ask model (a subagent allow can clear a substitution-floored ask; a subagent ask surfaces to the human or auto-denies; a subagent deny blocks). | -| `permissions.subagent.allow` | `[]string` | `(absent)` | Allow lists child-scoped rule specs with effect Allow (trust-gated for project tiers). | -| `permissions.subagent.ask` | `[]string` | `(absent)` | Ask lists child-scoped rule specs with effect Ask (always honoured). A configured subagent Ask is NEVER auto-approved by the isolation carve-out — it surfaces to a human or auto-denies. | -| `permissions.subagent.deny` | `[]string` | `(absent)` | Deny lists child-scoped rule specs with effect Deny (always honoured). | - -## `guardrails` - -Tier: **operator** - -OPERATOR-TIER LLM content-checker (issue #27). Parsed strictly. A project-tier guardrails: block is IGNORED with a WARN (a project cannot weaken a security checker). - -> **Enable:** Configuring `model:` ENABLES guardrails; `disabled: true` is the kill-switch (the CLI --guardrails=off also sets it). - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `guardrails.model` | `string` | `(empty)` | Model is the checker model id / alias. Empty leaves the CLI --guardrails-model to supply it; a value here is overridden by the CLI flag when both are set. **Enable:** Setting a model here ENABLES guardrails (the guardrails-parity enable model). A configured model with no rules runs the default BLOCK set (WebSearch/WebFetch/mcp__*/Bash, enforcing; downgrade via defaultMode: advisory). Leave empty (and pass no --guardrails-model) to keep guardrails OFF. | -| `guardrails.minContentBytes` | `int` | `0` | MinContentBytes skips the checker for content shorter than this. 0 = check all. | -| `guardrails.disabled` | `bool` | `false` | Disabled is the YAML-level kill switch (the CLI --guardrails=off also sets it). | -| `guardrails.onCheckerDown` | `string` | `(empty)` | OnCheckerDown sets the global posture when the checker model is unavailable (error/timeout): "warn" (default, fail-open) or "fail" (fail-closed for all rules). Per-rule failClosed overrides: failClosed:true tightens even under warn; failClosed:false (explicit) loosens even under fail. Empty = warn. | -| `guardrails.defaultMode` | `string` | `(empty)` | DefaultMode sets the enforcement mode for the built-in default rules when no explicit rules are configured: "block" (default), "advisory", or "sanitize". An explicit rules list replaces the defaults entirely (this key is ignored). | -| `guardrails.escape` | `bool` | `false` | Escape is the ADR-0080 escape knob: when true AND a checker model is configured, an out-of-root FS escape at posture auto routes through the guardrail checker (an unsafe verdict denies; a checker error fails closed to the write-escape Ask). Default false = the un-routed posture table. | -| `guardrails.rules` | `[]guardrailrulespec` | `(absent)` | Rules is the guardrail rule list. | -| `guardrails.rules[].match` | `string` | `(empty)` | Match is the tool-name matcher (exact / "prefix*" / "*"). | -| `guardrails.rules[].phases` | `[]string` | `(absent)` | Phases lists "pre"/"post"; empty = both. | -| `guardrails.rules[].mode` | `string` | `(empty)` | Mode is "block"/"sanitize"/"advisory"; empty defaults to block. | -| `guardrails.rules[].prompt` | `string` | `(empty)` | Prompt overrides the built-in inspection rubric. | -| `guardrails.rules[].failClosed` | `bool` | `false` | FailClosed flips the fail-open default for enforcing modes. | - -## `posture` - -Tier: **operator** - -OPERATOR-TIER posture-ladder scalar: strict < trusted < auto < yolo (the graduated trust/automation tier). A project-tier posture: is IGNORED with a WARN (a project cannot raise the automation posture). Empty = keep the CLI/default. - -| Value | Type | Default | Description | -| --- | --- | --- | --- | -| `posture` | `string` | `(empty)` | Posture is the OPERATOR-TIER posture-ladder scalar (the graduated trust/ automation tier: strict/trusted/auto/yolo). Like Guardrails it is honoured ONLY from the user-global + CLI tiers; a project-tier file's posture: key is IGNORED with a WARN (a project repo RAISING the automation posture — e.g. posture: yolo — is a security DOWNGRADE the tighten-only project gate forbids, the fail-closed core of this feature). Empty = absent (the resolver returns "" and composition keeps the CLI/default). The composition layer parses the string; permconfig only reads the scalar. | - -## `reasoning-effort` - -Tier: **operator** - -OPERATOR-TIER reasoning-effort scalar (ADR 0055): "" / "auto" (unset — the provider default) / "low" / "medium" / "high" / "xhigh" / "max". OpenAI clamps xhigh/max down to high; Anthropic maps all five. A per-session CreateSession.reasoning_effort out-ranks this default. A project-tier reasoning-effort: is IGNORED with a WARN (a project cannot raise the model's reasoning spend). Empty = keep the CLI/default (provider default). - -| Value | Type | Default | Description | -| --- | --- | --- | --- | -| `reasoning-effort` | `string` | `(empty)` | ReasoningEffort is the OPERATOR-TIER reasoning-effort scalar (ADR 0055: the neutral vocabulary "" / "auto" / "low" / "medium" / "high" / "xhigh" / "max"). Like Posture it is honoured ONLY from the user-global + CLI tiers; a project-tier file's reasoning-effort: key is IGNORED with a WARN (operator-tier only, for consistency — a project cannot raise the model's reasoning spend). Empty = absent (the resolver returns "" and composition uses the provider default). The composition layer interprets + clamps the token; permconfig only reads the scalar. | - -## `plan-mode-auto-approve` - -Tier: **operator** - -OPERATOR-TIER plan-mode auto-approve flag (issue #206): when true, a plan-mode session that parks awaiting a plan-approval ask is auto-approved (flip to default mode and execute) WITHOUT a human reviewing the plan. DEFAULT OFF. A project-tier plan-mode-auto-approve: is IGNORED with a WARN (a project cannot grant an autonomous approval capability). - -| Value | Type | Default | Description | -| --- | --- | --- | --- | -| `plan-mode-auto-approve` | `bool` | `false` | PlanModeAutoApprove is the OPERATOR-TIER plan-mode-auto-approve flag (issue #206 Wave 6a). Like Posture/ReasoningEffort it is honoured ONLY from the user-global + CLI tiers; a project-tier file's plan-mode-auto-approve: key is IGNORED with a WARN (operator-tier only — a project repo enabling autonomous plan approval is a security DOWNGRADE). false = absent (the resolver returns false and composition keeps the default OFF). The composition layer interprets the bool; permconfig only reads the scalar. | - -## `providers` - -Tier: **operator** - -Strict operator-defined LLM providers. Project-tier definitions are ignored. Provider URLs must be HTTPS without userinfo, query, or fragment; credentials belong only in auth.yaml. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `providers.team-gateway` | `providerdefinition` | `(absent)` | | -| `providers.team-gateway.base_url` | `string` | `(required)` | | -| `providers.team-gateway.default_model` | `string` | `(required)` | | -| `providers.team-gateway.api_flavor` | `string` | `(required)` | | -| `providers.team-gateway.auth` | `providerauth` | `(absent)` | | -| `providers.team-gateway.auth.method` | `string` | `none` | | - -## `provider_overrides` - -Tier: **operator** - -Strict endpoint overrides for built-in openai, openrouter, anthropic, and opencode only. Codex and ToolHive policies cannot be overridden here. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `provider_overrides.openai` | `provideroverride` | `(absent)` | | -| `provider_overrides.openai.base_url` | `string` | `(required)` | | - -## `learning` - -Tier: **operator + project** - -Optional completed-trajectory observation policy. Off means no automatic completed-trajectory reflection or review; project settings may only tighten the operator ceiling off < review < auto. Separately configured consolidation schedules are independent. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `learning.mode` | `string` | `off` | Mode controls automatic completed-trajectory observation: off (default; no automatic reflection), review (signal-gated reflection stages durable proposals without memory writes), or auto (stage first, then conservatively promote only eligible non-conflicting facts). Operator settings establish the ceiling; project settings may only tighten it under off < review < auto and never raise autonomy. It does not override separately configured maintenance schedules such as --user-model-consolidate-interval. | -| `learning.sensitivity` | `string` | `balanced` | Sensitivity controls weighted automatic admission. Empty means balanced. | -| `learning.skills` | `learningskillssection` | `(absent)` | Skills controls learned-skill lifecycle policy. | -| `learning.skills.activation` | `string` | `validated when mode is explicitly auto; evaluated otherwise` | Activation is validated (default for Auto) or evaluated. Project settings may only tighten validated to evaluated. | -| `learning.automatic` | `learningautomaticsection` | `(absent)` | Automatic is operator-only admission policy. Standard non-off composition applies it through a durable ledger, making count/token windows, cooldown, and deduplication deployment-wide across cooperating processes. | -| `learning.automatic.cooldown` | `duration` | `10m` | Cooldown is the per-principal weighted-admission cooldown; zero disables it. | -| `learning.automatic.window` | `duration` | `1h` | Window is the sliding count/token window, strictly 1m..24h. | -| `learning.automatic.max_reflections` | `int` | `8` | MaxReflections is the global count cap; zero disables automatic reflection. | -| `learning.automatic.max_tokens` | `int` | `100000` | MaxTokens is the global reserved-token cap; zero disables automatic reflection. | -| `learning.automatic.max_reflections_per_principal` | `int` | `4` | MaxReflectionsPerPrincipal is the per-principal count cap; zero disables automatic reflection. | -| `learning.automatic.max_tokens_per_principal` | `int` | `50000` | MaxTokensPerPrincipal is the per-principal reserved-token cap; zero disables automatic reflection. | - -## `retention` - -Tier: **operator** - -Versioned automatic session cleanup policy. Operator-tier only; project values are ignored. Zero disables each limit. Explicit compatibility flags outrank these values. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `retention.version` | `int` | `1` | Version is the required schema version; the only supported value is 1. | -| `retention.main` | `retentionlimitsection` | `(absent)` | Main controls top-level operator/service sessions. | -| `retention.main.max_age` | `string` | `(empty)` | MaxAge deletes eligible rows older than this Go duration; 0 disables the age limit. | -| `retention.main.max_count` | `int` | `0` | MaxCount keeps the newest eligible rows up to this count; 0 disables the count limit. | -| `retention.child` | `retentionlimitsection` | `(absent)` | Child controls subagent, parallel-branch, and team-member sessions. | -| `retention.child.max_age` | `string` | `(empty)` | MaxAge deletes eligible rows older than this Go duration; 0 disables the age limit. | -| `retention.child.max_count` | `int` | `0` | MaxCount keeps the newest eligible rows up to this count; 0 disables the count limit. | -| `retention.scheduled` | `retentionlimitsection` | `(absent)` | Scheduled controls scheduled-fire sessions. | -| `retention.scheduled.max_age` | `string` | `(empty)` | MaxAge deletes eligible rows older than this Go duration; 0 disables the age limit. | -| `retention.scheduled.max_count` | `int` | `0` | MaxCount keeps the newest eligible rows up to this count; 0 disables the count limit. | -| `retention.sweep_cadence` | `duration` | `1h` | SweepCadence is the repeat interval; 0 disables repeats while retaining the compatibility startup sweep. | -| `retention.acknowledge_main_deletion` | `bool` | `false` | AcknowledgeMainDeletion explicitly consents to destructive main-session cleanup. | - -## `temporary_storage` - -Tier: **operator** - -Managed command temporary-storage policy. Read only from user-global settings.yaml; project and explicit CLI config values are ignored. Managed mode is Linux-only; system preserves inherited temporary-directory behavior. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `temporary_storage.mode` | `string` | `managed` | | -| `temporary_storage.managed_root` | `string` | `mecatl` | | -| `temporary_storage.system_temp_dir` | `string` | `inherited` | | -| `temporary_storage.command_reap_after` | `duration` | `1h` | | -| `temporary_storage.reap_interval` | `duration` | `1h` | | -| `temporary_storage.reap_timeout` | `duration` | `5m` | | -| `temporary_storage.shutdown_reap_timeout` | `duration` | `1m` | | - -## `storage_management` - -Tier: **operator** - -Exact verified OIDC issuer/subject pairs authorized for process-wide storage health, migration, and cleanup. Empty grants nobody; project values are ignored. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `storage_management.version` | `int` | `1` | Version is the required schema version; the only supported value is 1. | -| `storage_management.principals` | `[]storagemanagementprincipal` | `(absent)` | Principals lists exact verified OIDC issuer/subject pairs. Empty grants nobody. | -| `storage_management.principals[].issuer` | `string` | `(empty)` | Issuer must equal the verified token issuer byte-for-byte. | -| `storage_management.principals[].subject` | `string` | `(empty)` | Subject must equal the verified token subject byte-for-byte. | - -## `steer` - -Tier: **operator** - -OPERATOR-TIER mid-run steer knob (steer-while-running, issue #512): when true (the DEFAULT), a client may inject an operator instruction into an in-flight run, drained at the next turn boundary. Set false to disable the steer inbox (the capability echo then reads false and a steer frame reports too_late). A project-tier steer: is IGNORED with a WARN (the harness's operator surface is not a project repo's to flip). Omit = keep the CLI/default (steer ON). - -| Value | Type | Default | Description | -| --- | --- | --- | --- | -| `steer` | `bool` | `true` | Steer is the OPERATOR-TIER mid-run steer knob (steer-while-running, issue #512): enable (default) or disable the mid-run steer inbox. Like Posture/ReasoningEffort it is honoured ONLY from the user-global + CLI tiers; a project-tier file's steer: key is IGNORED with a WARN (operator-tier only — the harness's operator surface is not a project repo's to flip, in either direction). It is a *bool so ABSENT is distinguishable from an explicit false: nil = absent (the resolver reports not-present and composition keeps the DEFAULT-ON); a non-nil value is honoured (composition maps steer: false onto the opt-OUT DisableSteer). | - -## `models` - -Tier: **operator + project** - -Per-slot/alias/default model config (ADR 0030) + the operator allowlist cap and the semantic Subagent model-router taxonomy (ADR 0031/0042). At the operator tier all fields are honoured; a project tier honours slots/aliases/default within the operator allowlist on a trusted workspace (router/allowlist are operator-only). - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `models.slots` | `map[string]string` | `(absent)` | Slots binds a slot name (a call-slot "compaction"/"ask-reviewer"/"guardrail" or a tier "cheap"/"fast"/"reasoning") to a model selector (alias or concrete id). | -| `models.aliases` | `map[string]string` | `(absent)` | Aliases binds a short alias to a concrete model id (merged onto the CLI --model-alias map, CLI winning per key). | -| `models.default` | `string` | `(empty)` | Default is the session-default model selector (alias or concrete id). It is the project-overridable session default (ADR 0030 Phase 4) — within the operator allowlist; the operator's own Default is uncapped. Empty = absent. | -| `models.subagent` | `string` | `(empty)` | Subagent is the OPERATOR-TIER def-less child-default model selector (alias or concrete id): the settings.yaml twin of the --subagent-model flag (issue #288). It sets the global default model for every Subagent / Parallel-branch / team-member child that does not pin its own model (via an agent definition or a per-call override). Operator-tier ONLY: a project-tier subagent: is IGNORED with a WARN (the child-default model is an operator decision — the same operator-only captureModels discipline as default_provider/allowlist/router). The CLI --subagent-model WINS when both are set. Validated FAIL-FAST at Build (normalizeSubagentModel): a value that does not resolve to a usable model id is a startup error (unlike fail-soft models.default). Empty = absent (the flag/inherit-parent behaviour is unchanged). | -| `models.default_provider` | `string` | `(empty)` | DefaultProvider is the OPERATOR-TIER deployment-wide default provider id (e.g. openai, openrouter, anthropic, toolhive). It mirrors the --default-provider flag (app.Config.DefaultProvider) so an operator can declare "toolhive is my default despite my API key" persistently in settings.yaml without unsetting the key. It feeds the UNCHANGED preferredDefaultProvider ladder as an explicit override — it does NOT lower the precedence of key-driven providers. Operator-tier only: a project-tier default_provider: is IGNORED with a WARN (the same operator-only captureModels discipline as posture/guardrails/allowlist). Validated FAIL-FAST at Build (validateDefaultModel): an unknown/unavailable provider is a startup error. Empty = absent (the ladder's preferred default wins). The name pair (default = model, default_provider = provider) mirrors the wire grammar exactly. | -| `models.allowlist` | `[]string` | `(absent)` | Allowlist is the OPERATOR-TIER, non-wideable cap (ADR 0030 Phase 4): the set of model selectors (alias names and/or concrete ids) a PROJECT-tier models: block may bind to. An empty/absent allowlist means project models stay WARN-ignored (the opt-in: no cap ⇒ no project override, byte-identical to pre-Phase-4). It is honoured ONLY from the operator tiers; a project-tier allowlist: key is ignored with a WARN (a project cannot widen its own cap). | -| `models.router` | `routersection` | `(absent)` | Router is the OPERATOR-TIER semantic Subagent model-router taxonomy (ADR 0031, Phase 5; enable model superseded by ADR 0042): a classifier slot, the routing categories, the default category, and the YAML kill-switch. It is operator-tier ONLY — a project-tier router: sub-block is STRIPPED with a WARN (the taxonomy is an autonomous-spend/capability decision the operator owns, like the allowlist). nil/absent = no taxonomy ⇒ the router is OFF (byte-identical, silent). Per ADR 0042 the TAXONOMY is the enable: a non-empty router: with categories turns the router ON unless `disabled: true` (or the CLI kill-switch) forces it off — the guardrails-parity enable model, replacing 0031's flag-to-enable. **Enable:** A non-empty `categories` list ENABLES the router (taxonomy-presence enable, ADR 0042 — NOT a CLI enable-flag); `disabled: true` (or --subagent-model-router=false) is the kill-switch. Operator-tier only. | -| `models.router.classifier-slot` | `string` | `(empty)` | ClassifierSlot names the model slot the CLASSIFIER itself runs on (the tiny, cheap one-turn classification call). Empty falls through to the `router` slot's default tier (cheap) — the classifier is housekeeping, not the routed work. | -| `models.router.categories` | `[]routercategory` | `(absent)` | Categories are the routing choices. Each carries a Name (the classifier's verdict key), a Description (the classifier's only signal — make them distinct), and a Model selector (an alias / slot / concrete id, resolved through the operator- merged alias map; operator taxonomy targets are UNCAPPED). | -| `models.router.categories[].name` | `string` | `(empty)` | Name is the routing key the classifier echoes back as its verdict and the key composition maps to Model. | -| `models.router.categories[].description` | `string` | `(empty)` | Description is the one-line summary the classifier reads to choose this category. | -| `models.router.categories[].model` | `string` | `(empty)` | Model is the model selector (alias / slot / concrete id) a task classified into this category is minted on, resolved through the operator-merged alias map. | -| `models.router.default-category` | `string` | `(empty)` | DefaultCategory is the category the classifier is told to choose when none clearly fits (advisory to the classifier; the real safety net is the fail-soft inherit). | -| `models.router.disabled` | `bool` | `false` | Disabled is the YAML-level kill switch (ADR 0042, mirroring GuardrailsSection.Disabled): per ADR 0042 a non-empty taxonomy ENABLES the router, so `disabled: true` is the "taxonomy defined but temporarily off" override. The CLI kill-switch --subagent-model-router=false also sets it (the two OR together). Default false ⇒ the router is enabled whenever categories are present. | -| `models.context_windows` | `map[string]map[string]int` | `(absent)` | ContextWindows is the OPERATOR-TIER exact provider ID → exact final model ID → total context token override map. It is intentionally not a selector map: aliases and slots are resolved before this lookup, and project values are ignored. | - -## `openrouter` - -Tier: **operator** - -OPERATOR-TIER OpenRouter downstream-provider routing (issue #480): a per-model preferred DOWNSTREAM provider order, sent as OpenRouter's `provider` request-body object. Setting an order disables OpenRouter's default price load-balancing; allow_fallbacks: false pins hard to the order. A project-tier openrouter: block is IGNORED with a WARN (a project cannot pick the downstream provider). - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `openrouter.models` | `map[string]openroutermodelroute` | `(absent)` | Models maps a model id (or alias, resolved in composition) to its downstream-provider routing preference. | -| `openrouter.models..order` | `[]string` | `(absent)` | Order lists downstream provider slugs (lowercase-kebab, e.g. "anthropic", "google-vertex", "deepinfra/turbo") tried in order. Setting it disables OpenRouter's default price load-balancing. Base-slug matching applies: "google-vertex" matches all its regions/variants (service tiers excepted). | -| `openrouter.models..allow_fallbacks` | `bool` | `(absent)` | AllowFallbacks, when explicitly false, pins the request to Order with no fallback to other downstreams. Omit the key to keep OpenRouter's default (true); set it to false to disable fallback. | - -## `telemetry` - -Tier: **operator** - -OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). Honoured ONLY from the user-global + CLI tiers; a project-tier telemetry: block is IGNORED with a WARN (a project repo cannot flip a user's own telemetry choice in either direction). Omit entirely to fall through to the DO_NOT_TRACK env var and finally the enabled-by-default posture. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `telemetry.productMetrics` | `productmetricssection` | `(absent)` | ProductMetrics is the opt-out product/adoption metrics config. | -| `telemetry.productMetrics.enabled` | `bool` | `(absent)` | Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit false: nil = absent (composition falls through to DO_NOT_TRACK then the enabled-by-default posture); a non-nil value is honoured exactly. | - -## `mcp` - -Tier: **operator** - -Strict OPERATOR-TIER Streamable HTTP MCP authority configuration. Mode selects one mutually exclusive global or session-broker authority; broker mode carries its callback configuration and neutral route declarations. Authentication is a closed none/static_bearer/oauth union. Broker OAuth may use trusted explicit OAuth2 endpoints; all secret-shaped values are MECATL_* environment references, never values in YAML. Project mcp blocks are ignored with a value-free warning. - -| Key | Type | Default | Description | -| --- | --- | --- | --- | -| `mcp.mode` | `string` | `(empty)` | Mode selects global or broker authority. Empty uses the command-root default. | -| `mcp.broker` | `mcpbrokerprofile` | `(absent)` | Broker contains options meaningful only in broker mode. | -| `mcp.broker.callback_url` | `string` | `(empty)` | CallbackURL is required exactly when broker mode contains an OAuth route. It must be an absolute HTTPS URL without userinfo, query, or fragment; an omitted path or / is normalized to /. | -| `mcp.servers` | `[]mcpserverprofile` | `(absent)` | Servers is the ordered list of neutral Streamable HTTP route declarations. | -| `mcp.servers[].name` | `string` | `(empty)` | Name is an ASCII [A-Za-z0-9_]+ identifier, unique case-insensitively. | -| `mcp.servers[].url` | `string` | `(empty)` | URL is an absolute HTTP(S) endpoint without userinfo or a fragment. | -| `mcp.servers[].auth` | `mcpauthprofile` | `(absent)` | Auth selects exactly one of none, static_bearer, or oauth. | -| `mcp.servers[].auth.mode` | `string` | `(empty)` | Mode is exactly none, static_bearer, or oauth. | -| `mcp.servers[].auth.static_bearer` | `mcpstaticbearerprofile` | `(absent)` | StaticBearer names the bearer-token environment reference. | -| `mcp.servers[].auth.static_bearer.token_env` | `string` | `(empty)` | TokenEnv is a MECATL_* environment variable name containing the opaque token. | -| `mcp.servers[].auth.oauth` | `mcpoauthprofile` | `(absent)` | OAuth declares the OAuth identity, client, credentials, scopes, and network policy. | -| `mcp.servers[].auth.oauth.profile` | `string` | `(empty)` | Profile is the required global-mode credential identity profile and is forbidden in broker mode. | -| `mcp.servers[].auth.oauth.principal` | `string` | `(empty)` | Principal is the required global-mode credential identity principal and is forbidden in broker mode. | -| `mcp.servers[].auth.oauth.issuer` | `string` | `(empty)` | Issuer is the canonical exact origin used by OIDC discovery. It is forbidden when Upstream explicitly selects generic OAuth2. | -| `mcp.servers[].auth.oauth.upstream` | `mcpoauthupstreamprofile` | `(absent)` | Upstream optionally selects OIDC discovery or explicit generic OAuth2. Omitted defaults to OIDC. | -| `mcp.servers[].auth.oauth.upstream.mode` | `string` | `(empty)` | | -| `mcp.servers[].auth.oauth.upstream.oauth2` | `mcpoauth2upstreamprofile` | `(absent)` | | -| `mcp.servers[].auth.oauth.upstream.oauth2.authorization_endpoint` | `string` | `(empty)` | | -| `mcp.servers[].auth.oauth.upstream.oauth2.token_endpoint` | `string` | `(empty)` | TokenEndpoint is a canonical HTTPS URL with no query string or fragment: the hardened runtime token client pins the exact origin and controls the request query itself. | -| `mcp.servers[].auth.oauth.client` | `mcpoauthclientprofile` | `(absent)` | Client selects exactly one preregistered, CIMD, or DCR client declaration. | -| `mcp.servers[].auth.oauth.client.mode` | `string` | `(empty)` | Mode is exactly preregistered, cimd, or dcr. | -| `mcp.servers[].auth.oauth.client.preregistered` | `mcppreregisteredclientprofile` | `(absent)` | Preregistered declares a confidential client registered with the issuer. | -| `mcp.servers[].auth.oauth.client.preregistered.id` | `string` | `(empty)` | ID is the required preregistered OAuth client identifier. | -| `mcp.servers[].auth.oauth.client.preregistered.secret_env` | `string` | `(empty)` | SecretEnv is a MECATL_* environment variable name containing the client secret. | -| `mcp.servers[].auth.oauth.client.cimd` | `mcpcimdclientprofile` | `(absent)` | CIMD declares an HTTPS client-id metadata document URL. | -| `mcp.servers[].auth.oauth.client.cimd.document_url` | `string` | `(empty)` | DocumentURL is the required HTTPS metadata-document URL. | -| `mcp.servers[].auth.oauth.client.dcr` | `mcpdcrclientprofile` | `(absent)` | DCR declares an RFC 8414 metadata URL for RFC 7591 registration. | -| `mcp.servers[].auth.oauth.client.dcr.discovery_url` | `string` | `(empty)` | DiscoveryURL is the required HTTPS authorization-server metadata URL. | -| `mcp.servers[].auth.oauth.scopes` | `[]string` | `(absent)` | Scopes is the non-empty allowlist of OAuth scopes the client may request. | -| `mcp.servers[].auth.oauth.request_refresh_token` | `bool` | `false` | RequestRefreshToken asks the authorization server for refresh capability. | -| `mcp.servers[].auth.oauth.credentials` | `mcpoauthcredentialprofile` | `(absent)` | Credentials selects one global-mode local or environment credential source and is forbidden in broker mode. | -| `mcp.servers[].auth.oauth.credentials.mode` | `string` | `(empty)` | Mode is exactly local or environment. | -| `mcp.servers[].auth.oauth.credentials.local` | `mcplocalcredentialprofile` | `(absent)` | Local declares encrypted mutable credentials rooted at an absolute path. | -| `mcp.servers[].auth.oauth.credentials.local.root` | `string` | `(empty)` | Root is the required absolute credential-store root. | -| `mcp.servers[].auth.oauth.credentials.local.key_env` | `string` | `(empty)` | KeyEnv is a MECATL_* environment variable name containing the encryption key. | -| `mcp.servers[].auth.oauth.credentials.environment` | `mcpenvironmentcredentialprofile` | `(absent)` | Environment declares one externally provisioned read-only credential record. | -| `mcp.servers[].auth.oauth.credentials.environment.credential_env` | `string` | `(empty)` | CredentialEnv is a MECATL_* environment variable containing the opaque credential record. | -| `mcp.servers[].auth.oauth.credentials.environment.allow_process_local_refresh` | `bool` | `false` | AllowProcessLocalRefresh permits refreshed credentials to live only in this process. | -| `mcp.servers[].auth.oauth.network` | `mcpoauthnetworkprofile` | `(absent)` | Network is required. Global profiles enforce its exact-origin egress policy; broker OAuth accepts only an explicit empty mapping until ToolHive can enforce it equivalently. | -| `mcp.servers[].auth.oauth.network.additional_origins` | `[]string` | `(absent)` | AdditionalOrigins lists canonical exact origins additionally allowed for OAuth traffic. | -| `mcp.servers[].auth.oauth.network.private_origins` | `[]string` | `(absent)` | PrivateOrigins lists allowed origins that may resolve only to RFC1918 IPv4 or ULA IPv6 addresses. Loopback, link-local, metadata, unspecified, multicast, mapped, public, and other special addresses remain denied. | -| `mcp.servers[].auth.oauth.network.max_redirects` | `int` | `0` | MaxRedirects is the redirect bound, from zero through five. | -| `mcp.servers[].auth.oauth.tools` | `[]mcpstatictoolprofile` | `(absent)` | Tools optionally declares this protected backend's tool catalogue statically. Declarations are visible before connection; the first call starts ToolHive's aggregate authorization for every protected backend. The granted bundle unlocks the declared surface only. Omitted, the backend remains discoverable only through pre-prompt workspace enrollment. | -| `mcp.servers[].auth.oauth.tools[].name` | `string` | `(empty)` | | -| `mcp.servers[].auth.oauth.tools[].description` | `string` | `(empty)` | | -| `mcp.servers[].auth.oauth.tools[].input_schema` | `[]uint8` | `(absent)` | | -| `mcp.servers[].auth.oauth.tools[].read_only` | `bool` | `false` | | - -## Flag- / file-configured features (NOT in `settings.yaml`) - -By design, `settings.yaml` covers the subtrees above. Several other -operator features are configured through **CLI flags** (and, for some, their own -files) rather than this YAML. See [Run mecated standalone](/building/deployment/mecated.md) -for the full flag tables. The pointers below are the starting points: - -| Feature | How it is configured | See | -| --- | --- | --- | -| Soul (operator persona) | `--soul-file` / `--no-soul` (+ its own `soul.md` file) | [Skills, commands, and soul](/features/skills-commands-and-soul.md) | -| User-model learning | `--user-model-dir` / `--user-model-review` | [Memory and knowledge](/building/what-you-get/memory.md) | -| Memory | `--memory-dir` / `--memory-store-url` | [Memory and knowledge](/building/what-you-get/memory.md) | -| Slash commands | `--commands-dir` (+ the command `.md` files) | [Skills, commands, and soul](/features/skills-commands-and-soul.md) | -| Session leasing | `--session-lease-*` | [Run mecated standalone](/building/deployment/mecated.md#multi-replica) | - -The model slots / aliases above also have CLI twins (`--model-slot` / -`--model-alias`); the guardrails checker model has `--guardrails-model`; the -router kill-switch has `--subagent-model-router=false`. The CLI flag and the -YAML key are two surfaces for the same setting. See [Configure Mecatl](/building/deployment/settings.md) -for precedence. +See the rendered [configuration reference](https://mecatl.dev/docs/reference/configuration). +See [ADR 0321](./adr/0321-canonical-user-documentation-ownership.md) for the ownership decision. diff --git a/user-docs/reference/configuration.md b/user-docs/reference/configuration.md index 542eb1032e..bf8a7b4d41 100644 --- a/user-docs/reference/configuration.md +++ b/user-docs/reference/configuration.md @@ -246,6 +246,17 @@ OPERATOR-TIER OpenRouter downstream-provider routing (issue #480): a per-model p | `openrouter.models..order` | `[]string` | `(absent)` | Order lists downstream provider slugs (lowercase-kebab, e.g. "anthropic", "google-vertex", "deepinfra/turbo") tried in order. Setting it disables OpenRouter's default price load-balancing. Base-slug matching applies: "google-vertex" matches all its regions/variants (service tiers excepted). | | `openrouter.models..allow_fallbacks` | `bool` | `(absent)` | AllowFallbacks, when explicitly false, pins the request to Order with no fallback to other downstreams. Omit the key to keep OpenRouter's default (true); set it to false to disable fallback. | +## `telemetry` + +Tier: **operator** + +OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). Honoured ONLY from the user-global + CLI tiers; a project-tier telemetry: block is IGNORED with a WARN (a project repo cannot flip a user's own telemetry choice in either direction). Omit entirely to fall through to the DO_NOT_TRACK env var and finally the enabled-by-default posture. + +| Key | Type | Default | Description | +| --- | --- | --- | --- | +| `telemetry.productMetrics` | `productmetricssection` | `(absent)` | ProductMetrics is the opt-out product/adoption metrics config. | +| `telemetry.productMetrics.enabled` | `bool` | `(absent)` | Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit false: nil = absent (composition falls through to DO_NOT_TRACK then the enabled-by-default posture); a non-nil value is honoured exactly. | + ## `mcp` Tier: **operator** From 9a4f3b177779281cf4977c435c8ff2931da0fa6b Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Fri, 11 Sep 2026 09:51:39 -0400 Subject: [PATCH 40/47] fix(docs): renumber ADR 0319 to 0326 after a numbering collision with main origin/main independently used ADR number 0319 for an unrelated topic (release-archives-and-homebrew-tap) since this branch's last merge. Renumbers this branch's ADR to 0326 (the next free number) and fixes its three cross-references (the ADR index, the disclosure notice string, and the user-facing observability doc). The historical plan document under docs/superpowers/plans/ is left referencing the old number, matching this repo's convention that historical planning artifacts are not retroactively updated. Co-Authored-By: Claude Sonnet 5 --- docs/adr/{0319-product-metrics.md => 0326-product-metrics.md} | 2 +- docs/adr/README.md | 2 +- internal/cliconfig/productmetrics.go | 2 +- user-docs/building/what-you-get/observability.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) rename docs/adr/{0319-product-metrics.md => 0326-product-metrics.md} (99%) diff --git a/docs/adr/0319-product-metrics.md b/docs/adr/0326-product-metrics.md similarity index 99% rename from docs/adr/0319-product-metrics.md rename to docs/adr/0326-product-metrics.md index 5dda955c79..9975c6a980 100644 --- a/docs/adr/0319-product-metrics.md +++ b/docs/adr/0326-product-metrics.md @@ -1,4 +1,4 @@ -# ADR 0319 — Product (adoption) metrics over OTLP +# ADR 0326 — Product (adoption) metrics over OTLP - Status: Accepted - Date: 2026-09-09 diff --git a/docs/adr/README.md b/docs/adr/README.md index 3803bc0590..da7892d191 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -204,7 +204,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) -- [0319 — Product (adoption) metrics over OTLP](./0319-product-metrics.md) +- [0326 — Product (adoption) metrics over OTLP](./0326-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index d34fc42b64..317da906c6 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -28,7 +28,7 @@ file path, raw tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass --product-metrics=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: -see docs/adr/0319-product-metrics.md. +see docs/adr/0326-product-metrics.md. ` // ProductMetricsHandles bundles the handles a cmd main threads into its diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index f1cc21ae74..666cb17acc 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -236,7 +236,7 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. -**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0319](https://github.com/stacklok/mecatl/blob/main/docs/adr/0319-product-metrics.md). +**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0326](https://github.com/stacklok/mecatl/blob/main/docs/adr/0326-product-metrics.md). **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: From b335e0a8cac6cc7b85c6dedb7585029501212f40 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Fri, 11 Sep 2026 10:07:27 -0400 Subject: [PATCH 41/47] fix(docs): renumber ADR 0326 to 0327 after another numbering collision with main main claimed 0326 for an unrelated ADR (lazy-toolhive-metadata-refresh) between this branch's last fetch and this push. Renumbers to 0327 (the next free number after fetching main's current tip) and fixes the same three cross-references as the prior renumbering. Co-Authored-By: Claude Sonnet 5 --- docs/adr/{0326-product-metrics.md => 0327-product-metrics.md} | 2 +- docs/adr/README.md | 2 +- internal/cliconfig/productmetrics.go | 2 +- user-docs/building/what-you-get/observability.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) rename docs/adr/{0326-product-metrics.md => 0327-product-metrics.md} (99%) diff --git a/docs/adr/0326-product-metrics.md b/docs/adr/0327-product-metrics.md similarity index 99% rename from docs/adr/0326-product-metrics.md rename to docs/adr/0327-product-metrics.md index 9975c6a980..690bf8eddb 100644 --- a/docs/adr/0326-product-metrics.md +++ b/docs/adr/0327-product-metrics.md @@ -1,4 +1,4 @@ -# ADR 0326 — Product (adoption) metrics over OTLP +# ADR 0327 — Product (adoption) metrics over OTLP - Status: Accepted - Date: 2026-09-09 diff --git a/docs/adr/README.md b/docs/adr/README.md index 8cae8c0c05..c6ddc1f020 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -205,7 +205,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) -- [0326 — Product (adoption) metrics over OTLP](./0326-product-metrics.md) +- [0327 — Product (adoption) metrics over OTLP](./0327-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 317da906c6..5b242f1f7b 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -28,7 +28,7 @@ file path, raw tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass --product-metrics=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: -see docs/adr/0326-product-metrics.md. +see docs/adr/0327-product-metrics.md. ` // ProductMetricsHandles bundles the handles a cmd main threads into its diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 666cb17acc..9190648e56 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -236,7 +236,7 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. -**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0326](https://github.com/stacklok/mecatl/blob/main/docs/adr/0326-product-metrics.md). +**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0327](https://github.com/stacklok/mecatl/blob/main/docs/adr/0327-product-metrics.md). **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: From 43b34dddd803985dc3b8ffa9526255525f00a526 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Fri, 11 Sep 2026 15:25:54 -0400 Subject: [PATCH 42/47] fix(productmetrics): correct the OTLP ingest endpoint hostname The hardcoded endpoint was https://metrics.stacklok.com/v1/metrics, but the actual listener stacklok/infra#5604 provisions is https://mecatl.metrics.stacklok.com/v1/metrics (a mecatl. subdomain). As written, every export would have silently failed against the wrong host regardless of a correctly baked key. Fixes the endpoint var and its three references in the ADR. Co-Authored-By: Claude Sonnet 5 --- docs/adr/0327-product-metrics.md | 6 +++--- internal/adapter/productmetrics/provider.go | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/adr/0327-product-metrics.md b/docs/adr/0327-product-metrics.md index 690bf8eddb..6032212dd0 100644 --- a/docs/adr/0327-product-metrics.md +++ b/docs/adr/0327-product-metrics.md @@ -11,7 +11,7 @@ mecatl had no visibility into community adoption: no install counts, no feature-adoption signal, no aggregate usage depth. Stacklok's infra team stood up a dedicated, internet-facing OTLP/HTTP metrics ingest at -`https://metrics.stacklok.com/v1/metrics` specifically for mecatl binaries +`https://mecatl.metrics.stacklok.com/v1/metrics` specifically for mecatl binaries running on infrastructure Stacklok does not control (`stacklok/infra#5604`): API-key-gated at the edge (`x-mecatl-metrics-key` header, stripped before the collector) and server-side filtered to accept only metric names matching @@ -49,7 +49,7 @@ Zero import relationship with `internal/adapter/telemetry`. It owns: - Its own `metric.MeterProvider` (`Provider`, wrapping `toolhive-core/telemetry/providers.CompositeProvider`), built against a - **hardcoded** endpoint (`https://metrics.stacklok.com/v1/metrics`) and a + **hardcoded** endpoint (`https://mecatl.metrics.stacklok.com/v1/metrics`) and a **hardcoded** header key baked into the binary at build time via `-X …productmetrics.bakedKey=…` (`Taskfile.yml`'s `BUILD_LDFLAGS`) — neither is operator-configurable, and there is exactly one place this data can go. @@ -223,7 +223,7 @@ just a code-review norm that erodes over time. `DryRunRecorder` mirrors Additional structural safeguards: exporter failures are silent to the app (lazy-dial exporter, matching the existing OTLP exporter pattern — a dead -`metrics.stacklok.com` never blocks or slows a session); `Shutdown` is +`mecatl.metrics.stacklok.com` never blocks or slows a session); `Shutdown` is bounded so a hung network path can never delay process exit; the `MeterProvider` is never installed as the process-global provider (mirrors `internal/adapter/telemetry`'s own discipline), so it structurally cannot diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go index 5afb2d4f80..049c95f85f 100644 --- a/internal/adapter/productmetrics/provider.go +++ b/internal/adapter/productmetrics/provider.go @@ -11,14 +11,14 @@ import ( // endpoint and headerKeyName are the ONE destination this pipeline can ever // send to (stacklok/infra#5604): a dedicated, internet-facing OTLP/HTTP -// ingest at metrics.stacklok.com, gated by a single shared key baked into +// ingest at mecatl.metrics.stacklok.com, gated by a single shared key baked into // the binary. Neither is operator-configurable — an operator's own // --otlp-endpoint has zero effect on this path, and this path has zero // effect on the operator's own OTLP/Prometheus pipeline (a completely // separate MeterProvider, never installed as global). endpoint is a var // (not a const) so tests can point it at an httptest server. var ( - endpoint = "https://metrics.stacklok.com/v1/metrics" + endpoint = "https://mecatl.metrics.stacklok.com/v1/metrics" headerKeyName = "x-mecatl-metrics-key" ) From b8edd201ba3c59ab038fd6cf3ab5f77114ddd4f0 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Fri, 11 Sep 2026 15:27:50 -0400 Subject: [PATCH 43/47] fix(docs): renumber ADR 0327 to 0329 after another numbering collision with main main claimed 0327 for an unrelated ADR (self-repository-action-refs) in the window between the last fetch and this push. Renumbers to 0329 (main's own tip was already at 0328) and fixes the same three cross-references. Co-Authored-By: Claude Sonnet 5 --- docs/adr/{0327-product-metrics.md => 0329-product-metrics.md} | 2 +- docs/adr/README.md | 2 +- internal/cliconfig/productmetrics.go | 2 +- user-docs/building/what-you-get/observability.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) rename docs/adr/{0327-product-metrics.md => 0329-product-metrics.md} (99%) diff --git a/docs/adr/0327-product-metrics.md b/docs/adr/0329-product-metrics.md similarity index 99% rename from docs/adr/0327-product-metrics.md rename to docs/adr/0329-product-metrics.md index 6032212dd0..0ccfa18267 100644 --- a/docs/adr/0327-product-metrics.md +++ b/docs/adr/0329-product-metrics.md @@ -1,4 +1,4 @@ -# ADR 0327 — Product (adoption) metrics over OTLP +# ADR 0329 — Product (adoption) metrics over OTLP - Status: Accepted - Date: 2026-09-09 diff --git a/docs/adr/README.md b/docs/adr/README.md index f6480021c1..85c048a21c 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -207,7 +207,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) -- [0327 — Product (adoption) metrics over OTLP](./0327-product-metrics.md) +- [0329 — Product (adoption) metrics over OTLP](./0329-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 5b242f1f7b..310c7ce1a5 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -28,7 +28,7 @@ file path, raw tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass --product-metrics=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: -see docs/adr/0327-product-metrics.md. +see docs/adr/0329-product-metrics.md. ` // ProductMetricsHandles bundles the handles a cmd main threads into its diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 9190648e56..f9c3b9ac99 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -236,7 +236,7 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. -**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0327](https://github.com/stacklok/mecatl/blob/main/docs/adr/0327-product-metrics.md). +**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0329](https://github.com/stacklok/mecatl/blob/main/docs/adr/0329-product-metrics.md). **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: From 9fd64c8ead32ca8083323d7fc2e8b8bb35775458 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Fri, 11 Sep 2026 16:24:48 -0400 Subject: [PATCH 44/47] feat(release): wire the product-metrics ingest key into the real release pipeline Adds an "Extract the product-metrics ingest key" step to all four release jobs that build a real mecated/mecatui/mecak8s artifact (the three ko image jobs plus the GoReleaser archive job): it unwraps the plain key from the MECATL_METRICS_KEY secret's {"mecatl": ""} JSON shape (mirroring the AWS Secrets Manager property stacklok/infra#5604 reads) into MECATL_METRICS_INGEST_KEY, masks it, and exports it via GITHUB_ENV for the build step that follows. .goreleaser.yaml and .ko.yaml now read that env var into a -X .../productmetrics.bakedKey=... ldflag alongside the existing BuildID stamp, in every build block that ships a productmetrics-enabled binary (mecated, mecatui, mecak8s). An absent/empty secret degrades to an empty ldflag (product metrics stay disabled, matching every local/dev/CI build's existing posture) rather than failing the release; a SET-but-malformed secret (no non-empty .mecatl property) is a hard error, since that indicates a real misconfiguration. Verified end-to-end offline: a snapshot goreleaser build and a local ko build --local both actually bake a test key into the resulting binary (confirmed via strings). goreleaser check, task lint:actions (actionlint + embedded shellcheck), and YAML parsing all clean. Co-Authored-By: Claude Sonnet 5 --- .github/workflows/release.yml | 109 ++++++++++++++++++++++++++++++++++ .goreleaser.yaml | 10 ++++ .ko.yaml | 10 ++++ 3 files changed, 129 insertions(+) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 2d5042d9af..0497afc4ea 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -131,6 +131,33 @@ jobs: - name: Login to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | ko login ghcr.io -u "${{ github.actor }}" --password-stdin + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .ko.yaml's ldflags read. GitHub + # already masks the raw secret in logs; ::add-mask:: additionally masks + # the unwrapped value, since deriving it doesn't inherit that automatically. + # An absent/empty secret degrades to a disabled pipeline (a bare `=` + # ldflag) rather than failing the release — only a SET-but-malformed + # secret (present, but no non-empty .mecatl property) is a hard error, + # since that is a real misconfiguration worth surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # Build multi-arch, push by digest, and have ko generate + push an SPDX # SBOM next to the image. --bare keeps the repo path clean (no import-path # suffix), matching `task ko:publish`. We tag both the version and latest. @@ -264,6 +291,33 @@ jobs: - name: Login to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | ko login ghcr.io -u "${{ github.actor }}" --password-stdin + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .ko.yaml's ldflags read. GitHub + # already masks the raw secret in logs; ::add-mask:: additionally masks + # the unwrapped value, since deriving it doesn't inherit that automatically. + # An absent/empty secret degrades to a disabled pipeline (a bare `=` + # ldflag) rather than failing the release — only a SET-but-malformed + # secret (present, but no non-empty .mecatl property) is a hard error, + # since that is a real misconfiguration worth surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # Build multi-arch, push by digest, and have ko generate + push an SPDX # SBOM next to the image. --bare keeps the repo path clean (no import-path # suffix). We tag both the version and latest. The brood-box agent label @@ -397,6 +451,33 @@ jobs: - name: Login to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | ko login ghcr.io -u "${{ github.actor }}" --password-stdin + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .ko.yaml's ldflags read. GitHub + # already masks the raw secret in logs; ::add-mask:: additionally masks + # the unwrapped value, since deriving it doesn't inherit that automatically. + # An absent/empty secret degrades to a disabled pipeline (a bare `=` + # ldflag) rather than failing the release — only a SET-but-malformed + # secret (present, but no non-empty .mecatl property) is a hard error, + # since that is a real misconfiguration worth surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # Build multi-arch, push by digest, and have ko generate + push an SPDX # SBOM next to the image. --bare keeps the repo path clean (no import-path # suffix). We tag both the version and latest. The digest is captured for @@ -851,6 +932,34 @@ jobs: repositories: homebrew-tap permission-contents: write + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .goreleaser.yaml's ldflags read. + # GitHub already masks the raw secret in logs; ::add-mask:: additionally + # masks the unwrapped value, since deriving it doesn't inherit that + # automatically. An absent/empty secret degrades to a disabled pipeline + # (a bare `=` ldflag) rather than failing the release — only a + # SET-but-malformed secret (present, but no non-empty .mecatl property) + # is a hard error, since that is a real misconfiguration worth + # surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # `release --clean` wipes dist/, then builds, archives, SBOMs, checksums, # signs, creates the Release, uploads, and LAST pushes the formula. # diff --git a/.goreleaser.yaml b/.goreleaser.yaml index 3c08a97859..e7024b5041 100644 --- a/.goreleaser.yaml +++ b/.goreleaser.yaml @@ -56,9 +56,18 @@ builds: # `--version` byte-identical across ko, Task and GoReleaser builds. # Not `{{ .Tag }}` either: under --snapshot that reports the LAST tag, which # would lie about what was built. + # + # The product-metrics ingest key (stacklok/infra#5604): {{ .Env.MECATL_METRICS_INGEST_KEY }} + # reads the plain key the release workflow extracts from the MECATL_METRICS_KEY + # secret (a {"mecatl": ""} JSON blob — see .github/workflows/release.yml) + # into that env var BEFORE this step runs. An empty/unset value renders as a + # bare trailing `=`, which is a harmless empty-string ldflag: NewProvider + # refuses to construct on an empty bakedKey, so a run with no key configured + # degrades to the same never-phones-home posture as any local/dev build. ldflags: - -s -w - -X github.com/stacklok/mecatl/internal/buildinfo.BuildID=v{{ .Version }} + - -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{ .Env.MECATL_METRICS_INGEST_KEY }} - id: mecatui main: ./cmd/mecatui @@ -73,6 +82,7 @@ builds: ldflags: - -s -w - -X github.com/stacklok/mecatl/internal/buildinfo.BuildID=v{{ .Version }} + - -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{ .Env.MECATL_METRICS_INGEST_KEY }} # NOTE: no `gomod:` block, deliberately. `gomod.proxy` defaults to false; setting # it true would make GoReleaser fetch github.com/stacklok/mecatl/engine from diff --git a/.ko.yaml b/.ko.yaml index b63ec9e92a..a19aa30cde 100644 --- a/.ko.yaml +++ b/.ko.yaml @@ -36,6 +36,7 @@ builds: ldflags: - -s -w - '{{with index .Env "VERSION"}}-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.}}{{end}}' + - '{{with index .Env "MECATL_METRICS_INGEST_KEY"}}-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.}}{{end}}' # mecak8s (ADR 0048): the storage-free k8s-native agent binary. Same distroless # base + build flags as mecated — it is a thin peer of mecated that composes # app.Build with k8s-native defaults (Redis store + k8s lease + drain gate). @@ -48,11 +49,19 @@ builds: ldflags: - -s -w - '{{with index .Env "VERSION"}}-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.}}{{end}}' + - '{{with index .Env "MECATL_METRICS_INGEST_KEY"}}-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.}}{{end}}' # mecatui (issue #302): the optional Bubble Tea TUI binary. It overrides the # distroless base with the brood-box wolfi base (see baseImageOverrides above — # brood-box connects over SSH and needs a shell) and ships a kodata/agent.yaml # brood-box manifest alongside the binary. VERSION, when nonempty, is retained # verbatim as the explicit linker stamp. + # + # MECATL_METRICS_INGEST_KEY (all three builds above and below): the plain + # product-metrics ingest key the release workflow extracts from the + # MECATL_METRICS_KEY secret's {"mecatl": ""} JSON blob (see + # .github/workflows/release.yml) before invoking `ko build`. Absent/empty + # (any non-release build) omits the ldflag entirely, matching Taskfile.yml's + # local-build posture: NewProvider refuses to construct on an empty bakedKey. - id: mecatui main: ./cmd/mecatui flags: @@ -62,3 +71,4 @@ builds: ldflags: - -s -w - '{{with index .Env "VERSION"}}-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.}}{{end}}' + - '{{with index .Env "MECATL_METRICS_INGEST_KEY"}}-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.}}{{end}}' From f837b8ef2de7cade5492a0bea6d83ef4ac5df423 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Fri, 11 Sep 2026 17:17:53 -0400 Subject: [PATCH 45/47] feat(productmetrics): add MECATL_PRODUCT_METRICS as a mecatl-specific opt-out override Adds MECATL_PRODUCT_METRICS as a second env-var opt-out signal, checked BEFORE the generic DO_NOT_TRACK convention so it can win in either direction: a user can opt mecatl back in despite an ambient DO_NOT_TRACK=1 set for other tools, or opt mecatl out specifically without touching DO_NOT_TRACK. Named to match the existing --product-metrics flag and telemetry.productMetrics.enabled settings key exactly, deliberately not *_DO_NOT_TRACK or *_TELEMETRY -- internal/adapter/telemetry already means something else in this codebase (the unrelated, opt-in operator OTLP/Prometheus pipeline), so a same-flavored name here would misleadingly suggest this variable also touches that pipeline. Accepts any strconv.ParseBool-recognized spelling; unset, empty, or unparseable falls through to the next precedence tier (DO_NOT_TRACK, then the operator settings.yaml, then default-enabled) rather than being treated as a false positive. Updates the five surfaces that document or announce the opt-out mechanisms: the ADR's precedence list, the startup disclosure notice, each of the four binaries' --product-metrics flag help text, and the user-facing observability doc. Co-Authored-By: Claude Sonnet 5 --- cmd/mecak8s/flags.go | 2 +- cmd/mecated/main.go | 2 +- cmd/mecatequi/flags.go | 2 +- cmd/mecatui/config.go | 2 +- docs/adr/0329-product-metrics.md | 32 +++++++--- internal/cliconfig/productmetrics.go | 4 +- internal/cliconfig/productmetrics_config.go | 34 +++++++++-- .../cliconfig/productmetrics_config_test.go | 58 +++++++++++++++---- .../building/what-you-get/observability.md | 1 + 9 files changed, 108 insertions(+), 29 deletions(-) diff --git a/cmd/mecak8s/flags.go b/cmd/mecak8s/flags.go index 6e339bcee2..b4e38982f6 100644 --- a/cmd/mecak8s/flags.go +++ b/cmd/mecak8s/flags.go @@ -487,7 +487,7 @@ func parseFlags(argv []string) (config, error) { fs.StringVar(&cfg.installationID, "telemetry-installation-id", os.Getenv("MECATL_INSTALLATION_ID"), "stable canonical UUID exported as the optional mecatl.installation.id OTel resource attribute (default: MECATL_INSTALLATION_ID; empty omits it)") fs.BoolVar(&cfg.productMetrics, "product-metrics", true, - "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index 96962729bd..4f612dee31 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -1717,7 +1717,7 @@ func parseFlagsModeOut(mode commandMode, argv []string, out io.Writer) (*flag.Fl fs.BoolVar(&cfg.otlpInsecure, "otlp-insecure", false, "skip TLS when dialing the OTLP collector (development only)") fs.BoolVar(&cfg.productMetrics, "product-metrics", true, - "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") diff --git a/cmd/mecatequi/flags.go b/cmd/mecatequi/flags.go index e3656370ca..47c98b7614 100644 --- a/cmd/mecatequi/flags.go +++ b/cmd/mecatequi/flags.go @@ -250,7 +250,7 @@ func parseFlags(argv []string) (flags, error) { fs.DurationVar(&f.otlpShutdownTimeout, "otlp-shutdown-timeout", 5*time.Second, "bound on the telemetry flush at exit (so a dead collector cannot hang the run). 0 disables the bound (flush until it completes); the flush runs BEFORE the diff/summary emit defer unwinds") fs.BoolVar(&f.productMetrics, "product-metrics", true, - "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") fs.BoolVar(&f.productMetricsDryRun, "product-metrics-dry-run", false, "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") diff --git a/cmd/mecatui/config.go b/cmd/mecatui/config.go index 8270e497fe..33afd92731 100644 --- a/cmd/mecatui/config.go +++ b/cmd/mecatui/config.go @@ -475,7 +475,7 @@ func parseTransportFlags(mode transportMode, out io.Writer, args []string, brows fs.StringVar(&cfg.skillsDir, "skills-dir", "", "embedded server only: directory of skill units (/SKILL.md); empty = the conventional dirs (e.g. .claude/skills)") fs.BoolVar(&cfg.noSkills, "no-skills", false, "embedded server only: disable skill discovery (the Skill tool) entirely") fs.BoolVar(&cfg.productMetrics, "product-metrics", true, - "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") diff --git a/docs/adr/0329-product-metrics.md b/docs/adr/0329-product-metrics.md index 0ccfa18267..a609f105ea 100644 --- a/docs/adr/0329-product-metrics.md +++ b/docs/adr/0329-product-metrics.md @@ -181,20 +181,36 @@ or reused from `engine/session` (`StopReason`). ### Opt-out precedence and the operator-tier-only settings gate Product metrics are **enabled by default** (opt-out). `internal/cliconfig.ResolveProductMetricsEnabled` -(`productmetrics_config.go`) folds four inputs, highest precedence first: +(`productmetrics_config.go`) folds five inputs, highest precedence first: 1. An explicit CLI flag: `--product-metrics=false` (all four binaries). -2. The `DO_NOT_TRACK` environment variable set to a truthy value (`""`/`"0"`/ +2. The `MECATL_PRODUCT_METRICS` environment variable, when it parses as a + `strconv.ParseBool` boolean (`1`/`true`/`0`/`false`, case-insensitive, and + their variants) — a mecatl-specific override, checked BEFORE the generic + `DO_NOT_TRACK` convention so it can win in either direction (e.g. opt + mecatl back IN despite an ambient `DO_NOT_TRACK=1` set for other tools, or + opt mecatl OUT specifically without touching `DO_NOT_TRACK`). Named to + match `--product-metrics`/`telemetry.productMetrics.enabled` exactly — + deliberately NOT `*_DO_NOT_TRACK` or `*_TELEMETRY`: this package's own + `internal/adapter/telemetry` already means something else (the unrelated, + opt-in operator OTLP/Prometheus pipeline), so a same-flavored name here + would misleadingly suggest this variable also touches that pipeline. An + unset/empty/unparseable value falls through to the next tier. +3. The `DO_NOT_TRACK` environment variable set to a truthy value (`""`/`"0"`/ `"false"`, case-insensitive, are NOT an opt-out) — the cross-ecosystem convention (donottrack.sh), so the one env var that already opts CI fleets and dev machines out of *other* tools' telemetry covers mecatl - too, with no mecatl-specific variable to remember. (A dedicated - `MECATL_PRODUCT_METRICS=0` was deliberately not added on top of it — one - standard signal beats two overlapping ones.) -3. `telemetry.productMetrics.enabled: false` in the **operator-tier** + too, with no mecatl-specific variable to remember. (Originally the ONLY + env-var signal, on the reasoning that "one standard signal beats two + overlapping ones" — `MECATL_PRODUCT_METRICS` was added afterward for + users who want mecatl-specific control independent of their `DO_NOT_TRACK` + setting; the two are complementary, not redundant, since one is a + cross-tool convention and the other is a same-named override matching + this package's own flag/settings vocabulary.) +4. `telemetry.productMetrics.enabled: false` in the **operator-tier** settings file (`~/.config/mecatl/settings.yaml` + CLI-loaded equivalents) — `permconfig.Resolver.OperatorProductMetricsEnabled()`. -4. Default: enabled. +5. Default: enabled. The settings toggle is **operator-tier only**, the same trust boundary as `guardrails:`/`openrouter:` (AGENTS.md's existing operator-tier-only @@ -255,7 +271,7 @@ one and the balance shifts back toward requiring opt-in: binary is about to actually send product metrics (telemetry enabled, and this install's telemetry-id file did not yet exist), it prints `cliconfig.ProductMetricsDisclosureNotice` once to stderr — what is - collected, that it is on by default, and the exact three ways to turn it + collected, that it is on by default, and the exact four ways to turn it off. It never blocks. An opt-out default with no visible disclosure is the pattern that burns community trust; this is the whole of that disclosure, and it is not optional or hidden in a man page. diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 310c7ce1a5..348e432642 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -26,8 +26,8 @@ major features you have enabled, an anonymous per-install identifier, and coarse session/run/tool-call counts by bounded category — never a prompt, file path, raw tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass ---product-metrics=false, set DO_NOT_TRACK=1, or set -telemetry.productMetrics.enabled: false in your settings.yaml. Details: +--product-metrics=false, set MECATL_PRODUCT_METRICS=false, set DO_NOT_TRACK=1, +or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: see docs/adr/0329-product-metrics.md. ` diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go index 7a02c009cc..f3e2ba68eb 100644 --- a/internal/cliconfig/productmetrics_config.go +++ b/internal/cliconfig/productmetrics_config.go @@ -4,6 +4,7 @@ package cliconfig import ( "os" + "strconv" "strings" "time" @@ -24,17 +25,39 @@ func doNotTrackOptOut(v string) bool { } } +// mecatlProductMetricsOverride parses the MECATL_PRODUCT_METRICS env var: a +// mecatl-specific boolean override, distinct from the generic DO_NOT_TRACK +// convention. Named to match --product-metrics/telemetry.productMetrics.enabled +// exactly (one vocabulary word across all three surfaces) rather than a +// "track"/"telemetry"-flavored name — this codebase's OWN "telemetry" already +// means the unrelated, opt-in operator OTLP/Prometheus pipeline +// (internal/adapter/telemetry), so a same-flavored name here would misleadingly +// suggest it also touches that pipeline; it does not and never should. +// set is false when the var is empty/unset/unparseable, letting the caller +// fall through to the next precedence tier. +func mecatlProductMetricsOverride(v string) (value, set bool) { + if v == "" { + return false, false + } + b, err := strconv.ParseBool(v) + if err != nil { + return false, false + } + return b, true +} + // ProductMetricsPrecedence carries the opt-out inputs // ResolveProductMetricsEnabled folds, highest precedence first: an explicit -// CLI flag, then the DO_NOT_TRACK env var convention (donottrack.sh), -// then the operator settings.yaml value, then default-enabled. +// CLI flag, then the mecatl-specific MECATL_PRODUCT_METRICS env var, then the +// DO_NOT_TRACK env var convention (donottrack.sh), then the operator +// settings.yaml value, then default-enabled. type ProductMetricsPrecedence struct { // FlagSet/FlagValue report whether --product-metrics was explicitly // passed on the command line and its value. FlagSet bool FlagValue bool - // Getenv abstracts os.Getenv for DO_NOT_TRACK / testing. Defaults to - // os.Getenv when nil. + // Getenv abstracts os.Getenv for MECATL_PRODUCT_METRICS / DO_NOT_TRACK / + // testing. Defaults to os.Getenv when nil. Getenv func(string) string // SettingsEnabled is permconfig.Resolver.OperatorProductMetricsEnabled() // — nil when the operator set no telemetry.productMetrics.enabled value. @@ -52,6 +75,9 @@ func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { if getenv == nil { getenv = os.Getenv } + if v, set := mecatlProductMetricsOverride(getenv("MECATL_PRODUCT_METRICS")); set { + return v + } if doNotTrackOptOut(getenv("DO_NOT_TRACK")) { return false } diff --git a/internal/cliconfig/productmetrics_config_test.go b/internal/cliconfig/productmetrics_config_test.go index b0adbbd1ef..dafadca459 100644 --- a/internal/cliconfig/productmetrics_config_test.go +++ b/internal/cliconfig/productmetrics_config_test.go @@ -10,23 +10,29 @@ import ( func boolPtr(b bool) *bool { return &b } -func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { - getenvSet := func(string) string { return "1" } - getenvUnset := func(string) string { return "" } +// envMap builds a Getenv func from a name->value map; an unlisted name +// returns "" (unset), matching os.Getenv's own behavior. +func envMap(m map[string]string) func(string) string { + return func(name string) string { return m[name] } +} +func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { cases := []struct { name string p ProductMetricsPrecedence want bool }{ - {"flag true wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: true, Getenv: getenvSet, SettingsEnabled: boolPtr(false)}, true}, - {"flag false wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: false, Getenv: getenvUnset, SettingsEnabled: boolPtr(true)}, false}, - {"DO_NOT_TRACK disables when no flag", ProductMetricsPrecedence{Getenv: getenvSet, SettingsEnabled: boolPtr(true)}, false}, - {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: boolPtr(false)}, false}, - {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: nil}, true}, - {"DO_NOT_TRACK=0 is not an opt-out", ProductMetricsPrecedence{Getenv: func(string) string { return "0" }, SettingsEnabled: boolPtr(true)}, true}, - {"DO_NOT_TRACK=false is not an opt-out", ProductMetricsPrecedence{Getenv: func(string) string { return "false" }, SettingsEnabled: boolPtr(true)}, true}, - {"DO_NOT_TRACK=true disables", ProductMetricsPrecedence{Getenv: func(string) string { return "true" }, SettingsEnabled: boolPtr(true)}, false}, + {"flag true wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: true, Getenv: envMap(map[string]string{"DO_NOT_TRACK": "1", "MECATL_PRODUCT_METRICS": "0"}), SettingsEnabled: boolPtr(false)}, true}, + {"flag false wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: false, Getenv: envMap(nil), SettingsEnabled: boolPtr(true)}, false}, + {"MECATL_PRODUCT_METRICS wins over DO_NOT_TRACK", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"MECATL_PRODUCT_METRICS": "true", "DO_NOT_TRACK": "1"}), SettingsEnabled: boolPtr(false)}, true}, + {"MECATL_PRODUCT_METRICS=false wins over an unset DO_NOT_TRACK and settings", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"MECATL_PRODUCT_METRICS": "false"}), SettingsEnabled: boolPtr(true)}, false}, + {"MECATL_PRODUCT_METRICS unparseable falls through to DO_NOT_TRACK", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"MECATL_PRODUCT_METRICS": "yes", "DO_NOT_TRACK": "1"}), SettingsEnabled: boolPtr(true)}, false}, + {"DO_NOT_TRACK disables when no flag or MECATL_PRODUCT_METRICS", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "1"}), SettingsEnabled: boolPtr(true)}, false}, + {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: envMap(nil), SettingsEnabled: boolPtr(false)}, false}, + {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: envMap(nil), SettingsEnabled: nil}, true}, + {"DO_NOT_TRACK=0 is not an opt-out", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "0"}), SettingsEnabled: boolPtr(true)}, true}, + {"DO_NOT_TRACK=false is not an opt-out", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "false"}), SettingsEnabled: boolPtr(true)}, true}, + {"DO_NOT_TRACK=true disables", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "true"}), SettingsEnabled: boolPtr(true)}, false}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -37,6 +43,36 @@ func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { } } +// TestMecatlProductMetricsOverride pins the parsing rules for the +// MECATL_PRODUCT_METRICS env var directly: any strconv.ParseBool-recognized +// spelling is honored: unset/empty/unparseable means "not set" (falls through +// to the next precedence tier), never a false positive. +func TestMecatlProductMetricsOverride(t *testing.T) { + cases := []struct { + in string + wantValue bool + wantSet bool + }{ + {"", false, false}, + {"1", true, true}, + {"true", true, true}, + {"TRUE", true, true}, + {"0", false, true}, + {"false", false, true}, + {"FALSE", false, true}, + {"yes", false, false}, + {"no", false, false}, + } + for _, tc := range cases { + t.Run(tc.in, func(t *testing.T) { + value, set := mecatlProductMetricsOverride(tc.in) + if value != tc.wantValue || set != tc.wantSet { + t.Errorf("mecatlProductMetricsOverride(%q) = (%v, %v), want (%v, %v)", tc.in, value, set, tc.wantValue, tc.wantSet) + } + }) + } +} + func TestTeeToolCallRecorderCallsEveryNonNilRecorder(t *testing.T) { var calls []string rec := func(name string) port.ToolCallRecorder { diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index f9c3b9ac99..4f9c2d5539 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -241,6 +241,7 @@ The four channels above are all **operator-facing**: they help you observe your **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: - `--product-metrics=false` on the command line (all four binaries). +- The `MECATL_PRODUCT_METRICS` environment variable set to `false`/`0` — a mecatl-specific override, checked before `DO_NOT_TRACK` below, so it can also opt you back **in** even if you have `DO_NOT_TRACK` set globally for other tools. - The `DO_NOT_TRACK` environment variable set to a truthy value (`"0"`/`"false"` do not opt out) — the same convention other tools already respect. - `telemetry.productMetrics.enabled: false` in your **operator-tier** `~/.config/mecatl/settings.yaml`. This setting is operator-tier only: a project repo's `.mecatl/settings.yaml` cannot change your telemetry choice in either direction. From 76ff27054ae23e34864d9b2cf1de13a8ab295e13 Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Mon, 14 Sep 2026 11:24:56 -0400 Subject: [PATCH 46/47] fix(productmetrics): address PR #1278 review blockers Fixes 5 blocking findings from code review: - Rebuild the OTLP exporter directly over otlpmetrichttp instead of toolhive-core's otlp.NewMetricReader, which doubled the metrics path into /v1/metrics/v1/metrics. - Add an allowlistExporter that strips every resource attribute outside the declared closed set at export time, closing an ambient OTEL_RESOURCE_ATTRIBUTES/hostname leak that survived even after dropping toolhive-core's CompositeProvider (the OTel SDK's own metric.WithResource unconditionally merges resource.Environment()). - Make BuildProductMetrics invoke a notify callback synchronously, before starting the heartbeat goroutine and before returning, so the ADR 0329 disclosure notice can never be skipped by a deferred FirstRun check; gate local install-id file creation behind a new productmetrics.Available() check so a keyless build never mints state a later release build would misread as "not first run". - Populate Guardrails/MCP/Scheduling/Provider/Memory on mecatui's, mecatequi's, and mecak8s's heartbeat snapshots via a new shared cliconfig.ResolveProviderFamily helper, instead of emitting an empty provider_configured{family=""}. - Make the first-value marker creation an atomic O_CREATE|O_EXCL claim and thread its real ownership result back through recordFn/claim(), instead of discarding it and always recording a sample. All five come with regression tests. Co-Authored-By: Claude Sonnet 5 --- cmd/mecak8s/observability.go | 48 +++-- cmd/mecak8s/productmetrics_test.go | 55 ++++++ cmd/mecated/main.go | 34 ++-- cmd/mecated/productmetrics_test.go | 52 +++++ cmd/mecatequi/main.go | 2 +- cmd/mecatequi/observability.go | 38 +++- cmd/mecatequi/productmetrics_test.go | 51 +++++ cmd/mecatequi/telemetry_test.go | 2 +- cmd/mecatui/main.go | 51 +++-- cmd/mecatui/productmetrics_test.go | 37 ++++ internal/adapter/productmetrics/firstvalue.go | 57 ++++-- .../adapter/productmetrics/firstvalue_test.go | 84 +++++++- internal/adapter/productmetrics/metrics.go | 61 ++++-- internal/adapter/productmetrics/provider.go | 185 +++++++++++++++--- .../adapter/productmetrics/provider_test.go | 103 +++++++++- internal/cliconfig/productmetrics.go | 45 ++++- internal/cliconfig/productmetrics_config.go | 28 +++ .../cliconfig/productmetrics_config_test.go | 29 +++ internal/cliconfig/productmetrics_test.go | 133 ++++++++++--- 19 files changed, 929 insertions(+), 166 deletions(-) create mode 100644 cmd/mecak8s/productmetrics_test.go create mode 100644 cmd/mecatequi/productmetrics_test.go create mode 100644 cmd/mecatui/productmetrics_test.go diff --git a/cmd/mecak8s/observability.go b/cmd/mecak8s/observability.go index 00c82b28eb..4f3612c3ac 100644 --- a/cmd/mecak8s/observability.go +++ b/cmd/mecak8s/observability.go @@ -45,14 +45,16 @@ type observability struct { // so it degrades to a no-op Shutdown rather than failing this function's error // return (which stays meaningful for the OTLP half only). // -// The first-run disclosure notice is printed HERE, at setup time — NOT in -// flushTelemetry — because mecak8s is a long-running daemon (unlike -// mecatequi's single-shot process, where setup and flush are seconds apart): -// printing only at shutdown would leave the notice invisible for as long as -// the process runs (potentially days/weeks) and never printed at all on a -// SIGKILL/OOM-kill with no graceful shutdown path. This mirrors -// cmd/mecated/main.go's setupProductMetrics, which prints at setup for the -// same reason. +// The first-run disclosure notice is printed via a notify callback +// BuildProductMetrics itself invokes SYNCHRONOUSLY — before it starts the +// heartbeat goroutine and before it returns — not deferred to a check on the +// returned handles' FirstRun field afterward. mecak8s is a long-running +// daemon (unlike mecatequi's single-shot process, where setup and flush are +// seconds apart): printing only at shutdown (flushTelemetry) would leave the +// notice invisible for as long as the process runs (potentially days/weeks) +// and never printed at all on a SIGKILL/OOM-kill with no graceful shutdown +// path. This mirrors cmd/mecated/main.go's setupProductMetrics, which prints +// at setup for the same reason. func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecak8s", @@ -101,7 +103,12 @@ func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) installIDOverride := os.Getenv("MECATL_PRODUCT_METRICS_INSTALL_ID") pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, cfg.productMetricsDryRun, productmetrics.BinaryMecak8s, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productmetrics.FeatureSnapshot{Mode: productmetrics.ModeK8s}, installIDOverride, diag) + productMetricsSnapshot(cfg), installIDOverride, diag, + // stderr, not diag: mecak8s already writes plain informational lines to + // stderr elsewhere (e.g. boundedClose's timeout line in main.go), and the + // disclosure banner is a one-time, human-facing notice rather than a + // structured operational log line. + func(notice string) { _, _ = fmt.Fprint(os.Stderr, notice) }) if pmErr != nil { // Mirror the existing telemetry-setup-failure posture: a warning, never // a fatal error — product metrics are best-effort and must not block @@ -109,17 +116,26 @@ func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) diag.Log(ctx, port.LevelWarn, "product metrics disabled: setup failed", "err", pmErr) pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} } - if pm.FirstRun { - // stderr, not diag: mecak8s already writes plain informational lines to - // stderr elsewhere (e.g. boundedClose's timeout line in main.go), and the - // disclosure banner is a one-time, human-facing notice rather than a - // structured operational log line. - _, _ = fmt.Fprint(os.Stderr, cliconfig.ProductMetricsDisclosureNotice) - } return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on cfg — never a +// model id/alias, only whether each feature is configured at all. No Memory: +// mecak8s runs storage-free with no PVC (ADR 0048) — the same reason its +// install-id comes from a Helm ConfigMap rather than a local file (see +// buildObservability's installIDOverride handling, above). +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{ + Guardrails: cfg.guardrailsModel != "", + MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, + Scheduling: !cfg.noScheduler, + Provider: cliconfig.ResolveProviderFamily(cfg.useOpenAI, cfg.defaultProvider), + Mode: productmetrics.ModeK8s, + } +} + // flushTelemetry runs the OTLP + product-metrics Shutdown (flush) with a // bounded ctx so a dead collector cannot hang SIGTERM shutdown. Safe on a zero // observability (both Shutdowns are no-ops when telemetry is disabled). A diff --git a/cmd/mecak8s/productmetrics_test.go b/cmd/mecak8s/productmetrics_test.go new file mode 100644 index 0000000000..fe49ad8a2a --- /dev/null +++ b/cmd/mecak8s/productmetrics_test.go @@ -0,0 +1,55 @@ +package main + +import ( + "flag" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins that mecak8s's +// heartbeat surfaces Guardrails/MCP/Scheduling/Provider from its own config +// rather than leaving them at their zero values — an empty Provider would +// heartbeat as the invalid provider_configured{family=""}, outside the +// documented anthropic|openai|openrouter|other enum. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + mcpServers := cliconfig.RegisterMCPServerFlag(flag.NewFlagSet("test", flag.ContinueOnError), "") + if err := mcpServers.Set("example=https://mcp.example.com"); err != nil { + t.Fatalf("mcpServers.Set: %v", err) + } + if err := mcpServers.Finalize(); err != nil { + t.Fatalf("mcpServers.Finalize: %v", err) + } + + snap := productMetricsSnapshot(config{ + guardrailsModel: "claude-haiku", + mcpServers: mcpServers, + noScheduler: false, + defaultProvider: "openrouter/some-model", + }) + if !snap.Guardrails { + t.Error("Guardrails = false, want true (guardrailsModel configured)") + } + if !snap.MCP { + t.Error("MCP = false, want true (an mcp server is configured)") + } + if !snap.Scheduling { + t.Error("Scheduling = false, want true (noScheduler is false)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } + if snap.Mode != productmetrics.ModeK8s { + t.Errorf("Mode = %q, want %q", snap.Mode, productmetrics.ModeK8s) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value config still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(config{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index 4f612dee31..79569a4ecf 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -1104,23 +1104,12 @@ func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { if cfg.headless { mode = productmetrics.ModeHeadless } - provider := productmetrics.ProviderOther - switch { - case cfg.useOpenAI: - provider = productmetrics.ProviderOpenAI - case strings.Contains(strings.ToLower(cfg.defaultProvider), "openrouter"): - provider = productmetrics.ProviderOpenRouter - case strings.Contains(strings.ToLower(cfg.defaultProvider), "openai"): - provider = productmetrics.ProviderOpenAI - case cfg.defaultProvider == "" || strings.Contains(strings.ToLower(cfg.defaultProvider), "anthropic"): - provider = productmetrics.ProviderAnthropic - } return productmetrics.FeatureSnapshot{ Memory: cfg.memoryDir != "", Guardrails: cfg.guardrailsModel != "", MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, Scheduling: !cfg.noScheduler, - Provider: provider, + Provider: cliconfig.ResolveProviderFamily(cfg.useOpenAI, cfg.defaultProvider), Mode: mode, } } @@ -1134,11 +1123,16 @@ func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { // (loadMCPLoginProfiles). settings.yaml is parsed twice at boot (once here, // once inside app.Build); an accepted, negligible boot-time cost. // -// It logs its own build failure and prints the first-run disclosure, so run() -// only threads the resulting handles and the heartbeat-context cancel func -// (both callers must defer unconditionally: the handles' Shutdown is always -// a safe no-op when disabled/errored). The returned error is informational -// only — a caller that just wants the handles can discard it. +// It logs its own build failure. The first-run disclosure notice is printed +// via a notify callback BuildProductMetrics itself invokes SYNCHRONOUSLY, +// before starting the heartbeat goroutine and before returning — never +// deferred to a check on the returned handles' FirstRun field afterward, +// which would leave a window where the pipeline could record/export before +// a human ever saw the notice (ADR 0329). run() only threads the resulting +// handles and the heartbeat-context cancel func (both callers must defer +// unconditionally: the handles' Shutdown is always a safe no-op when +// disabled/errored). The returned error is informational only — a caller +// that just wants the handles can discard it. func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) (cliconfig.ProductMetricsHandles, func(), error) { permResolver := permconfig.NewWithEnv(permconfig.Options{ Conventional: cfg.permissionsConventional, @@ -1154,13 +1148,11 @@ func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productMetricsSnapshot(cfg), "" /* no install-id override: local-file mechanism */, diag) + productMetricsSnapshot(cfg), "" /* no install-id override: local-file mechanism */, diag, + func(notice string) { fmt.Fprint(os.Stderr, notice) }) if err != nil { slog.Warn("product metrics disabled: setup failed", "err", err) } - if pm.FirstRun { - fmt.Fprint(os.Stderr, cliconfig.ProductMetricsDisclosureNotice) - } return pm, cancelHeartbeat, err } diff --git a/cmd/mecated/productmetrics_test.go b/cmd/mecated/productmetrics_test.go index 592039cc89..039e3dfa5b 100644 --- a/cmd/mecated/productmetrics_test.go +++ b/cmd/mecated/productmetrics_test.go @@ -1,9 +1,11 @@ package main import ( + "flag" "testing" "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" ) // TestProductMetricsSnapshotModeReflectsHeadless pins that the deployment @@ -19,3 +21,53 @@ func TestProductMetricsSnapshotModeReflectsHeadless(t *testing.T) { t.Errorf("Mode = %q, want %q (--headless)", got, productmetrics.ModeHeadless) } } + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins the full +// heartbeat contract: every configured feature must surface on the +// FeatureSnapshot, and Provider must resolve to a real member of the +// documented anthropic|openai|openrouter|other enum — never the zero value +// (empty string), which would heartbeat as the invalid +// provider_configured{family=""}. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + mcpServers := cliconfig.RegisterMCPServerFlag(flag.NewFlagSet("test", flag.ContinueOnError), "") + if err := mcpServers.Set("example=https://mcp.example.com"); err != nil { + t.Fatalf("mcpServers.Set: %v", err) + } + if err := mcpServers.Finalize(); err != nil { + t.Fatalf("mcpServers.Finalize: %v", err) + } + + cfg := config{ + memoryDir: "/tmp/memory", + guardrailsModel: "claude-haiku", + mcpServers: mcpServers, + noScheduler: false, + defaultProvider: "openrouter/some-model", + } + snap := productMetricsSnapshot(cfg) + + if !snap.Memory { + t.Error("Memory = false, want true (memoryDir configured)") + } + if !snap.Guardrails { + t.Error("Guardrails = false, want true (guardrailsModel configured)") + } + if !snap.MCP { + t.Error("MCP = false, want true (an mcp server is configured)") + } + if !snap.Scheduling { + t.Error("Scheduling = false, want true (noScheduler is false)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value config still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(config{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/cmd/mecatequi/main.go b/cmd/mecatequi/main.go index 40205504e5..de2d525a87 100644 --- a/cmd/mecatequi/main.go +++ b/cmd/mecatequi/main.go @@ -73,7 +73,7 @@ func realMain(argv []string, stdout, stderr io.Writer) int { // Observability (issue #343, ADR 0098): OPT-IN OTLP push. Built right after // flag parse so the flush-on-exit defer covers EVERY exit path (setup-failure // included). With no --otlp-* flags this is a no-op (byte-identical default). - obs, oerr := buildObservability(context.Background(), f, diag) + obs, oerr := buildObservability(context.Background(), f, diag, stderr) if oerr != nil { _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry: %v\n", oerr) return 2 diff --git a/cmd/mecatequi/observability.go b/cmd/mecatequi/observability.go index 32bd40e93c..572f6ef947 100644 --- a/cmd/mecatequi/observability.go +++ b/cmd/mecatequi/observability.go @@ -42,7 +42,14 @@ type observability struct { // product metrics are best-effort — so it degrades to a no-op Shutdown rather // than failing this function's error return (which stays meaningful for the // OTLP half only). -func buildObservability(ctx context.Context, f flags, diag port.Diagnostics) (observability, error) { +// +// stderr receives the first-run disclosure notice, written SYNCHRONOUSLY by +// BuildProductMetrics itself — before it starts the heartbeat goroutine and +// before it returns — rather than deferred to a check on the returned +// handles' FirstRun field at flush time (flushTelemetry previously printed +// it there, AFTER already calling Shutdown/flushing the provider: exactly +// the ordering ADR 0329 forbids for opt-out collection). +func buildObservability(ctx context.Context, f flags, diag port.Diagnostics, stderr io.Writer) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecatequi", OTLPTraceEndpoint: f.otlpEndpoint, @@ -74,8 +81,9 @@ func buildObservability(ctx context.Context, f flags, diag port.Diagnostics) (ob }) pm, pmErr := cliconfig.BuildProductMetrics(ctx, context.Background(), enabled, f.productMetricsDryRun, productmetrics.BinaryMecatequi, buildinfo.BuildID, 0, /* single fire, short-lived */ - productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}, - "" /* no install-id override: local-file mechanism */, diag) + productMetricsSnapshot(f), + "" /* no install-id override: local-file mechanism */, diag, + func(notice string) { _, _ = fmt.Fprint(stderr, notice) }) if pmErr != nil { // Mirror the existing telemetry-setup-failure posture: a warning, never // a fatal error — product metrics are best-effort and must not block a @@ -87,13 +95,30 @@ func buildObservability(ctx context.Context, f flags, diag port.Diagnostics) (ob return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on f — never a +// model id/alias, only whether each feature is configured at all. mecatequi +// has no memory or scheduler flags (single-shot: no per-project memory +// store, no persistent scheduler to opt out of), so Memory/Scheduling stay +// false; Guardrails/MCP/Provider mirror mecated's productMetricsSnapshot. +func productMetricsSnapshot(f flags) productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{ + Guardrails: f.guardrailsModel != "", + MCP: f.mcpServers != nil && len(f.mcpServers.Servers()) > 0, + Provider: cliconfig.ResolveProviderFamily(f.useOpenAI, f.defaultProvider), + Mode: productmetrics.ModeHeadless, + } +} + // flushTelemetry runs the OTLP + product-metrics Shutdown (flush) with a // bounded ctx so a dead collector cannot hang the run. It is safe to call on a // zero observability (both Shutdowns are no-ops when telemetry is disabled). A // flush failure is logged to stderr and never aborts — telemetry is // best-effort at exit. The product-metrics first-run disclosure notice is -// printed to stderr here too (mecatequi already writes plain informational -// lines to stderr — see emitAuthFileWarning/verdictLine). +// NOT printed here: buildObservability's notify callback already wrote it, +// synchronously, before the pipeline could ever record/export anything — see +// buildObservability's doc comment. Printing it here instead (after +// Shutdown/flush has already run) is exactly the ordering ADR 0329 forbids. func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { ctx := context.Background() if timeout > 0 { @@ -111,9 +136,6 @@ func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) _, _ = fmt.Fprintf(stderr, "mecatequi: product metrics flush: %v\n", err) } } - if obs.productMetrics.FirstRun { - _, _ = fmt.Fprint(stderr, cliconfig.ProductMetricsDisclosureNotice) - } } // productMetricsSink fans obs.Sink (the OTLP sink, nil when telemetry is off) diff --git a/cmd/mecatequi/productmetrics_test.go b/cmd/mecatequi/productmetrics_test.go new file mode 100644 index 0000000000..b12fce45ee --- /dev/null +++ b/cmd/mecatequi/productmetrics_test.go @@ -0,0 +1,51 @@ +package main + +import ( + "flag" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins that mecatequi's +// heartbeat surfaces Guardrails/MCP/Provider from its own flags rather than +// leaving them at their zero values — an empty Provider would heartbeat as +// the invalid provider_configured{family=""}, outside the documented +// anthropic|openai|openrouter|other enum. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + mcpServers := cliconfig.RegisterMCPServerFlag(flag.NewFlagSet("test", flag.ContinueOnError), "") + if err := mcpServers.Set("example=https://mcp.example.com"); err != nil { + t.Fatalf("mcpServers.Set: %v", err) + } + if err := mcpServers.Finalize(); err != nil { + t.Fatalf("mcpServers.Finalize: %v", err) + } + + snap := productMetricsSnapshot(flags{ + guardrailsModel: "claude-haiku", + mcpServers: mcpServers, + defaultProvider: "openrouter/some-model", + }) + if !snap.Guardrails { + t.Error("Guardrails = false, want true (guardrailsModel configured)") + } + if !snap.MCP { + t.Error("MCP = false, want true (an mcp server is configured)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } + if snap.Mode != productmetrics.ModeHeadless { + t.Errorf("Mode = %q, want %q", snap.Mode, productmetrics.ModeHeadless) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value flags still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(flags{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/cmd/mecatequi/telemetry_test.go b/cmd/mecatequi/telemetry_test.go index 35135076aa..b51e23a067 100644 --- a/cmd/mecatequi/telemetry_test.go +++ b/cmd/mecatequi/telemetry_test.go @@ -136,7 +136,7 @@ func TestTelemetryDefaultIsNil(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), f, newDiagnostics()) + obs, err := buildObservability(context.Background(), f, newDiagnostics(), io.Discard) if err != nil { t.Fatalf("buildObservability: %v", err) } diff --git a/cmd/mecatui/main.go b/cmd/mecatui/main.go index ad3a8a8706..2b2999c297 100644 --- a/cmd/mecatui/main.go +++ b/cmd/mecatui/main.go @@ -954,11 +954,6 @@ func resolveTransport(ctx context.Context, cfg config) (target string, dial clie _ = diagCloser.Close() return target, client.DialConfig{}, noop, fmt.Errorf("start embedded server: %w", err) } - if pm.FirstRun { - // Through diag, never stderr: stderr would corrupt the Bubble Tea - // alt-screen once the TUI program starts. - diag.Log(ctx, port.LevelInfo, cliconfig.ProductMetricsDisclosureNotice) - } if toFile { // One line, written to the FILE sink (never the TUI), so an operator can find // where the embedded server's diagnostics went. @@ -994,12 +989,21 @@ func resolveTransport(ctx context.Context, cfg config) (target string, dial clie } // productMetricsSnapshot derives the closed-set FeatureSnapshot the product- -// metrics heartbeat reports for the embedded server. mecatui's feature-flag -// detection is out of scope for this task (the same simplification mecated's -// Task 11 made): only Mode is populated here; Memory/Guardrails/MCP/Scheduling -// stay false and Provider stays the zero value. -func productMetricsSnapshot() productmetrics.FeatureSnapshot { - return productmetrics.FeatureSnapshot{Mode: productmetrics.ModeInteractive} +// metrics heartbeat reports for the embedded server, from fields already +// resolved on cfg — never a model id/alias, only whether each feature is +// configured at all. mecatui has no dedicated flags for guardrails/MCP/the +// scheduler on its embedded-server config (those are resolved deeper inside +// app.Build from settings.yaml, not surfaced back to main.go), so +// Guardrails/MCP/Scheduling stay false — but Memory and Provider ARE +// available on cfg and must not be left at their zero values (an empty +// Provider heartbeats as the invalid provider_configured{family=""}, +// outside the documented anthropic|openai|openrouter|other enum). +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{ + Memory: cfg.memoryDir != "", + Provider: cliconfig.ResolveProviderFamily(false, cfg.defaultProvider), + Mode: productmetrics.ModeInteractive, + } } // setupProductMetrics resolves the opt-out product-metrics precedence and builds @@ -1014,12 +1018,22 @@ func productMetricsSnapshot() productmetrics.FeatureSnapshot { // negligible boot-time cost. // // It logs its own build failure via diag (NEVER stderr — stderr would corrupt -// the Bubble Tea alt-screen) and does NOT print the disclosure notice itself; -// the caller prints cliconfig.ProductMetricsDisclosureNotice through diag.Log -// when the returned handles' FirstRun is true, after the embedded server has -// started successfully. The returned cancel func must be called/deferred -// unconditionally by the caller (Shutdown on the handles is always a safe -// no-op when disabled/errored). +// the Bubble Tea alt-screen once the TUI program starts). The first-run +// disclosure notice IS written to stderr — through a notify callback +// BuildProductMetrics itself invokes SYNCHRONOUSLY, before starting the +// heartbeat goroutine and before returning — because at the point this runs +// (resolveTransport, well before tea.NewProgram(...).Run() ever enters the +// alt-screen) stderr is still plain, unbuffered terminal output; a +// diag.Log-routed notice would instead land only in the diagnostics FILE +// (invisible, and dropped entirely under --quiet), defeating ADR 0329's +// visible-disclosure requirement. This also runs BEFORE embed.Start, not +// deferred to a check on the returned handles' FirstRun field after the +// embedded server has started (which left a window where a failed +// embed.Start could flush an already-recording pipeline via +// shutdownProductMetrics without the notice ever having been shown). The +// returned cancel func must be called/deferred unconditionally by the +// caller (Shutdown on the handles is always a safe no-op when +// disabled/errored). func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) (cliconfig.ProductMetricsHandles, func()) { permResolver := permconfig.NewWithEnv(permconfig.Options{ Conventional: true, @@ -1034,7 +1048,8 @@ func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, productmetrics.BinaryMecatui, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, - productMetricsSnapshot(), "" /* no install-id override: local-file mechanism */, diag) + productMetricsSnapshot(cfg), "" /* no install-id override: local-file mechanism */, diag, + func(notice string) { fmt.Fprint(os.Stderr, notice) }) if err != nil { diag.Log(ctx, port.LevelWarn, "mecatui: product metrics disabled: setup failed", "err", err.Error()) } diff --git a/cmd/mecatui/productmetrics_test.go b/cmd/mecatui/productmetrics_test.go new file mode 100644 index 0000000000..9365f2f8df --- /dev/null +++ b/cmd/mecatui/productmetrics_test.go @@ -0,0 +1,37 @@ +package main + +import ( + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins that mecatui's +// embedded-server heartbeat surfaces the fields it CAN see on cfg (Memory, +// Provider) rather than leaving them at their zero values — an empty +// Provider would heartbeat as the invalid provider_configured{family=""}, +// outside the documented anthropic|openai|openrouter|other enum. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + snap := productMetricsSnapshot(config{ + memoryDir: "/tmp/memory", + defaultProvider: "openrouter/some-model", + }) + if !snap.Memory { + t.Error("Memory = false, want true (memoryDir configured)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } + if snap.Mode != productmetrics.ModeInteractive { + t.Errorf("Mode = %q, want %q", snap.Mode, productmetrics.ModeInteractive) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value config still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(config{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/internal/adapter/productmetrics/firstvalue.go b/internal/adapter/productmetrics/firstvalue.go index fb71ff3f61..2eabef07d6 100644 --- a/internal/adapter/productmetrics/firstvalue.go +++ b/internal/adapter/productmetrics/firstvalue.go @@ -63,39 +63,68 @@ func FirstValueRecordedDefault() (bool, error) { // the install-id file) resets the install and lets time_to_first_value fire // once more. // -// readFile/writeFile/mkdirAll are injected for testing; +// The create is an ATOMIC cross-process claim, not a read-then-write check: +// createExclusive must fail when the marker already exists (os.IsExist), +// e.g. via O_CREATE|O_EXCL. Two processes racing this call therefore never +// both observe already==false — exactly one create wins, and the loser +// reliably reports already==true, even when the two calls are strictly +// sequential rather than concurrent (the marker created by an earlier +// process's call is still there when a later process's call runs). +// firstValueTracker.claim() depends on this: it treats a false "won" from +// its recordFn as "another process already has this install's one sample" +// and skips recording, so a non-atomic check-then-act here would silently +// let two processes each record a sample. +// +// mkdirAll/createExclusive are injected for testing; // LoadOrCreateFirstValueMarkerDefault binds the real filesystem. func LoadOrCreateFirstValueMarker( env xdgconfig.ResolveEnv, - readFile func(string) ([]byte, error), - writeFile func(string, []byte, os.FileMode) error, mkdirAll func(string, os.FileMode) error, + createExclusive func(string, []byte, os.FileMode) error, ) (already bool, err error) { path, err := firstValueMarkerPath(env) if err != nil { return false, err } - if readFile != nil { - if _, rerr := readFile(path); rerr == nil { - return true, nil - } - } if mkdirAll != nil { if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { return false, fmt.Errorf("productmetrics: create state dir: %w", merr) } } - if writeFile != nil { - if werr := writeFile(path, []byte("1"), 0o600); werr != nil { - return false, fmt.Errorf("productmetrics: write first-value marker: %w", werr) - } + if createExclusive == nil { + return false, nil + } + switch cerr := createExclusive(path, []byte("1"), 0o600); { + case cerr == nil: + return false, nil + case os.IsExist(cerr): + return true, nil + default: + return false, fmt.Errorf("productmetrics: write first-value marker: %w", cerr) + } +} + +// createFileExclusive creates path only if it does not already exist, +// returning an os.IsExist-satisfying error otherwise (O_CREATE|O_EXCL) — the +// atomic cross-process claim LoadOrCreateFirstValueMarker's contract +// depends on; a plain os.WriteFile (create-or-truncate) would let two +// concurrent callers both "win". +func createFileExclusive(path string, data []byte, perm os.FileMode) error { + f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm) + if err != nil { + return err + } + _, werr := f.Write(data) + cerr := f.Close() + if werr != nil { + return werr } - return false, nil + return cerr } // LoadOrCreateFirstValueMarkerDefault binds LoadOrCreateFirstValueMarker to the // real process environment and filesystem. func LoadOrCreateFirstValueMarkerDefault() (already bool, err error) { - return LoadOrCreateFirstValueMarker(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) + return LoadOrCreateFirstValueMarker(xdgconfig.OSEnv, os.MkdirAll, createFileExclusive) } diff --git a/internal/adapter/productmetrics/firstvalue_test.go b/internal/adapter/productmetrics/firstvalue_test.go index 22acbf630d..785e3b2c7e 100644 --- a/internal/adapter/productmetrics/firstvalue_test.go +++ b/internal/adapter/productmetrics/firstvalue_test.go @@ -34,7 +34,15 @@ func (f *fakeMarkerFS) readFile(p string) ([]byte, error) { return nil, os.ErrNotExist } -func (f *fakeMarkerFS) writeFile(p string, d []byte, _ os.FileMode) error { +// createExclusive is the fake's atomic-claim primitive, mirroring the real +// createFileExclusive's O_CREATE|O_EXCL contract: it fails with an +// os.IsExist-satisfying error when the path already exists, so tests can +// exercise the SAME "second caller loses" semantics the real filesystem +// enforces via one flag. +func (f *fakeMarkerFS) createExclusive(p string, d []byte, _ os.FileMode) error { + if _, ok := f.written[p]; ok { + return os.ErrExist + } f.written[p] = d return nil } @@ -44,7 +52,7 @@ func (*fakeMarkerFS) mkdirAll(string, os.FileMode) error { return nil } func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T) { env, fs := testResolveEnv(), newFakeMarkerFS() - already, err := LoadOrCreateFirstValueMarker(env, fs.readFile, fs.writeFile, fs.mkdirAll) + already, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive) if err != nil { t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) } @@ -52,7 +60,7 @@ func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T t.Error("already = true on first call, want false") } - already2, err := LoadOrCreateFirstValueMarker(env, fs.readFile, fs.writeFile, fs.mkdirAll) + already2, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive) if err != nil { t.Fatalf("second LoadOrCreateFirstValueMarker: %v", err) } @@ -61,12 +69,38 @@ func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T } } +// TestLoadOrCreateFirstValueMarkerClaimIsAtomicAcrossCallers pins the fix for +// the discarded-ownership-result finding: TWO callers racing (or even just +// calling sequentially before either has observed the other) the SAME +// createExclusive-backed marker must have EXACTLY ONE winner (already == +// false) and every other caller must observe already == true — never both +// reporting false, which would let two processes each record the "at most +// once per install, ever" sample. +func TestLoadOrCreateFirstValueMarkerClaimIsAtomicAcrossCallers(t *testing.T) { + env, fs := testResolveEnv(), newFakeMarkerFS() + + var wins int + const callers = 8 + for i := 0; i < callers; i++ { + already, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive) + if err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker (call %d): %v", i, err) + } + if !already { + wins++ + } + } + if wins != 1 { + t.Errorf("wins = %d across %d calls, want exactly 1", wins, callers) + } +} + func TestLoadOrCreateFirstValueMarkerFailsClosedWithNoStateDir(t *testing.T) { env := xdgconfig.ResolveEnv{ Getenv: func(string) string { return "" }, UserHomeDir: func() (string, error) { return "", errors.New("no home") }, } - if _, err := LoadOrCreateFirstValueMarker(env, nil, nil, nil); err == nil { + if _, err := LoadOrCreateFirstValueMarker(env, nil, nil); err == nil { t.Fatal("expected an error when no state dir can be resolved, got nil") } if _, err := FirstValueRecorded(env, nil); err == nil { @@ -91,7 +125,7 @@ func TestFirstValueRecordedNeverCreatesTheMarker(t *testing.T) { t.Fatalf("FirstValueRecorded wrote %v, want no writes", fs.written) } - if _, err := LoadOrCreateFirstValueMarker(env, fs.readFile, fs.writeFile, fs.mkdirAll); err != nil { + if _, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive); err != nil { t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) } already, err = FirstValueRecorded(env, fs.readFile) @@ -129,7 +163,7 @@ func qualifyingRun(t *testing.T, r *Recorder, runID string) { func TestRecorderRecordsTimeToFirstValueOnceOnly(t *testing.T) { r, reader := newTestRecorder(t) var marks int - r.EnableFirstValueTracking(time.Now().Add(-90*time.Second), false, func() error { marks++; return nil }) + r.EnableFirstValueTracking(time.Now().Add(-90*time.Second), false, func() (bool, error) { marks++; return true, nil }) qualifyingRun(t, r, "run-1") qualifyingRun(t, r, "run-2") @@ -147,10 +181,46 @@ func TestRecorderRecordsTimeToFirstValueOnceOnly(t *testing.T) { } } +// TestRecorderSkipsTimeToFirstValueWhenRecordFnLosesTheClaim pins the fix for +// the discarded-ownership-result finding at the Recorder level: even though +// THIS process's in-memory tracker had not yet observed the metric as +// recorded (alreadyRecorded=false at arm time — the cross-process race +// window), recordFn's own atomic claim can still report won=false (another +// process's LoadOrCreateFirstValueMarker call got there first). The +// Recorder must honor that and emit NOTHING, not record a duplicate sample +// just because ITS in-memory state hadn't caught up. +func TestRecorderSkipsTimeToFirstValueWhenRecordFnLosesTheClaim(t *testing.T) { + r, reader := newTestRecorder(t) + var calls int + r.EnableFirstValueTracking(time.Now().Add(-time.Second), false, func() (bool, error) { + calls++ + return false, nil // another process already won the cross-process claim. + }) + + qualifyingRun(t, r, "run-1") + + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded despite recordFn reporting won=false: %+v", agg) + } + if calls != 1 { + t.Errorf("recordFn called %d times, want exactly 1", calls) + } + + // A second qualifying run must not retry the claim: this process already + // made its one attempt. + qualifyingRun(t, r, "run-2") + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded on a second run after losing the claim: %+v", agg) + } + if calls != 1 { + t.Errorf("recordFn called %d times after a second run, want still exactly 1 (no retry)", calls) + } +} + func TestRecorderSkipsTimeToFirstValueWhenAlreadyRecorded(t *testing.T) { r, reader := newTestRecorder(t) var marks int - r.EnableFirstValueTracking(time.Now().Add(-time.Second), true, func() error { marks++; return nil }) + r.EnableFirstValueTracking(time.Now().Add(-time.Second), true, func() (bool, error) { marks++; return true, nil }) qualifyingRun(t, r, "run-1") diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go index 48eac56e95..b4f8ca97eb 100644 --- a/internal/adapter/productmetrics/metrics.go +++ b/internal/adapter/productmetrics/metrics.go @@ -81,25 +81,39 @@ type firstValueTracker struct { // firstSeenAt is this install's first-seen moment; the recorded duration is // measured from it. firstSeenAt time.Time - // done is true once the sample exists — either recorded by THIS process, or - // (per the persisted marker) by an earlier one. + // done is true once THIS process has attempted its one claim — either + // because the sample was already recorded (per the persisted marker) by + // an earlier process, or because this process itself just attempted the + // cross-process claim below. It does NOT by itself mean the sample was + // actually emitted — see claim()'s use of recordFn's won return. done bool - // recordFn persists the marker so a LATER process invocation also stays - // disabled. Called at most once, best-effort. - recordFn func() error + // recordFn persists the marker as an ATOMIC cross-process claim and + // reports whether THIS call actually won it (won=true: this call + // created the marker first) versus lost it (won=false: another process + // already owns it — do not record, even though this process's own + // in-memory tracker had not yet observed that). Called at most once. + // Its error is best-effort: a failed write risks re-recording once on a + // later process (a fidelity wobble in a coarse onboarding signal, not a + // correctness bug worth failing anything over), so claim() still treats + // an ERRORING call as a win rather than silently dropping the sample. + recordFn func() (won bool, err error) } // EnableFirstValueTracking arms mecatl.product.time_to_first_value recording. // firstSeenAt is this install's first-seen timestamp; alreadyRecorded, when // true, permanently disables recording for this Recorder's lifetime (this -// install already has its one sample). recordFn persists the local marker so a -// later process invocation also stays disabled; it is called at most once, and -// may be nil (in-memory-only tracking). +// install already has its one sample). recordFn persists the local marker as +// an ATOMIC cross-process claim so a later process invocation also stays +// disabled; it is called at most once, and may be nil (in-memory-only +// tracking). Its won return MUST reflect whether THIS call actually created +// the marker — discarding it (always treating the call as a win) breaks the +// "at most once per install, ever" contract across two processes that both +// pass the alreadyRecorded=false startup check before either has recorded. // // It is a separate arming step rather than a NewRecorder parameter so that // NewRecorder's signature — and every existing caller and test of it — stays // unchanged; an unarmed Recorder simply never records this instrument. -func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecorded bool, recordFn func() error) { +func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecorded bool, recordFn func() (bool, error)) { r.firstValue.mu.Lock() defer r.firstValue.mu.Unlock() r.firstValue.armed = true @@ -108,9 +122,12 @@ func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecord r.firstValue.recordFn = recordFn } -// claim reports whether THIS observation is the install's first-value moment, marking it claimed and persisting the marker as one atomic step. It -// returns the firstSeenAt to measure from; a false claim means the metric must -// not be recorded (unarmed, already recorded, or no usable firstSeenAt). +// claim reports whether THIS observation is the install's first-value +// moment, persisting the marker as one atomic cross-process step and +// honoring its outcome. It returns the firstSeenAt to measure from; a false +// claim means the metric must not be recorded (unarmed, already recorded, +// no usable firstSeenAt, OR — the cross-process case — this process's own +// atomic marker-claim call reports that another process already won it). func (t *firstValueTracker) claim() (time.Time, bool) { t.mu.Lock() defer t.mu.Unlock() @@ -119,10 +136,22 @@ func (t *firstValueTracker) claim() (time.Time, bool) { } t.done = true if t.recordFn != nil { - // Best-effort: a failed write risks re-recording once on a later - // process, which is a fidelity wobble in a coarse onboarding signal — - // not a correctness bug worth failing anything over. - _ = t.recordFn() + won, err := t.recordFn() + if err != nil { + // Best-effort: a failed write risks re-recording once on a later + // process, which is a fidelity wobble in a coarse onboarding + // signal — not a correctness bug worth failing anything over. + // Fall through and treat this call as the winner, matching the + // pre-existing behavior on a write failure. + return t.firstSeenAt, true + } + if !won { + // Another process's marker-claim call already won this + // install's one-ever sample — honor that result instead of + // discarding it, or two processes that both started before + // either recorded would each independently emit a sample. + return time.Time{}, false + } } return t.firstSeenAt, true } diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go index 049c95f85f..97e788f23d 100644 --- a/internal/adapter/productmetrics/provider.go +++ b/internal/adapter/productmetrics/provider.go @@ -3,10 +3,14 @@ package productmetrics import ( "context" "fmt" - "strings" - "github.com/stacklok/toolhive-core/telemetry/providers" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.opentelemetry.io/otel/sdk/resource" + semconv "go.opentelemetry.io/otel/semconv/v1.41.0" ) // endpoint and headerKeyName are the ONE destination this pipeline can ever @@ -17,6 +21,21 @@ import ( // effect on the operator's own OTLP/Prometheus pipeline (a completely // separate MeterProvider, never installed as global). endpoint is a var // (not a const) so tests can point it at an httptest server. +// +// endpoint is passed VERBATIM to otlpmetrichttp.WithEndpointURL, which sets +// the exporter's URL path to exactly endpoint's own path — an empty path is +// treated as the literal root "/", NOT as "use the exporter's documented +// /v1/metrics default" (see WithEndpointURL's own doc comment in +// go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp's +// internal/oconf package). endpoint therefore MUST spell out "/v1/metrics" +// itself. This is deliberately NOT toolhive-core's otlp.NewMetricReader, +// whose createMetricExporter instead SPLITS a supplied endpoint into +// host+basePath and, when a basePath is present, APPENDS its own +// "/v1/metrics" suffix onto it — so a base URL that already ends in +// "/v1/metrics" would there produce "/v1/metrics/v1/metrics" against the +// real collector. Building the exporter directly over otlpmetrichttp (as +// here) sidesteps that concatenation entirely: the path below is sent +// exactly as written, once. var ( endpoint = "https://mecatl.metrics.stacklok.com/v1/metrics" headerKeyName = "x-mecatl-metrics-key" @@ -30,12 +49,112 @@ var ( // accidentally phone home with an invalid or absent key. var bakedKey = "" -// Provider wraps the toolhive-core OTLP metrics provider. Its MeterProvider -// is NEVER installed as the process-global provider (mirrors -// internal/adapter/telemetry's own discipline in otlp.go), so it cannot -// collide with an operator's own OTel setup. +// Available reports whether this build has an ingest key baked in — i.e. +// whether NewProvider can ever construct a real pipeline. Composition MUST +// check this BEFORE minting/persisting any local install-id state (issue: +// disclosure/first-run ordering): a keyless build (every local/dev/CI-test +// build) that instead created the install-id file first, then failed here, +// would leave that file behind — so a LATER release build's genuine first +// export would read it back and report firstRun=false, silently skipping +// the disclosure notice ADR 0329 requires before that first export. +func Available() bool { return bakedKey != "" } + +// SetBakedKeyForTest overrides bakedKey for the duration of a test and +// returns a restore func the caller must defer. bakedKey is normally set +// exactly once, at build time, via the `-X …productmetrics.bakedKey=…` +// ldflag (see BUILD_LDFLAGS in Taskfile.yml); every other package's tests +// need a way to exercise the Available()==true path (e.g. cliconfig's +// disclosure-ordering tests) without actually shipping a real key, hence +// this seam — mirroring the ForTest helpers elsewhere in internal/adapter +// (e.g. scheduler.RunOnceForTest). +func SetBakedKeyForTest(key string) (restore func()) { + orig := bakedKey + bakedKey = key + return func() { bakedKey = orig } +} + +// SetEndpointForTest overrides endpoint for the duration of a test and +// returns a restore func the caller must defer — the cross-package sibling +// of SetBakedKeyForTest, so a caller in another package (e.g. cliconfig's +// disclosure-ordering tests) that also needs Available()==true can point +// the exporter at an httptest server instead of the real production +// ingest. +func SetEndpointForTest(url string) (restore func()) { + orig := endpoint + endpoint = url + return func() { endpoint = orig } +} + +// allowedResourceAttrs is the CLOSED set of resource attribute keys this +// pipeline may ever export, enforced by allowlistExporter below regardless +// of what the OTel SDK itself adds to a MeterProvider's resource. +var allowedResourceAttrs = map[string]bool{ + "service.name": true, + "service.version": true, + "mecatl.install.id": true, + "mecatl.binary": true, +} + +// allowlistExporter wraps a sdkmetric.Exporter and rewrites every exported +// ResourceMetrics.Resource down to allowedResourceAttrs immediately before +// serialization/transmission. This is the LAST line of defense against +// undeclared resource attributes reaching this vendor pipeline: the OTel SDK +// itself is not a clean pass-through here — metric.WithResource +// unconditionally merges the resource it is given with +// resource.Environment() (i.e. the ambient OTEL_RESOURCE_ATTRIBUTES env var) +// inside go.opentelemetry.io/otel/sdk/metric's own WithResource option, with +// no application-facing way to opt out. An operator's own +// OTEL_RESOURCE_ATTRIBUTES (meant for their OTLP/Prometheus pipeline) must +// never reach the Stacklok product-metrics destination, so filtering at +// export time — after the SDK's merge has already happened — is the only +// point that can enforce the closed catalog. +type allowlistExporter struct { + next sdkmetric.Exporter +} + +func (e *allowlistExporter) Temporality(k sdkmetric.InstrumentKind) metricdata.Temporality { + return e.next.Temporality(k) +} + +func (e *allowlistExporter) Aggregation(k sdkmetric.InstrumentKind) sdkmetric.Aggregation { + return e.next.Aggregation(k) +} + +func (e *allowlistExporter) Export(ctx context.Context, rm *metricdata.ResourceMetrics) error { + rm.Resource = allowlistResource(rm.Resource) + return e.next.Export(ctx, rm) +} + +func (e *allowlistExporter) ForceFlush(ctx context.Context) error { return e.next.ForceFlush(ctx) } + +func (e *allowlistExporter) Shutdown(ctx context.Context) error { return e.next.Shutdown(ctx) } + +// allowlistResource returns a fresh Resource carrying ONLY the attributes in +// allowedResourceAttrs from res — dropping anything else the SDK, an +// ambient env var, or a future dependency change might have added. +func allowlistResource(res *resource.Resource) *resource.Resource { + if res == nil { + return res + } + var kept []attribute.KeyValue + for _, kv := range res.Attributes() { + if allowedResourceAttrs[string(kv.Key)] { + kept = append(kept, kv) + } + } + return resource.NewSchemaless(kept...) +} + +// Provider wraps an OTLP metrics MeterProvider built directly over the OTel +// SDK's own otlpmetrichttp exporter (NOT toolhive-core's +// providers.NewCompositeProvider, which unconditionally adds +// resource.WithFromEnv() and resource.WithHost() to the exported resource on +// top of the SDK's own unconditional env merge — see allowlistExporter). +// Its MeterProvider is NEVER installed as the process-global provider +// (mirrors internal/adapter/telemetry's own discipline in otlp.go), so it +// cannot collide with an operator's own OTel setup. type Provider struct { - composite *providers.CompositeProvider + meterProvider *sdkmetric.MeterProvider } // NewProvider builds the product-metrics MeterProvider for one process. A @@ -46,13 +165,6 @@ func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { if bakedKey == "" { return nil, fmt.Errorf("productmetrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") } - // toolhive-core's OTLP metric exporter strips the scheme from the - // endpoint and defaults to a secure (TLS) connection regardless — so a - // plain http:// endpoint (only ever true in this package's own test, - // pointed at an httptest.Server) must explicitly opt into WithInsecure, - // or the exporter tries TLS against a plaintext listener and every - // export fails. The real production endpoint is always https://. - // // mecatl.install.id is a per-install random UUID, deliberately attached // as a resource attribute (so it flattens onto every instrument this // provider exports). This was removed once (see git history) over @@ -64,26 +176,41 @@ func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { // local install-id file — see internal/cliconfig's mecak8s wiring and // deploy/helm/mecak8s/templates/install-id-configmap.yaml), since a // pod-local file would mint a new id on every pod restart. - composite, err := providers.NewCompositeProvider(ctx, - providers.WithServiceName("mecatl"), - providers.WithServiceVersion(cfg.Version), - providers.WithOTLPEndpoint(endpoint), - providers.WithMetricsEnabled(true), - providers.WithInsecure(strings.HasPrefix(endpoint, "http://")), - providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), - providers.WithCustomAttributes(map[string]string{ - "mecatl.install.id": cfg.InstallID, - "mecatl.binary": string(cfg.Binary), - }), + res, err := resource.New(ctx, + resource.WithAttributes( + semconv.ServiceName("mecatl"), + semconv.ServiceVersion(cfg.Version), + attribute.String("mecatl.install.id", cfg.InstallID), + attribute.String("mecatl.binary", string(cfg.Binary)), + ), ) if err != nil { - return nil, fmt.Errorf("productmetrics: build provider: %w", err) + return nil, fmt.Errorf("productmetrics: build resource: %w", err) } - return &Provider{composite: composite}, nil + + // otlpmetrichttp.WithEndpointURL parses endpoint itself and derives the + // host, the URL path (verbatim — see the endpoint var doc), and + // TLS-vs-insecure transport (https:// scheme => secure, everything else + // => insecure) — so the plain http:// endpoint this package's own tests + // point at an httptest.Server against needs no separate WithInsecure() + // call. + exp, err := otlpmetrichttp.New(ctx, + otlpmetrichttp.WithEndpointURL(endpoint), + otlpmetrichttp.WithHeaders(map[string]string{headerKeyName: bakedKey}), + ) + if err != nil { + return nil, fmt.Errorf("productmetrics: build exporter: %w", err) + } + + mp := sdkmetric.NewMeterProvider( + sdkmetric.WithResource(res), + sdkmetric.WithReader(sdkmetric.NewPeriodicReader(&allowlistExporter{next: exp})), + ) + return &Provider{meterProvider: mp}, nil } // Meter returns the underlying metric.MeterProvider for instrument construction. -func (p *Provider) Meter() metric.MeterProvider { return p.composite.MeterProvider() } +func (p *Provider) Meter() metric.MeterProvider { return p.meterProvider } // Shutdown flushes and stops the provider, bounded by the caller's ctx. -func (p *Provider) Shutdown(ctx context.Context) error { return p.composite.Shutdown(ctx) } +func (p *Provider) Shutdown(ctx context.Context) error { return p.meterProvider.Shutdown(ctx) } diff --git a/internal/adapter/productmetrics/provider_test.go b/internal/adapter/productmetrics/provider_test.go index 62c736f335..662e59853a 100644 --- a/internal/adapter/productmetrics/provider_test.go +++ b/internal/adapter/productmetrics/provider_test.go @@ -2,9 +2,14 @@ package productmetrics import ( "context" + "io" "net/http" "net/http/httptest" + "os" "testing" + + otlpmetrics "go.opentelemetry.io/proto/otlp/collector/metrics/v1" + "google.golang.org/protobuf/proto" ) func TestNewProviderFailsClosedWithNoBakedKey(t *testing.T) { @@ -18,22 +23,76 @@ func TestNewProviderFailsClosedWithNoBakedKey(t *testing.T) { } } +// fakeMetricsCollector is a minimal httptest OTLP/HTTP metrics ingest that +// records the exact request path and the raw ExportMetricsServiceRequest +// body of every export, so tests can assert on both — not merely that a +// request without a path arrived (which cannot detect a doubled signal path +// or a leaked resource attribute). +type fakeMetricsCollector struct { + srv *httptest.Server + gotPath string + gotBody *otlpmetrics.ExportMetricsServiceRequest + gotAuth string +} + +func newFakeMetricsCollector(t *testing.T) *fakeMetricsCollector { + t.Helper() + c := &fakeMetricsCollector{} + c.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c.gotPath = r.URL.Path + c.gotAuth = r.Header.Get(headerKeyName) + body, err := io.ReadAll(r.Body) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + req := new(otlpmetrics.ExportMetricsServiceRequest) + if perr := proto.Unmarshal(body, req); perr != nil { + http.Error(w, perr.Error(), http.StatusBadRequest) + return + } + c.gotBody = req + w.WriteHeader(http.StatusOK) + })) + t.Cleanup(c.srv.Close) + return c +} + +// TestNewProviderExportsToConfiguredEndpoint pins the endpoint against a +// PRODUCTION-SHAPED base URL — the real `endpoint` var, not a bare +// pathless httptest.Server URL — so a regression that re-introduces the +// doubled "/v1/metrics/v1/metrics" signal path (toolhive-core's +// otlp.NewMetricReader appends its own "/v1/metrics" to any non-empty base +// path) fails this test instead of passing silently. It also seeds an +// ambient OTEL_RESOURCE_ATTRIBUTES value and asserts the exported +// ResourceMetrics.Resource carries ONLY the four declared attributes, so a +// regression back to providers.NewCompositeProvider's unconditional +// resource.WithFromEnv()/resource.WithHost() also fails here. func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { origKey := bakedKey bakedKey = "test-key" defer func() { bakedKey = origKey }() - var gotHeader string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotHeader = r.Header.Get(headerKeyName) - w.WriteHeader(http.StatusOK) - })) - defer srv.Close() + collector := newFakeMetricsCollector(t) origEndpoint := endpoint - endpoint = srv.URL + // Mirror the real production endpoint shape exactly: a base URL with NO + // signal path, matching `endpoint`'s documented contract. + endpoint = collector.srv.URL + "/v1/metrics" defer func() { endpoint = origEndpoint }() + origAttrs, hadAttrs := os.LookupEnv("OTEL_RESOURCE_ATTRIBUTES") + if err := os.Setenv("OTEL_RESOURCE_ATTRIBUTES", "tenant.id=customer-a"); err != nil { + t.Fatalf("Setenv: %v", err) + } + defer func() { + if hadAttrs { + os.Setenv("OTEL_RESOURCE_ATTRIBUTES", origAttrs) //nolint:errcheck // test cleanup + } else { + os.Unsetenv("OTEL_RESOURCE_ATTRIBUTES") //nolint:errcheck // test cleanup + } + }() + p, err := NewProvider(context.Background(), Config{ Binary: BinaryMecated, Version: "test", @@ -53,7 +112,33 @@ func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { if err := p.Shutdown(context.Background()); err != nil { t.Fatalf("Shutdown: %v", err) } - if gotHeader != "test-key" { - t.Errorf("collector received %s=%q, want %q", headerKeyName, gotHeader, "test-key") + + if collector.gotAuth != "test-key" { + t.Errorf("collector received %s=%q, want %q", headerKeyName, collector.gotAuth, "test-key") + } + if collector.gotPath != "/v1/metrics" { + t.Errorf("collector received request path %q, want %q (a doubled signal path regression)", collector.gotPath, "/v1/metrics") + } + if collector.gotBody == nil { + t.Fatal("collector received no decodable ExportMetricsServiceRequest") + } + + // Assert on the RESOURCE, not merely the instrument/data-point + // attributes: NewCompositeProvider's resource.WithFromEnv() + + // resource.WithHost() would surface both the seeded ambient + // OTEL_RESOURCE_ATTRIBUTES ("tenant.id") and the local hostname here. + allowed := map[string]bool{ + "service.name": true, + "service.version": true, + "mecatl.install.id": true, + "mecatl.binary": true, + } + for _, rm := range collector.gotBody.GetResourceMetrics() { + for _, attr := range rm.GetResource().GetAttributes() { + if !allowed[attr.GetKey()] { + t.Errorf("resource carries undeclared attribute %q=%q (privacy contract violation)", + attr.GetKey(), attr.GetValue().GetStringValue()) + } + } } } diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 348e432642..7de9aeb26d 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -73,6 +73,23 @@ type ProductMetricsHandles struct { // FirstRun (nothing was minted here, and the chart — not this process — owns // the id's lifecycle). Every other binary passes "" and keeps the local-file // behaviour unchanged. +// +// notify, when firstRun is true, is called EXACTLY ONCE, SYNCHRONOUSLY, +// BEFORE this function starts the heartbeat goroutine (whose first +// Heartbeat call fires immediately — see RunHeartbeat) and before it +// returns. ADR 0329 makes visible advance disclosure load-bearing for +// opt-out collection: printing the notice only after the caller later +// notices ProductMetricsHandles.FirstRun — e.g. after its own startup work, +// or worse, only at shutdown/flush time — leaves a window where the +// pipeline can record and export data before a human ever sees the notice. +// Calling notify here, before ANY export-capable state exists, closes that +// window regardless of what the caller does afterward (including an error +// path that flushes and discards the handles without ever consulting +// FirstRun). notify is nil-safe: a nil notify simply skips the call (kept +// for the disabled/dry-run/override paths and existing test callers that +// don't exercise disclosure). FirstRun is still returned on the handles for +// callers/tests that want to observe it, but it must never be the sole +// trigger for actually showing the notice. func BuildProductMetrics( ctx, heartbeatCtx context.Context, enabled, dryRun bool, @@ -82,6 +99,7 @@ func BuildProductMetrics( snap productmetrics.FeatureSnapshot, installIDOverride string, diag port.Diagnostics, + notify func(string), ) (ProductMetricsHandles, error) { noop := func(context.Context) error { return nil } if !enabled { @@ -98,9 +116,20 @@ func BuildProductMetrics( // below. Reinstated as a real, exported resource attribute (see // provider.go's doc comment) after its cardinality cost was sized and // accepted. An externally provisioned id (see installIDOverride) bypasses - // it: there is no file to read, write, or report a first run from. + // it entirely: there is no file to read, write, or report a first run + // from, so Available() gates ONLY this local-file branch — a keyless + // build (every local/dev/CI-test build) must not mint and persist an id + // file that a later release build's LoadOrCreateInstallIDDefault would + // then read back as "already exists", silently reporting firstRun=false + // for that build's genuine first export (see Available's doc comment). + // The override path still proceeds and fails later, at provider + // construction, exactly as before. installID, firstRun := installIDOverride, false if installID == "" { + if !productmetrics.Available() { + return ProductMetricsHandles{Shutdown: noop}, + fmt.Errorf("product metrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") + } var err error installID, firstRun, err = productmetrics.LoadOrCreateInstallIDDefault() if err != nil { @@ -123,6 +152,14 @@ func BuildProductMetrics( return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: recorder: %w", err) } + // Disclosure BEFORE the pipeline goes live: notify runs synchronously, + // here, before the heartbeat goroutine below is even started — so no + // export-capable state exists yet when a human is expected to have seen + // the notice. + if firstRun && notify != nil { + notify(ProductMetricsDisclosureNotice) + } + go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) armFirstValueTracking(ctx, recorder, installIDOverride, diag) @@ -180,8 +217,8 @@ func armFirstValueTracking(ctx context.Context, recorder *productmetrics.Recorde "error", err) already = false } - recorder.EnableFirstValueTracking(time.Now(), already, func() error { - _, werr := productmetrics.LoadOrCreateFirstValueMarkerDefault() - return werr + recorder.EnableFirstValueTracking(time.Now(), already, func() (bool, error) { + alreadyExisted, werr := productmetrics.LoadOrCreateFirstValueMarkerDefault() + return !alreadyExisted, werr }) } diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go index f3e2ba68eb..5e5e33c738 100644 --- a/internal/cliconfig/productmetrics_config.go +++ b/internal/cliconfig/productmetrics_config.go @@ -10,6 +10,7 @@ import ( "github.com/stacklok/mecatl/engine/port" "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" ) // doNotTrackOptOut reports whether a DO_NOT_TRACK env value means "opt out", @@ -87,6 +88,33 @@ func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { return true } +// ResolveProviderFamily derives the closed-set productmetrics.ProviderFamily +// from the same two CLI-level signals every one of the four mecatl binaries +// resolves at flag-parse time (useOpenAI is a dedicated --openai bool that +// exists on mecated/mecatequi/mecak8s; defaultProvider is --default-provider +// on all four). It NEVER returns the type's zero value — the enum has no +// zero-value member, only ProviderAnthropic/OpenAI/OpenRouter/Other — so a +// heartbeat can never emit the invalid provider_configured{family=""} that a +// hand-rolled, only-partly-populated FeatureSnapshot produced before. Kept +// here (not duplicated per binary) as the SINGLE shared oracle every +// productMetricsSnapshot in cmd/mecated, cmd/mecatui, cmd/mecatequi, and +// cmd/mecak8s calls, mirroring mecated's original Task 11 switch verbatim. +func ResolveProviderFamily(useOpenAI bool, defaultProvider string) productmetrics.ProviderFamily { + lower := strings.ToLower(defaultProvider) + switch { + case useOpenAI: + return productmetrics.ProviderOpenAI + case strings.Contains(lower, "openrouter"): + return productmetrics.ProviderOpenRouter + case strings.Contains(lower, "openai"): + return productmetrics.ProviderOpenAI + case defaultProvider == "" || strings.Contains(lower, "anthropic"): + return productmetrics.ProviderAnthropic + default: + return productmetrics.ProviderOther + } +} + // TeeToolCallRecorder combines multiple ToolCallRecorders into one — the // ToolCallRecorder twin of internal/adapter/telemetry.NewSink's EventSink // fan-out (no such helper existed before product metrics, because until now diff --git a/internal/cliconfig/productmetrics_config_test.go b/internal/cliconfig/productmetrics_config_test.go index dafadca459..0788a28818 100644 --- a/internal/cliconfig/productmetrics_config_test.go +++ b/internal/cliconfig/productmetrics_config_test.go @@ -6,6 +6,7 @@ import ( "github.com/stacklok/mecatl/engine/port" "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" ) func boolPtr(b bool) *bool { return &b } @@ -16,6 +17,34 @@ func envMap(m map[string]string) func(string) string { return func(name string) string { return m[name] } } +// TestResolveProviderFamily pins the closed-set mapping the product-metrics +// heartbeat relies on across all four mecatl binaries: it must never return +// the type's zero value (empty string) — the enum has no such member — so +// every case below lands on a real ProviderFamily. +func TestResolveProviderFamily(t *testing.T) { + cases := []struct { + name string + useOpenAI bool + defaultProvider string + want productmetrics.ProviderFamily + }{ + {"empty defaults to anthropic", false, "", productmetrics.ProviderAnthropic}, + {"useOpenAI wins regardless of defaultProvider", true, "openrouter/some-model", productmetrics.ProviderOpenAI}, + {"defaultProvider names openrouter", false, "openrouter/anthropic/claude", productmetrics.ProviderOpenRouter}, + {"defaultProvider names openai", false, "openai/gpt-5", productmetrics.ProviderOpenAI}, + {"defaultProvider names anthropic", false, "anthropic/claude-opus", productmetrics.ProviderAnthropic}, + {"defaultProvider is case-insensitive", false, "OpenRouter/x", productmetrics.ProviderOpenRouter}, + {"unrecognized defaultProvider falls to other", false, "some-custom-gateway", productmetrics.ProviderOther}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := ResolveProviderFamily(tc.useOpenAI, tc.defaultProvider); got != tc.want { + t.Errorf("ResolveProviderFamily(%v, %q) = %q, want %q", tc.useOpenAI, tc.defaultProvider, got, tc.want) + } + }) + } +} + func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { cases := []struct { name string diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go index 4ea5b97be5..e59d1a4602 100644 --- a/internal/cliconfig/productmetrics_test.go +++ b/internal/cliconfig/productmetrics_test.go @@ -2,6 +2,10 @@ package cliconfig import ( "context" + "net/http" + "net/http/httptest" + "os" + "path/filepath" "strings" "testing" @@ -15,7 +19,7 @@ import ( func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), false, false, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) if err != nil { t.Fatalf("BuildProductMetrics(enabled=false): %v", err) } @@ -35,7 +39,7 @@ func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { // surface the error rather than silently disabling, so a caller notices // its release build is missing the ldflag. _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) if err == nil { t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") } @@ -48,7 +52,7 @@ func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { // case (above) fails closed with no baked key. func TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), true, true, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) if err != nil { t.Fatalf("BuildProductMetrics(enabled=true, dryRun=true): %v", err) } @@ -71,7 +75,7 @@ func TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider(t *testing // regardless of the dry-run flag's value. func TestBuildProductMetricsDisabledDryRunStillNoop(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), false, true, - productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}) + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) if err != nil { t.Fatalf("BuildProductMetrics(enabled=false, dryRun=true): %v", err) } @@ -85,30 +89,28 @@ func TestBuildProductMetricsDisabledDryRunStillNoop(t *testing.T) { // must bypass LoadOrCreateInstallIDDefault ENTIRELY, not merely take // precedence over whatever it returns. // -// The oracle is which failure surfaces. With no resolvable state directory the -// local-file mechanism cannot even mint an id, so the no-override call fails at -// the install-id step; an override must get PAST that step and fail later, at -// provider construction (no baked ingest key in any non-release build). If the -// override were applied after the file read, both calls would report the same -// install-id error. +// The oracle is which failure surfaces, and at which layer. bakedKey is +// empty in every non-release build/test, so BuildProductMetrics.Available() +// gates the no-override path BEFORE it ever calls +// LoadOrCreateInstallIDDefault — its error therefore has NO "provider:" +// wrapping. An override skips that Available() gate (and the file read) +// entirely and instead fails later, inside productmetrics.NewProvider +// itself, whose error IS "provider:"-wrapped. If the override were applied +// after the Available()/install-id gate, both calls would report the exact +// same unwrapped error. func TestBuildProductMetricsInstallIDOverrideSkipsTheLocalFile(t *testing.T) { - // No XDG_STATE_HOME and no home dir => productmetrics.UserStateDir yields - // "" and LoadOrCreateInstallIDDefault fails closed. - t.Setenv("XDG_STATE_HOME", "") - t.Setenv("HOME", "") - _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, - "", port.NopDiagnostics{}) - if err == nil || !strings.Contains(err.Error(), "install id") { - t.Fatalf("no override with an unresolvable state dir: err = %v, want an install-id failure", err) + "", port.NopDiagnostics{}, nil) + if err == nil || strings.Contains(err.Error(), "provider:") { + t.Fatalf("no override: err = %v, want the unwrapped Available() gate error, not a provider-construction failure", err) } _, err = BuildProductMetrics(context.Background(), context.Background(), true, false, productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, - "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}) - if err == nil || strings.Contains(err.Error(), "install id") { - t.Fatalf("override with an unresolvable state dir: err = %v, want the install-id step skipped", err) + "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}, nil) + if err == nil || !strings.Contains(err.Error(), "provider:") { + t.Fatalf("override: err = %v, want a provider-construction failure (the Available()/install-id gate skipped)", err) } } @@ -128,7 +130,7 @@ func TestBuildProductMetricsInstallIDOverrideNeverReportsFirstRun(t *testing.T) t.Run(tc.name, func(t *testing.T) { h, err := BuildProductMetrics(context.Background(), context.Background(), tc.enabled, tc.dryRun, productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, - "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}) + "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}, nil) if err != nil { t.Fatalf("BuildProductMetrics with an install-id override: %v", err) } @@ -182,3 +184,90 @@ func TestArmFirstValueTrackingSkipsWhenInstallIDIsOverridden(t *testing.T) { } } } + +// TestBuildProductMetricsNeverMintsInstallIDWithoutABakedKey pins the +// keyless-build-to-release transition finding: without a baked ingest key +// (every local/dev/CI-test build), BuildProductMetrics must return an error +// WITHOUT ever creating the local install-id file. If it created that file +// anyway (the prior ordering), a LATER release build with a real baked key +// would read the file back as "already exists" and silently report +// FirstRun=false on its genuine first export — skipping the ADR +// 0329-mandated disclosure notice for that install's actual first +// transmission. +func TestBuildProductMetricsNeverMintsInstallIDWithoutABakedKey(t *testing.T) { + stateDir := t.TempDir() + t.Setenv("XDG_STATE_HOME", stateDir) + + _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) + if err == nil { + t.Fatal("expected an error with no baked ingest key, got nil") + } + + if _, statErr := os.Stat(filepath.Join(stateDir, "mecatl", "telemetry-id")); statErr == nil { + t.Fatal("BuildProductMetrics must not create the install-id file when no ingest key is baked into this build") + } else if !os.IsNotExist(statErr) { + t.Fatalf("unexpected error checking for the install-id file: %v", statErr) + } +} + +// TestBuildProductMetricsNotifiesSynchronouslyBeforeReturning pins the +// disclosure-ordering fix: notify must be called EXACTLY ONCE, synchronously +// — before BuildProductMetrics returns, before the heartbeat goroutine's +// first (immediate) export-eligible recording — precisely when firstRun is +// true, and never on a second call against the same install-id state (where +// firstRun is false). This uses SetBakedKeyForTest/SetEndpointForTest +// (pointed at a local httptest server, never the real production endpoint) +// so the Available()-gated real-provider path actually runs. +func TestBuildProductMetricsNotifiesSynchronouslyBeforeReturning(t *testing.T) { + restoreKey := productmetrics.SetBakedKeyForTest("test-key") + defer restoreKey() + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + restoreEndpoint := productmetrics.SetEndpointForTest(srv.URL + "/v1/metrics") + defer restoreEndpoint() + + t.Setenv("XDG_STATE_HOME", t.TempDir()) + + var notified []string + notify := func(s string) { notified = append(notified, s) } + + heartbeatCtx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h, err := BuildProductMetrics(context.Background(), heartbeatCtx, true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, notify) + if err != nil { + t.Fatalf("BuildProductMetrics: %v", err) + } + defer h.Shutdown(context.Background()) + + if !h.FirstRun { + t.Fatal("want FirstRun=true on a fresh install-id state directory") + } + if len(notified) != 1 || notified[0] != ProductMetricsDisclosureNotice { + t.Fatalf("notify calls = %v, want exactly one call carrying ProductMetricsDisclosureNotice", notified) + } + + // A second call against the SAME state directory reads back the + // already-minted id: firstRun is false, and notify must NOT fire again. + notified = nil + heartbeatCtx2, cancel2 := context.WithCancel(context.Background()) + defer cancel2() + h2, err := BuildProductMetrics(context.Background(), heartbeatCtx2, true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, notify) + if err != nil { + t.Fatalf("BuildProductMetrics (second call): %v", err) + } + defer h2.Shutdown(context.Background()) + + if h2.FirstRun { + t.Fatal("want FirstRun=false on the second call against an already-minted install id") + } + if len(notified) != 0 { + t.Fatalf("notify calls = %v, want none when firstRun is false", notified) + } +} From 414cb50bbaa3194d76c5995ba10d59bdd10295dc Mon Sep 17 00:00:00 2001 From: Reynier Ortiz Vega Date: Mon, 14 Sep 2026 11:39:30 -0400 Subject: [PATCH 47/47] fix(docs): renumber ADR 0329 to 0338 after another collision with main main gained a new ADR 0329 (native-llm-endpoint-gateway-credentials) after this branch's previous 0327->0329 renumbering, so the ADR uniqueness gate collided again. main is now at 0337, so this ADR moves to the next free number, 0338, with every code/doc reference updated. Co-Authored-By: Claude Sonnet 5 --- cmd/mecated/main.go | 2 +- cmd/mecatequi/observability.go | 4 ++-- cmd/mecatui/main.go | 2 +- docs/adr/{0329-product-metrics.md => 0338-product-metrics.md} | 2 +- docs/adr/README.md | 2 +- internal/adapter/productmetrics/provider.go | 2 +- internal/cliconfig/productmetrics.go | 4 ++-- internal/cliconfig/productmetrics_test.go | 2 +- user-docs/building/what-you-get/observability.md | 2 +- 9 files changed, 11 insertions(+), 11 deletions(-) rename docs/adr/{0329-product-metrics.md => 0338-product-metrics.md} (99%) diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index 79569a4ecf..d6cdd26f8d 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -1128,7 +1128,7 @@ func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { // before starting the heartbeat goroutine and before returning — never // deferred to a check on the returned handles' FirstRun field afterward, // which would leave a window where the pipeline could record/export before -// a human ever saw the notice (ADR 0329). run() only threads the resulting +// a human ever saw the notice (ADR 0338). run() only threads the resulting // handles and the heartbeat-context cancel func (both callers must defer // unconditionally: the handles' Shutdown is always a safe no-op when // disabled/errored). The returned error is informational only — a caller diff --git a/cmd/mecatequi/observability.go b/cmd/mecatequi/observability.go index 572f6ef947..189be1a1a2 100644 --- a/cmd/mecatequi/observability.go +++ b/cmd/mecatequi/observability.go @@ -48,7 +48,7 @@ type observability struct { // before it returns — rather than deferred to a check on the returned // handles' FirstRun field at flush time (flushTelemetry previously printed // it there, AFTER already calling Shutdown/flushing the provider: exactly -// the ordering ADR 0329 forbids for opt-out collection). +// the ordering ADR 0338 forbids for opt-out collection). func buildObservability(ctx context.Context, f flags, diag port.Diagnostics, stderr io.Writer) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecatequi", @@ -118,7 +118,7 @@ func productMetricsSnapshot(f flags) productmetrics.FeatureSnapshot { // NOT printed here: buildObservability's notify callback already wrote it, // synchronously, before the pipeline could ever record/export anything — see // buildObservability's doc comment. Printing it here instead (after -// Shutdown/flush has already run) is exactly the ordering ADR 0329 forbids. +// Shutdown/flush has already run) is exactly the ordering ADR 0338 forbids. func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { ctx := context.Background() if timeout > 0 { diff --git a/cmd/mecatui/main.go b/cmd/mecatui/main.go index 2b2999c297..a4ec6e3ad3 100644 --- a/cmd/mecatui/main.go +++ b/cmd/mecatui/main.go @@ -1025,7 +1025,7 @@ func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { // (resolveTransport, well before tea.NewProgram(...).Run() ever enters the // alt-screen) stderr is still plain, unbuffered terminal output; a // diag.Log-routed notice would instead land only in the diagnostics FILE -// (invisible, and dropped entirely under --quiet), defeating ADR 0329's +// (invisible, and dropped entirely under --quiet), defeating ADR 0338's // visible-disclosure requirement. This also runs BEFORE embed.Start, not // deferred to a check on the returned handles' FirstRun field after the // embedded server has started (which left a window where a failed diff --git a/docs/adr/0329-product-metrics.md b/docs/adr/0338-product-metrics.md similarity index 99% rename from docs/adr/0329-product-metrics.md rename to docs/adr/0338-product-metrics.md index a609f105ea..30e4d4a169 100644 --- a/docs/adr/0329-product-metrics.md +++ b/docs/adr/0338-product-metrics.md @@ -1,4 +1,4 @@ -# ADR 0329 — Product (adoption) metrics over OTLP +# ADR 0338 — Product (adoption) metrics over OTLP - Status: Accepted - Date: 2026-09-09 diff --git a/docs/adr/README.md b/docs/adr/README.md index 85c048a21c..103ad68e0d 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -207,7 +207,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) -- [0329 — Product (adoption) metrics over OTLP](./0329-product-metrics.md) +- [0338 — Product (adoption) metrics over OTLP](./0338-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go index 97e788f23d..36a1247786 100644 --- a/internal/adapter/productmetrics/provider.go +++ b/internal/adapter/productmetrics/provider.go @@ -56,7 +56,7 @@ var bakedKey = "" // build) that instead created the install-id file first, then failed here, // would leave that file behind — so a LATER release build's genuine first // export would read it back and report firstRun=false, silently skipping -// the disclosure notice ADR 0329 requires before that first export. +// the disclosure notice ADR 0338 requires before that first export. func Available() bool { return bakedKey != "" } // SetBakedKeyForTest overrides bakedKey for the duration of a test and diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go index 7de9aeb26d..d5895ddcb9 100644 --- a/internal/cliconfig/productmetrics.go +++ b/internal/cliconfig/productmetrics.go @@ -28,7 +28,7 @@ file path, raw tool name, or model id) to help Stacklok understand community adoption. This is on by default. To opt out: pass --product-metrics=false, set MECATL_PRODUCT_METRICS=false, set DO_NOT_TRACK=1, or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: -see docs/adr/0329-product-metrics.md. +see docs/adr/0338-product-metrics.md. ` // ProductMetricsHandles bundles the handles a cmd main threads into its @@ -77,7 +77,7 @@ type ProductMetricsHandles struct { // notify, when firstRun is true, is called EXACTLY ONCE, SYNCHRONOUSLY, // BEFORE this function starts the heartbeat goroutine (whose first // Heartbeat call fires immediately — see RunHeartbeat) and before it -// returns. ADR 0329 makes visible advance disclosure load-bearing for +// returns. ADR 0338 makes visible advance disclosure load-bearing for // opt-out collection: printing the notice only after the caller later // notices ProductMetricsHandles.FirstRun — e.g. after its own startup work, // or worse, only at shutdown/flush time — leaves a window where the diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go index e59d1a4602..5b304602e2 100644 --- a/internal/cliconfig/productmetrics_test.go +++ b/internal/cliconfig/productmetrics_test.go @@ -192,7 +192,7 @@ func TestArmFirstValueTrackingSkipsWhenInstallIDIsOverridden(t *testing.T) { // anyway (the prior ordering), a LATER release build with a real baked key // would read the file back as "already exists" and silently report // FirstRun=false on its genuine first export — skipping the ADR -// 0329-mandated disclosure notice for that install's actual first +// 0338-mandated disclosure notice for that install's actual first // transmission. func TestBuildProductMetricsNeverMintsInstallIDWithoutABakedKey(t *testing.T) { stateDir := t.TempDir() diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 4f9c2d5539..e39128c383 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -236,7 +236,7 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. -**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0329](https://github.com/stacklok/mecatl/blob/main/docs/adr/0329-product-metrics.md). +**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0338](https://github.com/stacklok/mecatl/blob/main/docs/adr/0338-product-metrics.md). **It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: