feat: add opt-out product/adoption metrics over OTLP - #1278
Open
reyortiz3 wants to merge 51 commits into
Open
Conversation
Design a separate, opt-out-by-default OTLP pipeline reporting bounded adoption/usage counters to Stacklok's public metrics collector, fully isolated from the existing operator-facing telemetry pipeline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
15-task TDD plan implementing the approved design: an independent internal/adapter/productmetrics adapter (toolhive-core-backed OTLP provider, bounded EventSink/ToolCallRecorder, heartbeat, install id, dry-run audit mode), operator-tier-only settings.yaml gate, opt-out precedence via cliconfig, and wiring into all four binaries. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 1 of the product-metrics-otel plan: create the internal/adapter/ productmetrics package skeleton with closed-set enums (Feature, ProviderFamily, DeploymentMode, Binary), the FeatureSnapshot value object + its enabled() method, and Config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds LoadOrCreateInstallID (+ LoadOrCreateInstallIDDefault), reading/ writing a bare v4 UUID under $XDG_STATE_HOME/mecatl/telemetry-id via xdgconfig.UserStateDir, regenerating on a missing/corrupt file and failing closed when no state dir can be resolved. Promotes github.com/google/uuid from indirect to direct in go.mod (task tidy). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… (revive) Adds a one-line doc comment above each of the four const blocks (Feature, ProviderFamily, DeploymentMode, Binary) in config.go to satisfy revive's exported-const-needs-comment rule. No names or values changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d-key gated Provider/NewProvider wraps toolhive-core's CompositeProvider to export mecatl adoption metrics to the fixed metrics.stacklok.com OTLP/HTTP ingest, gated by a build-time-baked ingest key (BUILD_LDFLAGS in Taskfile.yml). An empty baked key (every local/dev/CI build) disables the pipeline entirely, so NewProvider always errors. Also wires providers.WithInsecure off the endpoint's scheme, since toolhive-core strips the scheme and defaults to TLS regardless -- required for the package's own httptest-backed test to actually reach a plaintext listener. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… counts Recorder implements port.EventSink, constructing all ten adoption instruments up front (heartbeat/feature/provider/mode counters are wired for a later task's Heartbeat method; tool_calls for its ToolCall method). Emit reads only ev.Type, ev.Result.Stop, and ev.Result.Usage — never ev.Result.Text or ev.Result.Error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds Recorder.ToolCall (port.ToolCallRecorder) and Recorder.Heartbeat + RunHeartbeat, restoring the port.ToolCallRecorder compile-time assertion in metrics.go now that ToolCall exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tributes Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…chema Adds the operator-tier telemetry: subtree (telemetry.productMetrics.enabled) to permconfig.Config, parsed strictly (unknown keys error) mirroring OpenRouterSection. Also wires the top-level Config.UnmarshalYAML dispatch table (permconfig.go) so the new key is actually decoded, and registers the subtree in internal/configgen's BuildModel/tier-pin guard so the generated settings reference and skeleton stay in sync with the schema (both required for task lint && task test to stay green; this task adds schema only, resolution/wiring for the actual telemetry gate is a later task). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…opt-out Add the resolver half of the product-metrics config schema landed in Task 7: an operatorTelemetry field, captureTelemetry capture method mirroring captureOpenRouter, and the OperatorProductMetricsEnabled() accessor. Wired into both loadUserRules capture sites (CLI + user-global) and a project-tier WARN-ignore block in loadProjectRules, so a project repo can never flip a user's own product-metrics opt-out in either direction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…r fan-out Adds ProductMetricsPrecedence + ResolveProductMetricsEnabled (CLI flag > DO_NOT_TRACK env var > operator settings.yaml > default-enabled) and TeeToolCallRecorder, the ToolCallRecorder twin of telemetry.NewSink's EventSink fan-out, in a new internal/cliconfig/productmetrics_config.go. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…notice Adds the composition-level pipeline builder (install id -> provider -> recorder -> heartbeat goroutine) that a cmd main threads into its EventSink/ToolCallRecorder fan-out, plus the one-time opt-out disclosure notice text. Disabled returns zero handles with a no-op Shutdown; enabled with no baked ingest key surfaces the error rather than disabling silently. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds --product-metrics (default true) and threads the already-built productmetrics/cliconfig pipeline into mecated's EventSink/ToolCallRecorder fan-out. Resolves the effective enabled value via a throwaway permconfig resolver (mirroring mecated's own mcplogin.go precedent, since app.Build's internal resolver is never exposed back to run()) and registers the flag in helpmeta.go's completeness table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds --product-metrics (default true, embedded-server only) and threads the already-built productmetrics/cliconfig pipeline into mecatui's embedded server's EventSink/ToolCallRecorder fan-out. Unlike mecated, mecatui wires composition.Sink/ToolCallRecorder BEFORE embed.Start rather than at run() itself, because the perf-observability feature (independently OFF by default) can also overwrite those same fields inside embed.Start's setupPerf/wirePerfSinks. wirePerfSinks now captures whatever cfg.Sink/cfg.ToolCallRecorder already held before reassigning them, so the product-metrics tap survives untouched when perf is off (the default) and gets folded in via cliconfig.TeeToolCallRecorder alongside the perf metrics when perf is also enabled. The effective enabled value resolves through a throwaway permconfig resolver built with mecatui's own hardcoded embedded discovery posture (Conventional/ImportClaude: true), mirroring mecated's precedent since app.Build's own resolver is never exposed back to main.go. The disclosure notice goes through diag.Log (never stderr, which would corrupt the Bubble Tea alt-screen) and registers the flag in helpmeta.go's completeness table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extend buildObservability in both binaries to take a port.Diagnostics and build the opt-out product-metrics pipeline (cliconfig.BuildProductMetrics) alongside the existing opt-in OTLP telemetry, fanning both into the engine's EventSink/ToolCallRecorder (nil-guarded, since TeeToolCallRecorder is not nil-safe over an all-nil input). Adds --product-metrics (default true) to both flag surfaces, flushes the pipeline on exit/SIGTERM, and prints the first-run disclosure notice to stderr. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… not shutdown mecak8s is a long-running daemon, so printing the first-run disclosure only in flushTelemetry left it invisible until process exit (potentially days/weeks later) and never printed at all on a SIGKILL/OOM-kill. Move the print into buildObservability, right after BuildProductMetrics returns, mirroring cmd/mecated/main.go's setupProductMetrics shape. mecatequi is unaffected (single-shot process; setup and flush are seconds apart there). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… four binaries Adds DryRunRecorder (port.EventSink + port.ToolCallRecorder) which logs every would-be product-metrics observation via port.Diagnostics instead of exporting it over OTLP, reading only the same bounded fields the real Recorder reads (event type, stop reason, token counts by kind, feature/provider/mode enums) -- never a tool name, session id, or free text. BuildProductMetrics gains a dryRun bool (after enabled) and a port.Diagnostics parameter; when dryRun is true it short-circuits before any install-id read/write or real OTLP provider construction, firing one representative Heartbeat instead of a full ticker. Wires --product-metrics-dry-run (default false) into mecated, mecatui, mecatequi, and mecak8s, threading each binary's existing diag into its BuildProductMetrics call site. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds ADR 0317 recording the shipped product-metrics design (isolation architecture, the exact mecatl.adoption.* catalog, opt-out precedence, operator-tier-only settings gate, privacy guard test, dry-run mode, and why opt-out is defensible here), extends the existing observability.md user-docs page with a "Product / adoption metrics (opt-out)" section, and fills in ProductMetricsDisclosureNotice's doc-link placeholder. Also excludes docs/superpowers/ (the SDD workflow's committed plan/spec artifacts) from the matlatl docs corpus, matching its existing transient-artifact exclusions — required for `task docs --strict` to pass now that this feature's plan/spec files are the only content under that path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
main gained ADR 0317 (canonical-shell-command-tool) and 0318 while this branch was open; renumber the product-metrics ADR to the next free slot and fix every cross-reference (docs/adr/README.md, the disclosure notice string, user-docs). Caught by CI's TestRealADRNumberUniqueness. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Update docs/adr/README.md's index entry, the disclosure-notice string, and the user-docs link to point at the renumbered ADR file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…O_NOT_TRACK, dry-run heartbeat Address review findings from PR #1278: - Fix the mecated shutdown-defer order so cancelHeartbeat() runs before pm.Shutdown() (defers are LIFO), matching mecatui's correct order. - Derive productMetricsSnapshot's Mode from cfg.headless instead of hardcoding ModeInteractive, so headless mecated deployments report correctly. - Dedup subagent_used/team_used to one count per run (keyed by the loop-stamped RunID, cleared on EvResult), matching their documented "at least once per run" semantics instead of counting every EvSubagentStart/EvTeamStart. - Treat only a truthy DO_NOT_TRACK value as an opt-out ("0"/"false" no longer disable telemetry), per the consoledonottrack.com convention. - Run the dry-run pipeline's one-shot Heartbeat call synchronously instead of in an unawaited goroutine, so it can't be lost on a short-lived process like mecatequi. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ecatl.product.* mecatl.usage.* was considered and rejected: it would read ambiguously next to resource-usage metrics (CPU/RSS) in the same dashboard. mecatl.product.* is unambiguous and matches the feature's own name throughout the ADR/docs. Kept flat (no further category level under product.*) — ten instruments doesn't earn the added structure yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This pipeline's destination is a Prometheus-remote-write backend (stacklok/infra#5604), where every OTel resource attribute becomes a permanent label on EVERY instrument's time series. Attaching a random per-install UUID there would multiply active series by (installs x instrument count) with no bound as adoption grows -- an unbounded-cardinality cost for a precision (exact unique-install counts) the design never actually required. Remove Config.InstallID and the mecatl.install.id resource attribute entirely. installid.go's local file still exists to detect first-run for the disclosure notice, but its UUID value is now discarded rather than threaded to NewProvider. Unique-install counts become an approximation from heartbeat volume/cadence instead of an exact count. Also fixes two stale doc references caught while updating: the DO_NOT_TRACK docs said "any non-empty value" (already contradicted by an earlier fix that excludes "0"/"false"), and consoledonottrack.com has been domain-squatted since this was written -- both now point at the live donottrack.sh convention text. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion metrics Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds port.RunAwareToolCallRecorder, an OPTIONAL ToolCallRecorder extension that additionally receives the calling run's RunID. The dispatcher's one ToolCallRecorder call site (engine/agent/dispatch.go execute) type-asserts for it and prefers ToolCallForRun when implemented, falling back to the plain ToolCall otherwise — purely additive, no existing ToolCallRecorder implementer is affected. Mirrors the HookApprovalLearner precedent in engine/port/hookrunner.go. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rdinality cost Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…reToolCallRecorder
Consume the new optional port.RunAwareToolCallRecorder so this adapter can
correlate a tool call to the run that made it, and derive two new bounded
facts from that correlation:
- runs_completed gains had_tool_call ("true"/"false"): whether the run made
at least one SUCCESSFUL tool call. An errored-only run is false - the
product definition is "this run took an action", not "attempted one".
- tool_calls gains category + outcome. outcome is {success,error}; category
is the closed-set projection in toolCategory.
The prior subagent/team per-run dedup (runFamiliesUsed/usedFamilies/
firstInRun/clearRun) is unified into ONE mutex-guarded per-run record,
perRunTracker/perRunState, which the later run_duration and
tool_calls_per_run work extends. Its lock discipline is the point: every
method does its map lookup AND its field mutation as a single critical
section and no *perRunState escapes a locked region, because Emit (per
event) and ToolCallForRun (per tool call) run concurrently on a fan-out
run. finish() returns a value copy so the had_tool_call read at EvResult is
race-free, and clears the entry so the map stays bounded to live runs.
DEVIATION from the plan's stated approach, flagged for review: the plan
specified deriving category from the structural mcp__ prefix check ALONE,
"never a maintained allowlist". That design emits any non-mcp__ name
verbatim, which fails this package's existing privacy-guard test (it drives
ToolCall with Name: "secret-tool-name-marker" and asserts no marker reaches
an attribute value) and would put an unbounded label on the counter - a
catalog is not only built-ins plus MCP tools, since an agent def, a learned
skill, or a future extension seam can register a name derived from operator
or model input. toolCategory therefore keeps the mcp__ prefix bucket AND
gates verbatim names through builtinToolCategories, falling back to the
single literal "other". A new built-in showing up as "other" is the
visible, harmless prompt to add a line; a leak is impossible either way.
DryRunRecorder is brought into lockstep in the same commit (part of the
plan's Task 6): an audit path that understates what the real Recorder sends
defeats the purpose it exists for. Its remaining Task 6 work (the later
histograms) is untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the six team-coordination tools (SendMessage, AddTask, ClaimTask, CompleteTask, ListTasks, RecordFinding) and InspectSession, which were missing and collapsing into "other" -- losing exactly the delegation-adoption signal this metrics catalog cares about. Removes BashSystemTemp, a permission-gate name that is never a dispatched ToolCall.Name and so could never match. Fixes review findings from Task 3 of the activation-extension plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add mecatl.product.run_duration, a wall-clock histogram from a run's EvSessionInit to its EvResult. perRunState gains a startedAt field stamped by a new markStarted tracker method (same lock discipline as markFamilyUsed/markToolCall), and recordResult records the duration only when startedAt was actually observed, so a run whose EvSessionInit this Recorder never saw does not record a bogus duration. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Publish the per-run tool-call tally that perRunState already tracked as mecatl.product.tool_calls_per_run, an Int64 histogram recorded at every EvResult (a tool-less run contributes an honest 0, so the distribution keeps its left tail). Add mecatl.product.time_to_first_value: a Float64 histogram recorded at most ONCE per install, ever, measuring the duration from this install's first-seen moment to its first run that both made a successful tool call and ended on StopEndTurn. The once-ever contract has two halves: an in-memory firstValueTracker whose test-and-claim is one critical section (a fan-out deployment must not record two samples), and a local marker file under XDG_STATE_HOME mirroring installid.go's injected-filesystem shape, written at the instant the qualifying run is observed rather than at startup -- a process that never has a qualifying run must not burn the install's one sample. FirstValueRecorded is therefore a pure READ used at startup, distinct from LoadOrCreateFirstValueMarker (the write half used as the record callback). Recording is off unless armed: NewRecorder's signature is unchanged and an unarmed Recorder never touches the instrument, so every existing caller and test keeps its exact prior posture. BuildProductMetrics arms it with time.Now() as firstSeenAt -- deliberate approximation for a coarse onboarding signal, sound because the marker means the metric can only fire on an install that has not yet had a qualifying run. A marker-read failure degrades to "track it anyway" rather than failing the pipeline. The no-PII guard in bounded_test.go now walks histogram data points too, and fails hard on an aggregation shape it does not recognise, so a future instrument cannot silently fall outside it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion The instrument description claimed it approximates "install first-seen moment", but the value actually recorded is the START OF THE PROCESS that reaches the qualifying run -- the LAST startup with no marker, not the first. A user who runs mecatl repeatedly over several days and only reaches a qualifying run on day three would see minutes, not three days, which is not what the old description promised a dashboard consumer. Fixes a review finding from Task 5 of the activation-extension plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…per_run/time_to_first_value DryRunRecorder already logged category/outcome/had_tool_call (from an earlier merged task); it did not yet mirror Recorder's run_duration, tool_calls_per_run, or time_to_first_value fields on runs_completed. Emit's EvSessionInit case now marks the run started via the existing perRunTracker, and emitResult logs tool_calls_per_run and (when this process observed the run's start) run_duration_seconds alongside the existing fields. time_to_first_value diverges from Recorder by design: dry-run stays stateless (no XDG_STATE_HOME reads/writes), so instead of the real once-ever, persisted-marker sample it logs a would-be time_to_first_value observation on every qualifying run (StopEndTurn + at least one successful tool call) — more verbose than the real pipeline, but the honest choice for a "show what could be sent" audit surface. Documented on DryRunRecorder's doc comment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…onfigMap mecak8s runs storage-free with no PVC (ADR 0048), so it cannot use the local-file product-metrics install-id mechanism the other three binaries share: every pod restart would mint a fresh, never-reused id — the worst-case cardinality pattern for that pipeline. The chart now provisions ONE id per release into a `<fullname>-install-id` ConfigMap, using the standard `lookup` idiom so a `helm upgrade` reuses the existing value verbatim and only a first install mints a new one, and mounts it into the container via `MECATL_PRODUCT_METRICS_INSTALL_ID`. `BuildProductMetrics` gains an `installIDOverride` parameter: non-empty skips `LoadOrCreateInstallIDDefault` entirely (and never reports FirstRun — the chart, not this process, owns the id's lifecycle). The other three call sites pass "" and are behaviourally unchanged. Because the env var is unconditional, the Deployment now always renders an `env:` block; the chart tests' env-shape assertions run through a new `appEnv` helper so they keep asserting exactly what they did before, and the one render-equality test masks the per-render uuidv4. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…id reinstatement Updates ADR 0319's catalog table and adds two new sections covering this round's changes: the RunAwareToolCallRecorder engine port extension (why it exists, how the had_tool_call/category/outcome derivation works, and why category needs a maintained allowlist alongside the mcp__ prefix rule) and mecak8s's Helm ConfigMap install-id mechanism (why the local-file approach can't work there, how the lookup/uuidv4 idiom keeps it stable across upgrades). Updates the startup disclosure notice and the user-facing observability doc to honestly reflect that an anonymous per-install identifier is now collected, reversing the prior "no ... identifier" framing now that its reintroduction has been sized and accepted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…mposition decorators The final whole-branch review of the had_tool_call/tool_calls_per_run/ time_to_first_value work found that neither decorator between productmetrics.Recorder and Deps.ToolCallRecorder implemented the new port.RunAwareToolCallRecorder interface: cliconfig.multiToolCallRecorder (TeeToolCallRecorder's fan-out) and internal/adapter/server's capabilityToolCallRecorder (SessionMutationCapability.GuardToolCallRecorder) both had only ToolCall. Since dispatch.go's type-assertion is on the composed Deps.ToolCallRecorder value itself, it always failed and fell back to plain ToolCall -- silently making had_tool_call, tool_calls_per_run, and time_to_first_value inert in every real binary, despite every per-task test and gate passing (nothing exercised the recorder the composition layer actually installs). Both decorators now implement ToolCallForRun, forwarding to each wrapped recorder's richer form when available and falling back to ToolCall otherwise -- capabilityToolCallRecorder under the same capability.allows(id) gate ToolCall already uses. Adds a regression test in each package asserting the composed/guarded value still satisfies port.RunAwareToolCallRecorder, driven the same way dispatch.go drives it (type-assert, then call). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d-run bias, doc drift
Three more findings from the final whole-branch review:
- mecak8s has no durable local marker for time_to_first_value's
once-ever contract (the same storage-free problem, ADR 0048,
install-id solves via a Helm ConfigMap -- time_to_first_value never
got the same treatment). armFirstValueTracking now no-ops entirely
when installIDOverride is non-empty, so mecak8s simply doesn't
report this metric rather than silently emitting one sample per pod
restart under a "once per install, ever" label.
- perRunTracker.finish now returns (perRunState, tracked bool):
tracked is false for an empty RunID or a RunID this Recorder never
saw an EvSessionInit/tool call for (e.g. RetryFailedStep's
RunRequest{}, which mints no RunID). recordResult and DryRunRecorder
now skip tool_calls_per_run entirely for an untracked run instead of
recording a fabricated 0 -- a 0 there was indistinguishable from a
genuine zero-tool-call run and would have biased the distribution
downward for a retried run that may have made many real tool calls.
- ADR 0319 doc drift: a stale "deliberately never exported" line two
sections above the section that reverses it, a privacy paragraph
that both said "never a tool name" three rows below a table showing
tool_calls' category attribute IS one (bounded, allowlisted, but a
name) and cited the wrong file for the closed set, and a stale
"a later task publishes" comment in toolcall.go now that task has
landed. The bounded_test.go guard's vacuity floor is tightened from
a "< 10" floor to an exact 13 (NewRecorder's real instrument count),
and its driving code now exercises run_duration too (it previously
never observed an EvSessionInit for its run-aware scenario).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ics-otel # Conflicts: # Taskfile.yml # cmd/mecak8s/flags.go # deploy/helm/mecak8s/chart_test.go # deploy/helm/mecak8s/templates/deployment.yaml # docs/configuration-reference.md
…l location My merge-conflict resolution for docs/configuration-reference.md mistakenly regenerated the OLD full-content target -- ADR 0321's docs reorg had already landed on main, replacing that file with a short redirect stub and repointing the generator (per Taskfile.yml's docs:configref task) at user-docs/reference/configuration.md instead. Restores the stub and regenerates the correct target, which also picks up this branch's telemetry.productMetrics config documentation for the first time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… main origin/main independently used ADR number 0319 for an unrelated topic (release-archives-and-homebrew-tap) since this branch's last merge. Renumbers this branch's ADR to 0326 (the next free number) and fixes its three cross-references (the ADR index, the disclosure notice string, and the user-facing observability doc). The historical plan document under docs/superpowers/plans/ is left referencing the old number, matching this repo's convention that historical planning artifacts are not retroactively updated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…n with main main claimed 0326 for an unrelated ADR (lazy-toolhive-metadata-refresh) between this branch's last fetch and this push. Renumbers to 0327 (the next free number after fetching main's current tip) and fixes the same three cross-references as the prior renumbering. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
reyortiz3
marked this pull request as ready for review
September 11, 2026 17:18
The hardcoded endpoint was https://metrics.stacklok.com/v1/metrics, but the actual listener stacklok/infra#5604 provisions is https://mecatl.metrics.stacklok.com/v1/metrics (a mecatl. subdomain). As written, every export would have silently failed against the wrong host regardless of a correctly baked key. Fixes the endpoint var and its three references in the ADR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…n with main main claimed 0327 for an unrelated ADR (self-repository-action-refs) in the window between the last fetch and this push. Renumbers to 0329 (main's own tip was already at 0328) and fixes the same three cross-references. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…ase pipeline
Adds an "Extract the product-metrics ingest key" step to all four
release jobs that build a real mecated/mecatui/mecak8s artifact (the
three ko image jobs plus the GoReleaser archive job): it unwraps the
plain key from the MECATL_METRICS_KEY secret's {"mecatl": "<key>"}
JSON shape (mirroring the AWS Secrets Manager property
stacklok/infra#5604 reads) into MECATL_METRICS_INGEST_KEY, masks it,
and exports it via GITHUB_ENV for the build step that follows.
.goreleaser.yaml and .ko.yaml now read that env var into a
-X .../productmetrics.bakedKey=... ldflag alongside the existing
BuildID stamp, in every build block that ships a productmetrics-enabled
binary (mecated, mecatui, mecak8s). An absent/empty secret degrades to
an empty ldflag (product metrics stay disabled, matching every
local/dev/CI build's existing posture) rather than failing the release;
a SET-but-malformed secret (no non-empty .mecatl property) is a hard
error, since that indicates a real misconfiguration.
Verified end-to-end offline: a snapshot goreleaser build and a local
ko build --local both actually bake a test key into the resulting
binary (confirmed via strings). goreleaser check, task lint:actions
(actionlint + embedded shellcheck), and YAML parsing all clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… opt-out override Adds MECATL_PRODUCT_METRICS as a second env-var opt-out signal, checked BEFORE the generic DO_NOT_TRACK convention so it can win in either direction: a user can opt mecatl back in despite an ambient DO_NOT_TRACK=1 set for other tools, or opt mecatl out specifically without touching DO_NOT_TRACK. Named to match the existing --product-metrics flag and telemetry.productMetrics.enabled settings key exactly, deliberately not *_DO_NOT_TRACK or *_TELEMETRY -- internal/adapter/telemetry already means something else in this codebase (the unrelated, opt-in operator OTLP/Prometheus pipeline), so a same-flavored name here would misleadingly suggest this variable also touches that pipeline. Accepts any strconv.ParseBool-recognized spelling; unset, empty, or unparseable falls through to the next precedence tier (DO_NOT_TRACK, then the operator settings.yaml, then default-enabled) rather than being treated as a false positive. Updates the five surfaces that document or announce the opt-out mechanisms: the ADR's precedence list, the startup disclosure notice, each of the four binaries' --product-metrics flag help text, and the user-facing observability doc. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a fully independent, opt-out-by-default product/adoption-metrics pipeline (
internal/adapter/productmetrics) reporting bounded, privacy-safe counters to a dedicated public OTLP collector (stacklok/infra#5604), wired into all four mecatl binaries (mecated,mecatui,mecatequi,mecak8s).internal/adapter/telemetry): separateMeterProvider, separate destination, zero shared state — combined only via a fan-out at the composition edge.mecatl.product.*): an anonymous per-install identifier, install heartbeat, closed feature-flag/provider-family/deployment-mode enums, coarse session/run/tool-call/token counts, run duration, tool-calls-per-run, and a one-time-per-install time-to-first-value sample — never a session id, model id, raw tool/MCP-server name, or free text. Enforced by a reflect-based guard test onRecorder's public API.--product-metrics=false>MECATL_PRODUCT_METRICSenv var (a mecatl-specific override, wins in either direction overDO_NOT_TRACK) >DO_NOT_TRACKenv var (the cross-tool convention) > operator-tier-onlytelemetry.productMetrics.enabled: falsein settings.yaml > default enabled.--product-metrics-dry-runlogs every would-be observation viaport.Diagnosticsinstead of exporting, so an operator can confirm the "no PII" claim empirically.NewProviderrefuses to construct — a non-release build can never phone home. Wired end-to-end into the real release pipeline:MECATL_METRICS_KEY(a{"mecatl": "<key>"}GitHub Actions secret, mirroring the AWS Secrets Manager propertystacklok/infra#5604reads) is unwrapped and masked in.github/workflows/release.yml, then read by.goreleaser.yaml(Homebrew/archive builds) and.ko.yaml(container images) into a-X .../productmetrics.bakedKey=...ldflag for every binary that ships this pipeline (mecated,mecatui,mecak8s). An absent/malformed secret degrades to the same never-phones-home posture as a local build, not a failed release. Verified end-to-end offline: a snapshotgoreleaser buildand a localko build --localboth actually bake a test key into the resulting binary.https://mecatl.metrics.stacklok.com/v1/metrics(amecatl.subdomain — confirmed againststacklok/infra#5604's actual listener), headerx-mecatl-metrics-key.helm upgrade) rather than the local-file mechanism the other three binaries use. This coexists with the separate, independently-shipped operator-facingmecatl.installation.id(a different concern — an operator's own OTLP telemetry, not this pipeline) — both env vars and both ConfigMaps render unconditionally in the same Deployment.docs/adr/0329-product-metrics.md; user-facing docs inuser-docs/building/what-you-get/observability.md.Implemented via
superpowers:subagent-driven-developmentacross two rounds of task-by-task work, each with an isolated implementer + independent task review, culminating in a final whole-branch review each round with no unresolved Critical/Important findings.Metrics catalog (
mecatl.product.*)Resource attributes (set once per process, not per-metric labels):
service.name,service.version,mecatl.install.id(a per-install random UUID — see "Install identifier" below),mecatl.binary(mecated/mecatui/mecatequi/mecak8s).Heartbeat (on start, then every ~24h for long-running processes; single fire for
mecatequi):mecatl.product.heartbeatmecatl.product.feature_enabledfeature∈memory/guardrails/mcp/schedulingmecatl.product.provider_configuredfamily∈anthropic/openai/openrouter/other(never a model id/alias)mecatl.product.deployment_modemode∈interactive/headless/k8sCoarse usage & activation (derived from the event/tool-call tap):
mecatl.product.sessions_startedmecatl.product.runs_completedstop(bounded stop reason),had_tool_call(true/false: at least one successful tool call in the run)mecatl.product.tool_callscategory(a built-in tool's own name,mcpfor any MCP-server tool via the structuralmcp__prefix, orother— never a raw MCP server/tool name),outcome(success/error)mecatl.product.tokenskind∈input/output/cache_read/cache_write/reasoningmecatl.product.subagent_usedmecatl.product.team_usedmecatl.product.run_durationmecatl.product.tool_calls_per_runEvSessionInitwas observed)mecatl.product.time_to_first_valueNothing here is free text, a session/run/model identifier, a raw tool/MCP-server name, a file path, a prompt, or an output — every label value is drawn from a closed enum or a maintained allowlist with a safe
otherfallback. Full rationale, including the tool-category allowlist's design constraints, indocs/adr/0329-product-metrics.md.Install identifier
An anonymous per-install random UUID (
mecatl.install.id) is attached as a resource attribute. This was deliberately left out initially over Prometheus-remote-write cardinality concerns, then reintroduced after sizing the actual cost against the real backend (Amazon Managed Service for Prometheus): roughly $1,930/month at 100,000 installs under a worst-case assumption (every instrument, 24/7 uptime), roughly $650/month under a realistic one — well within what the adoption signal it enables (activation rate, time-to-first-value, weekly retention, exact unique-install counts) is worth.mecated/mecatui/mecatequipersist it in a local state file;mecak8s(storage-free, no PVC) gets it from a Helm-provisioned ConfigMap instead, so pod restarts don't churn the id.time_to_first_valueis not recorded onmecak8s— it needs a durable once-ever marker this pipeline doesn't yet have there, so it stays silent rather than emit a misleading per-restart sample.Engine change
Adds
port.RunAwareToolCallRecorder(engine/port/log.go), an optional, purely-additive extension toToolCallRecorderletting a recorder also learn which run a tool call belongs to — needed to correlateToolCall(session-scoped) with the event stream'sRunIDforhad_tool_call/tool_calls_per_run/time_to_first_value. Mirrors the existingHookApprovalLearnerprecedent exactly; no existingToolCallRecorderimplementer is affected.Added(minor) perengine/COMPATIBILITY.md. The optional capability is forwarded end-to-end through every composition-layer decorator betweenproductmetrics.RecorderandDeps.ToolCallRecorder(the tee fan-out and the session-mutation-capability guard) — verified with a dedicated regression test in each, since an optional-capability interface silently stops propagating the moment any decorator in the chain forwards only the base interface.Product questions this answers
Product framed the initial ask around four themes — activation, time to value, reliability, retention — as six concrete questions. Here's what this catalog can answer directly, and what needs query-time composition on top of it:
runs_completed{had_tool_call}+mecatl.install.idhad_tool_call="true"within a window. Not a single-metric readout — a query over the two.mecatl.product.time_to_first_valuestop=end_turn).mecatl.install.id+mecatl.product.heartbeat/sessions_startedruns_completed{stop, had_tool_call}had_tool_call="true", "success" =stop="end_turn". The rate iscount(had_tool_call=true AND stop=end_turn) / count(had_tool_call=true)— one query over one instrument's two attributes.mecatl.product.tool_calls{outcome}count(outcome=success) / count(outcome=success or error), optionally sliced by the boundedcategoryattribute to see which tool families are less reliable.mecatl.install.id+mecatl.product.heartbeat/sessions_startedThe two "not a dedicated metric" rows are the honest cost of keeping the catalog flat and bounded rather than pre-baking retention/WAU cohort logic into the pipeline itself — they're answerable with the data shipped here, just at query time in the observability backend, not as a single mecatl-emitted counter.
Test plan
task lint— 0 issues, every moduletask test— fully green (root, engine,internal/adapter/productmetrics/internal/cliconfig/internal/adapter/server, all provider submodules, GOWORK=off standalone hygiene proofs)go test ./deploy/helm/mecak8s/...— full chart suite green with the realhelmbinary, including the new install-id ConfigMap test and the coexistingmecatl.installation.idteststask build— all 5 binaries build, byte-identical local-dev posture confirmed (empty baked key)task docs/task site:build— 0 broken links/anchors/orphans/unreachablebin/mecated serve --help-allshows--product-metrics(default true) with accurate help textgoreleaser check,task lint:actions(actionlint + embedded shellcheck), and an offline snapshotgoreleaser build/ localko build --localall confirm the ingest key actually reaches the compiled binarymecatl.metrics.stacklok.comcollector (stacklok/infra#5604) is live/ready to receive traffic — confirmed live🤖 Generated with Claude Code