diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 2d5042d9af..0497afc4ea 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -131,6 +131,33 @@ jobs: - name: Login to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | ko login ghcr.io -u "${{ github.actor }}" --password-stdin + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .ko.yaml's ldflags read. GitHub + # already masks the raw secret in logs; ::add-mask:: additionally masks + # the unwrapped value, since deriving it doesn't inherit that automatically. + # An absent/empty secret degrades to a disabled pipeline (a bare `=` + # ldflag) rather than failing the release — only a SET-but-malformed + # secret (present, but no non-empty .mecatl property) is a hard error, + # since that is a real misconfiguration worth surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # Build multi-arch, push by digest, and have ko generate + push an SPDX # SBOM next to the image. --bare keeps the repo path clean (no import-path # suffix), matching `task ko:publish`. We tag both the version and latest. @@ -264,6 +291,33 @@ jobs: - name: Login to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | ko login ghcr.io -u "${{ github.actor }}" --password-stdin + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .ko.yaml's ldflags read. GitHub + # already masks the raw secret in logs; ::add-mask:: additionally masks + # the unwrapped value, since deriving it doesn't inherit that automatically. + # An absent/empty secret degrades to a disabled pipeline (a bare `=` + # ldflag) rather than failing the release — only a SET-but-malformed + # secret (present, but no non-empty .mecatl property) is a hard error, + # since that is a real misconfiguration worth surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # Build multi-arch, push by digest, and have ko generate + push an SPDX # SBOM next to the image. --bare keeps the repo path clean (no import-path # suffix). We tag both the version and latest. The brood-box agent label @@ -397,6 +451,33 @@ jobs: - name: Login to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | ko login ghcr.io -u "${{ github.actor }}" --password-stdin + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .ko.yaml's ldflags read. GitHub + # already masks the raw secret in logs; ::add-mask:: additionally masks + # the unwrapped value, since deriving it doesn't inherit that automatically. + # An absent/empty secret degrades to a disabled pipeline (a bare `=` + # ldflag) rather than failing the release — only a SET-but-malformed + # secret (present, but no non-empty .mecatl property) is a hard error, + # since that is a real misconfiguration worth surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # Build multi-arch, push by digest, and have ko generate + push an SPDX # SBOM next to the image. --bare keeps the repo path clean (no import-path # suffix). We tag both the version and latest. The digest is captured for @@ -851,6 +932,34 @@ jobs: repositories: homebrew-tap permission-contents: write + # Unwraps the plain ingest key from the MECATL_METRICS_KEY secret's + # {"mecatl": ""} JSON blob (stacklok/infra#5604's ExternalSecret + # mirrors the same AWS Secrets Manager property shape) into + # MECATL_METRICS_INGEST_KEY, which .goreleaser.yaml's ldflags read. + # GitHub already masks the raw secret in logs; ::add-mask:: additionally + # masks the unwrapped value, since deriving it doesn't inherit that + # automatically. An absent/empty secret degrades to a disabled pipeline + # (a bare `=` ldflag) rather than failing the release — only a + # SET-but-malformed secret (present, but no non-empty .mecatl property) + # is a hard error, since that is a real misconfiguration worth + # surfacing immediately. + - name: Extract the product-metrics ingest key + env: + MECATL_METRICS_KEY: ${{ secrets.MECATL_METRICS_KEY }} + run: | + set -euo pipefail + if [ -z "$MECATL_METRICS_KEY" ]; then + echo "::warning::MECATL_METRICS_KEY secret is not set; this build will ship with product metrics disabled" + exit 0 + fi + key=$(printf '%s' "$MECATL_METRICS_KEY" | jq -r '.mecatl // empty') + if [ -z "$key" ]; then + echo "::error::MECATL_METRICS_KEY secret is set but has no non-empty .mecatl property" + exit 1 + fi + echo "::add-mask::$key" + echo "MECATL_METRICS_INGEST_KEY=$key" >> "$GITHUB_ENV" + # `release --clean` wipes dist/, then builds, archives, SBOMs, checksums, # signs, creates the Release, uploads, and LAST pushes the formula. # diff --git a/.goreleaser.yaml b/.goreleaser.yaml index 3c08a97859..e7024b5041 100644 --- a/.goreleaser.yaml +++ b/.goreleaser.yaml @@ -56,9 +56,18 @@ builds: # `--version` byte-identical across ko, Task and GoReleaser builds. # Not `{{ .Tag }}` either: under --snapshot that reports the LAST tag, which # would lie about what was built. + # + # The product-metrics ingest key (stacklok/infra#5604): {{ .Env.MECATL_METRICS_INGEST_KEY }} + # reads the plain key the release workflow extracts from the MECATL_METRICS_KEY + # secret (a {"mecatl": ""} JSON blob — see .github/workflows/release.yml) + # into that env var BEFORE this step runs. An empty/unset value renders as a + # bare trailing `=`, which is a harmless empty-string ldflag: NewProvider + # refuses to construct on an empty bakedKey, so a run with no key configured + # degrades to the same never-phones-home posture as any local/dev build. ldflags: - -s -w - -X github.com/stacklok/mecatl/internal/buildinfo.BuildID=v{{ .Version }} + - -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{ .Env.MECATL_METRICS_INGEST_KEY }} - id: mecatui main: ./cmd/mecatui @@ -73,6 +82,7 @@ builds: ldflags: - -s -w - -X github.com/stacklok/mecatl/internal/buildinfo.BuildID=v{{ .Version }} + - -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{ .Env.MECATL_METRICS_INGEST_KEY }} # NOTE: no `gomod:` block, deliberately. `gomod.proxy` defaults to false; setting # it true would make GoReleaser fetch github.com/stacklok/mecatl/engine from diff --git a/.ko.yaml b/.ko.yaml index b63ec9e92a..a19aa30cde 100644 --- a/.ko.yaml +++ b/.ko.yaml @@ -36,6 +36,7 @@ builds: ldflags: - -s -w - '{{with index .Env "VERSION"}}-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.}}{{end}}' + - '{{with index .Env "MECATL_METRICS_INGEST_KEY"}}-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.}}{{end}}' # mecak8s (ADR 0048): the storage-free k8s-native agent binary. Same distroless # base + build flags as mecated — it is a thin peer of mecated that composes # app.Build with k8s-native defaults (Redis store + k8s lease + drain gate). @@ -48,11 +49,19 @@ builds: ldflags: - -s -w - '{{with index .Env "VERSION"}}-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.}}{{end}}' + - '{{with index .Env "MECATL_METRICS_INGEST_KEY"}}-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.}}{{end}}' # mecatui (issue #302): the optional Bubble Tea TUI binary. It overrides the # distroless base with the brood-box wolfi base (see baseImageOverrides above — # brood-box connects over SSH and needs a shell) and ships a kodata/agent.yaml # brood-box manifest alongside the binary. VERSION, when nonempty, is retained # verbatim as the explicit linker stamp. + # + # MECATL_METRICS_INGEST_KEY (all three builds above and below): the plain + # product-metrics ingest key the release workflow extracts from the + # MECATL_METRICS_KEY secret's {"mecatl": ""} JSON blob (see + # .github/workflows/release.yml) before invoking `ko build`. Absent/empty + # (any non-release build) omits the ldflag entirely, matching Taskfile.yml's + # local-build posture: NewProvider refuses to construct on an empty bakedKey. - id: mecatui main: ./cmd/mecatui flags: @@ -62,3 +71,4 @@ builds: ldflags: - -s -w - '{{with index .Env "VERSION"}}-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.}}{{end}}' + - '{{with index .Env "MECATL_METRICS_INGEST_KEY"}}-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.}}{{end}}' diff --git a/.matlatlignore b/.matlatlignore index f6dcbddb24..38ed9183e2 100644 --- a/.matlatlignore +++ b/.matlatlignore @@ -67,3 +67,12 @@ website/CLAUDE.md # Exclude both the committed reports and their ignored comparison copies. sdk/typescript/etc/*.api.md sdk/typescript/.api-extractor-temp/ + +# docs/superpowers/{plans,specs}/ are the SDD (spec-driven-development) workflow's +# committed historical record of a feature's implementation plan and design spec — +# a point-in-time process artifact analogous to review-*.md/handoff-*.md above, not +# navigational product documentation. They are intentionally standalone (no inbound +# or outbound doc links); the decisions they record land in a proper docs/adr/ entry +# instead, which DOES stay in the corpus. Left in, each new plan/spec adds a fresh +# orphan/unreachable pair that blocks `check --strict`. +docs/superpowers/ diff --git a/Taskfile.yml b/Taskfile.yml index ea78249a6f..13242d21b9 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -43,7 +43,8 @@ vars: else git describe --tags --match 'v[0-9]*' --always --dirty 2>/dev/null || printf %s dev fi - BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}}' + MECATL_METRICS_KEY: '{{.MECATL_METRICS_KEY | default ""}}' + BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}} -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.MECATL_METRICS_KEY}}' # The GoReleaser BINARY version. Must stay in lockstep with the `version:` # pinned in .github/workflows/release.yml and .github/workflows/ci.yml. # .goreleaser.yaml is loaded STRICTLY, so a config field newer than this pin diff --git a/cmd/mecak8s/flags.go b/cmd/mecak8s/flags.go index 441e887b16..b4e38982f6 100644 --- a/cmd/mecak8s/flags.go +++ b/cmd/mecak8s/flags.go @@ -298,7 +298,18 @@ type config struct { otlpMetricsEndpoint string otlpMetricsProtocol string otlpShutdownTimeout time.Duration - installationID string + + // productMetrics reports anonymous product-adoption metrics to Stacklok. + // OPT-OUT: ON by default. See the --product-metrics flag help text. + productMetrics bool + // productMetricsSet records whether --product-metrics was explicitly passed, + // so ResolveProductMetricsEnabled can let CLI out-rank DO_NOT_TRACK/settings. + productMetricsSet bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool + installationID string } // stringList is a repeatable string flag.Value, preserving order across @@ -475,6 +486,11 @@ func parseFlags(argv []string) (config, error) { fs.DurationVar(&cfg.otlpShutdownTimeout, "otlp-shutdown-timeout", 5*time.Second, "bound on the telemetry flush at SIGTERM (so a dead collector cannot hang shutdown). 0 disables the bound") fs.StringVar(&cfg.installationID, "telemetry-installation-id", os.Getenv("MECATL_INSTALLATION_ID"), "stable canonical UUID exported as the optional mecatl.installation.id OTel resource attribute (default: MECATL_INSTALLATION_ID; empty omits it)") + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") + fs.Usage = func() { _, _ = fmt.Fprint(fs.Output(), "Usage: mecak8s [flags]\n\n") flaghelp.PrintDefaults(fs.Output(), fs) @@ -507,6 +523,8 @@ func parseFlags(argv []string) (config, error) { cfg.reasoningEffortFlagSet = true case "subagent-model-router": cfg.subagentModelRouterSet = true + case "product-metrics": + cfg.productMetricsSet = true } markRetentionCLIFlag(&cfg.retentionCLISet, fl.Name) if fl.Name == "schedule-fire-retention" { @@ -730,8 +748,11 @@ func appConfig(cfg config, diag port.Diagnostics, obs observability) app.Config Diagnostics: diag, // Observability (issue #343, ADR 0098): OPT-IN. With no --otlp-* flags the // handles are zero-valued (nil) — the byte-identical no-metrics posture. - Sink: obs.Sink, - ToolCallRecorder: obs.ToolCallRecorder, + // The opt-out product-metrics Sink/ToolCallRecorder are folded in + // alongside (nil-guarded fan-out): both nil reproduces the + // byte-identical no-telemetry posture exactly. + Sink: productMetricsSink(obs), + ToolCallRecorder: productMetricsRecorder(obs), MetricsRoleScoper: obs.MetricsRoleScoper, SessionLoadFailureMetricsEmitter: obs.SessionLoadFailureMetricsEmitter, } diff --git a/cmd/mecak8s/main.go b/cmd/mecak8s/main.go index 0eb770fb6a..d1bcc0c430 100644 --- a/cmd/mecak8s/main.go +++ b/cmd/mecak8s/main.go @@ -76,7 +76,7 @@ func run() error { // flags this is a no-op (byte-identical default). The flush defer runs BEFORE // built.Close() (LIFO), so the OTLP flush completes before the service tears // down on the SIGTERM path. - obs, oerr := buildObservability(ctx, cfg) + obs, oerr := buildObservability(ctx, cfg, diag) if oerr != nil { return fmt.Errorf("telemetry: %w", oerr) } diff --git a/cmd/mecak8s/observability.go b/cmd/mecak8s/observability.go index aaf3bed39d..4f3612c3ac 100644 --- a/cmd/mecak8s/observability.go +++ b/cmd/mecak8s/observability.go @@ -4,26 +4,58 @@ import ( "context" "fmt" "io" + "os" "time" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/adapter/telemetry" + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" + "github.com/stacklok/mecatl/internal/buildinfo" "github.com/stacklok/mecatl/internal/cliconfig" ) // observability carries the telemetry handles run() threads into appConfig and // serve(), plus the flush-on-SIGTERM Shutdown. With no --otlp-* / --metrics-addr -// flags every field is zero-valued and Shutdown is a no-op (the byte-identical -// no-telemetry posture). +// flags every HeadlessTelemetryHandles field is zero-valued and Shutdown is a +// no-op (the byte-identical no-telemetry posture). productMetrics carries the +// opt-out product-adoption metrics handles (issue #343 follow-up): Shutdown is +// always non-nil (a no-op when disabled) so the caller can defer it +// unconditionally. type observability struct { cliconfig.HeadlessTelemetryHandles + productMetrics cliconfig.ProductMetricsHandles } // buildObservability wires the OPT-IN OTLP telemetry pipeline for mecak8s via // the shared cliconfig.HeadlessTelemetry helper (the SAME Setup→NewMetrics→ -// WithRole→scoper path mecated wires inline). With no --otlp-* endpoints the -// helper returns zero handles (byte-identical default). The caller owns the -// Shutdown defer (flush on SIGTERM). The /metrics loopback listener is wired -// separately in serve() from the returned Registry. -func buildObservability(ctx context.Context, cfg config) (observability, error) { +// WithRole→scoper path mecated wires inline), plus the OPT-OUT product-metrics +// pipeline. With no --otlp-* endpoints the OTLP helper returns zero handles +// (byte-identical default). The caller owns the Shutdown defer (flush on +// SIGTERM). The /metrics loopback listener is wired separately in serve() from +// the returned Registry. +// +// Product metrics: mecak8s is long-running, so heartbeatCtx is the SAME +// signal-driven ctx run() already has in scope (cancelled by its own +// signalCtx()/stop() chain on SIGTERM — no separate cancel function needed) +// and heartbeatInterval is productmetrics.DefaultHeartbeatInterval (the +// steady-state ticker). A build failure (e.g. --product-metrics=true forced on +// with no baked ingest key) is NEVER fatal — product metrics are best-effort — +// so it degrades to a no-op Shutdown rather than failing this function's error +// return (which stays meaningful for the OTLP half only). +// +// The first-run disclosure notice is printed via a notify callback +// BuildProductMetrics itself invokes SYNCHRONOUSLY — before it starts the +// heartbeat goroutine and before it returns — not deferred to a check on the +// returned handles' FirstRun field afterward. mecak8s is a long-running +// daemon (unlike mecatequi's single-shot process, where setup and flush are +// seconds apart): printing only at shutdown (flushTelemetry) would leave the +// notice invisible for as long as the process runs (potentially days/weeks) +// and never printed at all on a SIGKILL/OOM-kill with no graceful shutdown +// path. This mirrors cmd/mecated/main.go's setupProductMetrics, which prints +// at setup for the same reason. +func buildObservability(ctx context.Context, cfg config, diag port.Diagnostics) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecak8s", InstallationID: cfg.installationID, @@ -40,24 +72,126 @@ func buildObservability(ctx context.Context, cfg config) (observability, error) if err != nil { return observability{}, err } - return observability{HeadlessTelemetryHandles: h}, nil + + // Product metrics (opt-out): resolve the effective enabled value via a + // THROWAWAY resolver mirroring mecated's setupProductMetrics precedent — + // app.Build's own resolver is internal and never exposed back here, so + // this narrow read-only resolver re-parses the same operator + // settings.yaml (an accepted, negligible boot-time cost, same as the + // other two mains). + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: cfg.permissionsConventional, + ImportClaude: cfg.importClaudePermissions, + ExplicitFiles: []string(cfg.permissionConfigs), + Diagnostics: diag, + }, xdgconfig.OSEnv) + enabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.productMetricsSet, + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + // mecak8s cannot use the local-file install-id mechanism the other three + // binaries share: it runs storage-free with no PVC (ADR 0048), so every + // pod restart would mint a fresh, never-reused id — the worst-case + // cardinality pattern for this pipeline. The Helm chart instead provisions + // ONE stable id per release in a ConfigMap (see + // deploy/helm/mecak8s/templates/install-id-configmap.yaml) and threads it + // in through this env var. Empty (the binary run directly, outside the + // chart) falls back to BuildProductMetrics's own local-file default — + // still functional, just without the "one stable id per k8s deployment" + // guarantee the chart provides. + installIDOverride := os.Getenv("MECATL_PRODUCT_METRICS_INSTALL_ID") + pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, cfg.productMetricsDryRun, + productmetrics.BinaryMecak8s, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot(cfg), installIDOverride, diag, + // stderr, not diag: mecak8s already writes plain informational lines to + // stderr elsewhere (e.g. boundedClose's timeout line in main.go), and the + // disclosure banner is a one-time, human-facing notice rather than a + // structured operational log line. + func(notice string) { _, _ = fmt.Fprint(os.Stderr, notice) }) + if pmErr != nil { + // Mirror the existing telemetry-setup-failure posture: a warning, never + // a fatal error — product metrics are best-effort and must not block + // the daemon from starting. + diag.Log(ctx, port.LevelWarn, "product metrics disabled: setup failed", "err", pmErr) + pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} + } + + return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } -// flushTelemetry runs the telemetry Shutdown (flush) with a bounded ctx so a -// dead collector cannot hang SIGTERM shutdown. Safe on a zero observability -// (Shutdown is a no-op when telemetry is disabled). A flush failure is logged -// and never aborts — telemetry is best-effort at shutdown. -func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { - if obs.Shutdown == nil { - return +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on cfg — never a +// model id/alias, only whether each feature is configured at all. No Memory: +// mecak8s runs storage-free with no PVC (ADR 0048) — the same reason its +// install-id comes from a Helm ConfigMap rather than a local file (see +// buildObservability's installIDOverride handling, above). +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{ + Guardrails: cfg.guardrailsModel != "", + MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, + Scheduling: !cfg.noScheduler, + Provider: cliconfig.ResolveProviderFamily(cfg.useOpenAI, cfg.defaultProvider), + Mode: productmetrics.ModeK8s, } +} + +// flushTelemetry runs the OTLP + product-metrics Shutdown (flush) with a +// bounded ctx so a dead collector cannot hang SIGTERM shutdown. Safe on a zero +// observability (both Shutdowns are no-ops when telemetry is disabled). A +// flush failure is logged and never aborts — telemetry is best-effort at +// shutdown. (The first-run disclosure notice is printed at buildObservability +// setup time, not here — see that function's doc comment.) +func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { ctx := context.Background() if timeout > 0 { var cancel context.CancelFunc ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } - if err := obs.Shutdown(ctx); err != nil { - _, _ = fmt.Fprintf(stderr, "mecak8s: telemetry flush: %v\n", err) + if obs.Shutdown != nil { + if err := obs.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecak8s: telemetry flush: %v\n", err) + } + } + if obs.productMetrics.Shutdown != nil { + if err := obs.productMetrics.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecak8s: product metrics flush: %v\n", err) + } + } +} + +// productMetricsSink fans obs.Sink (the OTLP sink, nil when telemetry is off) +// together with obs.productMetrics.Sink (nil when product metrics are off) +// into one EventSink. telemetry.NewSink's fanOut.Emit calls every wrapped +// sink unconditionally, so a nil element would panic — both are filtered into +// a non-nil-only slice first (mirroring mecated/mecatui's Task 11/12 pattern). +// With both nil this returns nil, reproducing the byte-identical +// no-telemetry Sink posture exactly. +func productMetricsSink(obs observability) port.EventSink { + var sinks []port.EventSink + if obs.Sink != nil { + sinks = append(sinks, obs.Sink) + } + if obs.productMetrics.Sink != nil { + sinks = append(sinks, obs.productMetrics.Sink) + } + if len(sinks) == 0 { + return nil + } + return telemetry.NewSink(sinks...) +} + +// productMetricsRecorder fans obs.ToolCallRecorder together with +// obs.productMetrics.ToolCallRecorder via cliconfig.TeeToolCallRecorder — but +// ONLY when at least one is non-nil. TeeToolCallRecorder always returns a +// non-nil multiToolCallRecorder interface value even over an all-nil input +// (a typed-nil-slice wrapper, not a nil interface), which would break the +// byte-identical no-telemetry ToolCallRecorder-is-nil posture when both +// sources are off. With both nil this returns nil. +func productMetricsRecorder(obs observability) port.ToolCallRecorder { + if obs.ToolCallRecorder == nil && obs.productMetrics.ToolCallRecorder == nil { + return nil } + return cliconfig.TeeToolCallRecorder(obs.ToolCallRecorder, obs.productMetrics.ToolCallRecorder) } diff --git a/cmd/mecak8s/productmetrics_test.go b/cmd/mecak8s/productmetrics_test.go new file mode 100644 index 0000000000..fe49ad8a2a --- /dev/null +++ b/cmd/mecak8s/productmetrics_test.go @@ -0,0 +1,55 @@ +package main + +import ( + "flag" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins that mecak8s's +// heartbeat surfaces Guardrails/MCP/Scheduling/Provider from its own config +// rather than leaving them at their zero values — an empty Provider would +// heartbeat as the invalid provider_configured{family=""}, outside the +// documented anthropic|openai|openrouter|other enum. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + mcpServers := cliconfig.RegisterMCPServerFlag(flag.NewFlagSet("test", flag.ContinueOnError), "") + if err := mcpServers.Set("example=https://mcp.example.com"); err != nil { + t.Fatalf("mcpServers.Set: %v", err) + } + if err := mcpServers.Finalize(); err != nil { + t.Fatalf("mcpServers.Finalize: %v", err) + } + + snap := productMetricsSnapshot(config{ + guardrailsModel: "claude-haiku", + mcpServers: mcpServers, + noScheduler: false, + defaultProvider: "openrouter/some-model", + }) + if !snap.Guardrails { + t.Error("Guardrails = false, want true (guardrailsModel configured)") + } + if !snap.MCP { + t.Error("MCP = false, want true (an mcp server is configured)") + } + if !snap.Scheduling { + t.Error("Scheduling = false, want true (noScheduler is false)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } + if snap.Mode != productmetrics.ModeK8s { + t.Errorf("Mode = %q, want %q", snap.Mode, productmetrics.ModeK8s) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value config still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(config{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/cmd/mecak8s/telemetry_test.go b/cmd/mecak8s/telemetry_test.go index 7f9a82c30e..438aeab06d 100644 --- a/cmd/mecak8s/telemetry_test.go +++ b/cmd/mecak8s/telemetry_test.go @@ -185,7 +185,7 @@ func TestTelemetryDefaultIsNil(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), cfg) + obs, err := buildObservability(context.Background(), cfg, port.NopDiagnostics{}) if err != nil { t.Fatalf("buildObservability: %v", err) } @@ -240,7 +240,7 @@ func TestTelemetryMetricsAddrServesPrometheus(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), cfg) + obs, err := buildObservability(context.Background(), cfg, port.NopDiagnostics{}) if err != nil { t.Fatalf("buildObservability: %v", err) } @@ -355,7 +355,7 @@ func TestTelemetryPushesRunMetricsOnExit(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), cfg) + obs, err := buildObservability(context.Background(), cfg, port.NopDiagnostics{}) if err != nil { t.Fatalf("buildObservability: %v", err) } diff --git a/cmd/mecated/helpmeta.go b/cmd/mecated/helpmeta.go index bece9fffaf..b746b133d2 100644 --- a/cmd/mecated/helpmeta.go +++ b/cmd/mecated/helpmeta.go @@ -107,6 +107,8 @@ var flagMetaByFlag = map[string]flagMeta{ "perf-mcp": {group: groupObservability, common: false, acp: acpExclude}, "goroutine-warn-threshold": {group: groupObservability, common: false, acp: acpExclude}, "goroutine-warn-interval": {group: groupObservability, common: false, acp: acpExclude}, + "product-metrics": {group: groupObservability, common: true, acp: acpInclude}, + "product-metrics-dry-run": {group: groupObservability, common: false, acp: acpExclude}, // ── Driver connectivity (serve-only) ────────────────────────────────── "driver-auth-token": {group: groupDriver, common: false, acp: acpExclude}, diff --git a/cmd/mecated/main.go b/cmd/mecated/main.go index c263b8f1e6..d6cdd26f8d 100644 --- a/cmd/mecated/main.go +++ b/cmd/mecated/main.go @@ -50,6 +50,8 @@ import ( "github.com/stacklok/mecatl/internal/adapter/mcpauthority" "github.com/stacklok/mecatl/internal/adapter/mcpbroker" "github.com/stacklok/mecatl/internal/adapter/mcpperf" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" "github.com/stacklok/mecatl/internal/adapter/server" "github.com/stacklok/mecatl/internal/adapter/skills" "github.com/stacklok/mecatl/internal/adapter/slogdiag" @@ -182,6 +184,14 @@ type config struct { otlpProtocol string // OTLP transport: "grpc" (default) or "http" otlpInsecure bool // skip TLS when dialing the OTLP collector (dev only) + // productMetrics reports anonymous product-adoption metrics to Stacklok. + // OPT-OUT: ON by default. See the --product-metrics flag help text. + productMetrics bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool + // Runtime-introspection admin surface (loopback only, on the --metrics-addr // listener): pprof + expvar + a runtime/metrics snapshot + a FlightRecorder. // mutexProfileFraction arms runtime.SetMutexProfileFraction (0 = off); @@ -920,6 +930,21 @@ func run(mode commandMode, remaining []string) error { defer obs.recorder.Stop() } + // Product metrics (opt-out, Task 11): resolve the effective enabled value + // and build the pipeline. Extracted into a helper (mirroring + // setupObservability) so run()'s cyclomatic complexity stays under the + // lint gate; the helper owns the resolve/build/disclosure branches and + // logs its own failure, so run() only threads the resulting handles. + pm, cancelHeartbeat, _ := setupProductMetrics(ctx, cfg, diag) + defer func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if serr := pm.Shutdown(shutdownCtx); serr != nil { + slog.Warn("product metrics shutdown", "err", serr) + } + }() + defer cancelHeartbeat() + tracing := telemetry.NewTracing(otel.GetTracerProvider()) // Role-scoped main pair (issue #47): the MAIN engine records through the @@ -939,6 +964,9 @@ func run(mode commandMode, remaining []string) error { slowTurns = telemetry.NewSlowTurnBuffer(telemetry.DefaultSlowTurnCapacity, time.Now) sinks = append(sinks, slowTurns.WithRole(telemetry.RoleMain)) } + if pm.Sink != nil { + sinks = append(sinks, pm.Sink) + } sink := telemetry.NewSink(sinks...) // Child role scoper (issue #47): the composition hands each CHILD engine a @@ -955,7 +983,7 @@ func run(mode commandMode, remaining []string) error { return telemetry.NewSink(childSinks...), scoped } - composition := appConfig(cfg, sink, mainScoped, roleScoper, obs.metrics, diag) + composition := appConfig(cfg, sink, cliconfig.TeeToolCallRecorder(mainScoped, pm.ToolCallRecorder), roleScoper, obs.metrics, diag) built, err := app.Build(ctx, composition) if err != nil { return err @@ -1068,6 +1096,66 @@ func setupObservability(ctx context.Context, cfg config, diag port.Diagnostics) return observability{providers: providers, metrics: metrics, recorder: recorder}, nil } +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on cfg — never a +// model id/alias, only whether each feature is configured at all. +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + mode := productmetrics.ModeInteractive + if cfg.headless { + mode = productmetrics.ModeHeadless + } + return productmetrics.FeatureSnapshot{ + Memory: cfg.memoryDir != "", + Guardrails: cfg.guardrailsModel != "", + MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, + Scheduling: !cfg.noScheduler, + Provider: cliconfig.ResolveProviderFamily(cfg.useOpenAI, cfg.defaultProvider), + Mode: mode, + } +} + +// setupProductMetrics resolves the opt-out product-metrics precedence and +// builds the pipeline (Task 11). It reads the operator's +// telemetry.productMetrics.enabled setting via a THROWAWAY resolver built +// the SAME WAY internal/app/build.go's buildPermResolver constructs its — +// app.Build's own resolver is internal and never exposed back to run(), so +// this narrow read-only resolver mirrors mecated's mcplogin.go precedent +// (loadMCPLoginProfiles). settings.yaml is parsed twice at boot (once here, +// once inside app.Build); an accepted, negligible boot-time cost. +// +// It logs its own build failure. The first-run disclosure notice is printed +// via a notify callback BuildProductMetrics itself invokes SYNCHRONOUSLY, +// before starting the heartbeat goroutine and before returning — never +// deferred to a check on the returned handles' FirstRun field afterward, +// which would leave a window where the pipeline could record/export before +// a human ever saw the notice (ADR 0338). run() only threads the resulting +// handles and the heartbeat-context cancel func (both callers must defer +// unconditionally: the handles' Shutdown is always a safe no-op when +// disabled/errored). The returned error is informational only — a caller +// that just wants the handles can discard it. +func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) (cliconfig.ProductMetricsHandles, func(), error) { + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: cfg.permissionsConventional, + ImportClaude: cfg.importClaudePermissions, + ExplicitFiles: cfg.permissionConfigs, + Diagnostics: diag, + }, xdgconfig.OSEnv) + productMetricsEnabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.cliExplicit["product-metrics"], + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, + productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot(cfg), "" /* no install-id override: local-file mechanism */, diag, + func(notice string) { fmt.Fprint(os.Stderr, notice) }) + if err != nil { + slog.Warn("product metrics disabled: setup failed", "err", err) + } + return pm, cancelHeartbeat, err +} + // mecatedServerImplementation is the stable family reported to authenticated clients. const mecatedServerImplementation = "mecated" @@ -1620,6 +1708,11 @@ func parseFlagsModeOut(mode commandMode, argv []string, out io.Writer) (*flag.Fl fs.StringVar(&cfg.otlpProtocol, "otlp-protocol", telemetry.ProtocolGRPC, "OTLP transport: \"grpc\" (default) or \"http\"") fs.BoolVar(&cfg.otlpInsecure, "otlp-insecure", false, "skip TLS when dialing the OTLP collector (development only)") + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") + fs.IntVar(&cfg.mutexProfileFraction, "mutex-profile-fraction", 0, "runtime.SetMutexProfileFraction: report 1/N mutex contention events for /debug/pprof/mutex. 0 (default) disables it. Adds per-contention sampling overhead; enable only when investigating lock contention") fs.IntVar(&cfg.blockProfileRate, "block-profile-rate", 0, "runtime.SetBlockProfileRate in nanoseconds: sample one blocking event per N ns blocked for /debug/pprof/block. 0 (default) disables it. Adds per-block-event overhead; enable only when investigating blocking") fs.BoolVar(&cfg.flightRecorder, "flight-recorder", true, "arm the execution-trace FlightRecorder (bounded in-memory ring buffer) so /debug/flightrecorder can snapshot recent activity. ON by default (low, bounded overhead). Pass --flight-recorder=false to disable") diff --git a/cmd/mecated/productmetrics_test.go b/cmd/mecated/productmetrics_test.go new file mode 100644 index 0000000000..039e3dfa5b --- /dev/null +++ b/cmd/mecated/productmetrics_test.go @@ -0,0 +1,73 @@ +package main + +import ( + "flag" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// TestProductMetricsSnapshotModeReflectsHeadless pins that the deployment +// mode reported to the product-metrics heartbeat matches --headless — a +// headless mecated (autonomous/CI deployment) must report ModeHeadless, not +// the default ModeInteractive, or the adoption dashboard's headless/ +// interactive split is corrupted for every headless mecated server. +func TestProductMetricsSnapshotModeReflectsHeadless(t *testing.T) { + if got := productMetricsSnapshot(config{}).Mode; got != productmetrics.ModeInteractive { + t.Errorf("Mode = %q, want %q (default interactive)", got, productmetrics.ModeInteractive) + } + if got := productMetricsSnapshot(config{headless: true}).Mode; got != productmetrics.ModeHeadless { + t.Errorf("Mode = %q, want %q (--headless)", got, productmetrics.ModeHeadless) + } +} + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins the full +// heartbeat contract: every configured feature must surface on the +// FeatureSnapshot, and Provider must resolve to a real member of the +// documented anthropic|openai|openrouter|other enum — never the zero value +// (empty string), which would heartbeat as the invalid +// provider_configured{family=""}. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + mcpServers := cliconfig.RegisterMCPServerFlag(flag.NewFlagSet("test", flag.ContinueOnError), "") + if err := mcpServers.Set("example=https://mcp.example.com"); err != nil { + t.Fatalf("mcpServers.Set: %v", err) + } + if err := mcpServers.Finalize(); err != nil { + t.Fatalf("mcpServers.Finalize: %v", err) + } + + cfg := config{ + memoryDir: "/tmp/memory", + guardrailsModel: "claude-haiku", + mcpServers: mcpServers, + noScheduler: false, + defaultProvider: "openrouter/some-model", + } + snap := productMetricsSnapshot(cfg) + + if !snap.Memory { + t.Error("Memory = false, want true (memoryDir configured)") + } + if !snap.Guardrails { + t.Error("Guardrails = false, want true (guardrailsModel configured)") + } + if !snap.MCP { + t.Error("MCP = false, want true (an mcp server is configured)") + } + if !snap.Scheduling { + t.Error("Scheduling = false, want true (noScheduler is false)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value config still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(config{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/cmd/mecatequi/flags.go b/cmd/mecatequi/flags.go index a93d72818c..47c98b7614 100644 --- a/cmd/mecatequi/flags.go +++ b/cmd/mecatequi/flags.go @@ -162,6 +162,17 @@ type flags struct { otlpMetricsEndpoint string otlpMetricsProtocol string otlpShutdownTimeout time.Duration + + // productMetrics reports anonymous product-adoption metrics to Stacklok. + // OPT-OUT: ON by default. See the --product-metrics flag help text. + productMetrics bool + // productMetricsSet records whether --product-metrics was explicitly passed, + // so ResolveProductMetricsEnabled can let CLI out-rank DO_NOT_TRACK/settings. + productMetricsSet bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool } // parseFlags turns argv into a flags value, resolving env-derived defaults and @@ -238,6 +249,11 @@ func parseFlags(argv []string) (flags, error) { fs.StringVar(&f.otlpMetricsProtocol, "otlp-metrics-protocol", "grpc", "OTLP transport for metrics: \"grpc\" (default) or \"http\"") fs.DurationVar(&f.otlpShutdownTimeout, "otlp-shutdown-timeout", 5*time.Second, "bound on the telemetry flush at exit (so a dead collector cannot hang the run). 0 disables the bound (flush until it completes); the flush runs BEFORE the diff/summary emit defer unwinds") + fs.BoolVar(&f.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&f.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") + fs.Usage = usageEpilogue(fs) if err := fs.Parse(cliconfig.NormalizeLegacyNoBash(argv)); err != nil { @@ -267,6 +283,8 @@ func parseFlags(argv []string) (flags, error) { // --out-summary=- selects it. The unset default also resolves to "-" // but keeps the indented JSON — default behavior unchanged. f.summaryCompact = f.outSummary == "-" + case "product-metrics": + f.productMetricsSet = true } if fl.Name == "reasoning-effort" { f.reasoningEffortFlagSet = true @@ -450,9 +468,12 @@ func appConfig(f flags, diag port.Diagnostics, obs observability) app.Config { Diagnostics: diag, // Observability (issue #343, ADR 0098): OPT-IN OTLP push. With no --otlp-* // flags the handles are zero-valued (nil Sink/ToolCallRecorder/ - // MetricsRoleScoper) — the byte-identical no-telemetry posture. - Sink: obs.Sink, - ToolCallRecorder: obs.ToolCallRecorder, + // MetricsRoleScoper) — the byte-identical no-telemetry posture. The + // opt-out product-metrics Sink/ToolCallRecorder are folded in alongside + // (nil-guarded fan-out): both nil reproduces the byte-identical + // no-telemetry posture exactly. + Sink: productMetricsSink(obs), + ToolCallRecorder: productMetricsRecorder(obs), MetricsRoleScoper: obs.MetricsRoleScoper, SessionLoadFailureMetricsEmitter: obs.SessionLoadFailureMetricsEmitter, } diff --git a/cmd/mecatequi/main.go b/cmd/mecatequi/main.go index c09e6fbb50..de2d525a87 100644 --- a/cmd/mecatequi/main.go +++ b/cmd/mecatequi/main.go @@ -73,7 +73,7 @@ func realMain(argv []string, stdout, stderr io.Writer) int { // Observability (issue #343, ADR 0098): OPT-IN OTLP push. Built right after // flag parse so the flush-on-exit defer covers EVERY exit path (setup-failure // included). With no --otlp-* flags this is a no-op (byte-identical default). - obs, oerr := buildObservability(context.Background(), f) + obs, oerr := buildObservability(context.Background(), f, diag, stderr) if oerr != nil { _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry: %v\n", oerr) return 2 diff --git a/cmd/mecatequi/observability.go b/cmd/mecatequi/observability.go index b7dba950a6..189be1a1a2 100644 --- a/cmd/mecatequi/observability.go +++ b/cmd/mecatequi/observability.go @@ -6,24 +6,50 @@ import ( "io" "time" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/adapter/telemetry" + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" + "github.com/stacklok/mecatl/internal/buildinfo" "github.com/stacklok/mecatl/internal/cliconfig" ) // observability carries the telemetry handles realMain threads into appConfig, // plus the flush-on-exit Shutdown the main owns. With no --otlp-* flags every -// field is zero-valued and Shutdown is a no-op (the byte-identical no-telemetry -// posture). +// HeadlessTelemetryHandles field is zero-valued and Shutdown is a no-op (the +// byte-identical no-telemetry posture). productMetrics carries the opt-out +// product-adoption metrics handles (issue #343 follow-up): Shutdown is always +// non-nil (a no-op when disabled) so the caller can defer it unconditionally. type observability struct { cliconfig.HeadlessTelemetryHandles + productMetrics cliconfig.ProductMetricsHandles } // buildObservability wires the OPT-IN OTLP telemetry pipeline for mecatequi via // the shared cliconfig.HeadlessTelemetry helper (the SAME Setup→NewMetrics→ -// WithRole→scoper path mecated wires inline). It keeps appConfig a pure mapping -// over flags + these handles. With no --otlp-* endpoints the helper returns zero -// handles, so the no-telemetry default is byte-identical. The caller owns the -// Shutdown defer (flush-before-exit). -func buildObservability(ctx context.Context, f flags) (observability, error) { +// WithRole→scoper path mecated wires inline), plus the OPT-OUT product-metrics +// pipeline. It keeps appConfig a pure mapping over flags + these handles. With +// no --otlp-* endpoints the OTLP helper returns zero handles, so the +// no-telemetry default is byte-identical. The caller owns the Shutdown defer +// (flush-before-exit). +// +// Product metrics: mecatequi is single-shot/short-lived, so heartbeatCtx is +// context.Background() (nothing to cancel — the process exits right after) +// and heartbeatInterval is 0 (a single immediate fire only, no ticker), +// mirroring mecatequi's own push-before-exit OTLP shape. A build failure (e.g. +// --product-metrics=true forced on with no baked ingest key) is NEVER fatal — +// product metrics are best-effort — so it degrades to a no-op Shutdown rather +// than failing this function's error return (which stays meaningful for the +// OTLP half only). +// +// stderr receives the first-run disclosure notice, written SYNCHRONOUSLY by +// BuildProductMetrics itself — before it starts the heartbeat goroutine and +// before it returns — rather than deferred to a check on the returned +// handles' FirstRun field at flush time (flushTelemetry previously printed +// it there, AFTER already calling Shutdown/flushing the provider: exactly +// the ordering ADR 0338 forbids for opt-out collection). +func buildObservability(ctx context.Context, f flags, diag port.Diagnostics, stderr io.Writer) (observability, error) { h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ ServiceName: "mecatequi", OTLPTraceEndpoint: f.otlpEndpoint, @@ -36,24 +62,113 @@ func buildObservability(ctx context.Context, f flags) (observability, error) { if err != nil { return observability{}, err } - return observability{HeadlessTelemetryHandles: h}, nil + + // Product metrics (opt-out): resolve the effective enabled value via a + // THROWAWAY resolver mirroring mecated's setupProductMetrics / + // mecatui's setupProductMetrics precedent — app.Build's own resolver is + // internal and never exposed back here, so this narrow read-only resolver + // re-parses the same operator settings.yaml (an accepted, negligible + // boot-time cost, same as the other two mains). + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: true, + ExplicitFiles: []string(f.permissionConfigs), + Diagnostics: diag, + }, xdgconfig.OSEnv) + enabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: f.productMetricsSet, + FlagValue: f.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + pm, pmErr := cliconfig.BuildProductMetrics(ctx, context.Background(), enabled, f.productMetricsDryRun, + productmetrics.BinaryMecatequi, buildinfo.BuildID, 0, /* single fire, short-lived */ + productMetricsSnapshot(f), + "" /* no install-id override: local-file mechanism */, diag, + func(notice string) { _, _ = fmt.Fprint(stderr, notice) }) + if pmErr != nil { + // Mirror the existing telemetry-setup-failure posture: a warning, never + // a fatal error — product metrics are best-effort and must not block a + // CI run. + diag.Log(ctx, port.LevelWarn, "product metrics disabled: setup failed", "err", pmErr) + pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} + } + + return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil } -// flushTelemetry runs the telemetry Shutdown (flush) with a bounded ctx so a -// dead collector cannot hang the run. It is safe to call on a zero observability -// (Shutdown is a no-op when telemetry is disabled). A flush failure is logged -// to stderr and never aborts — telemetry is best-effort at exit. -func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { - if obs.Shutdown == nil { - return +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on f — never a +// model id/alias, only whether each feature is configured at all. mecatequi +// has no memory or scheduler flags (single-shot: no per-project memory +// store, no persistent scheduler to opt out of), so Memory/Scheduling stay +// false; Guardrails/MCP/Provider mirror mecated's productMetricsSnapshot. +func productMetricsSnapshot(f flags) productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{ + Guardrails: f.guardrailsModel != "", + MCP: f.mcpServers != nil && len(f.mcpServers.Servers()) > 0, + Provider: cliconfig.ResolveProviderFamily(f.useOpenAI, f.defaultProvider), + Mode: productmetrics.ModeHeadless, } +} + +// flushTelemetry runs the OTLP + product-metrics Shutdown (flush) with a +// bounded ctx so a dead collector cannot hang the run. It is safe to call on a +// zero observability (both Shutdowns are no-ops when telemetry is disabled). A +// flush failure is logged to stderr and never aborts — telemetry is +// best-effort at exit. The product-metrics first-run disclosure notice is +// NOT printed here: buildObservability's notify callback already wrote it, +// synchronously, before the pipeline could ever record/export anything — see +// buildObservability's doc comment. Printing it here instead (after +// Shutdown/flush has already run) is exactly the ordering ADR 0338 forbids. +func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { ctx := context.Background() if timeout > 0 { var cancel context.CancelFunc ctx, cancel = context.WithTimeout(ctx, timeout) defer cancel() } - if err := obs.Shutdown(ctx); err != nil { - _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry flush: %v\n", err) + if obs.Shutdown != nil { + if err := obs.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry flush: %v\n", err) + } + } + if obs.productMetrics.Shutdown != nil { + if err := obs.productMetrics.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: product metrics flush: %v\n", err) + } + } +} + +// productMetricsSink fans obs.Sink (the OTLP sink, nil when telemetry is off) +// together with obs.productMetrics.Sink (nil when product metrics are off) +// into one EventSink. telemetry.NewSink's fanOut.Emit calls every wrapped +// sink unconditionally, so a nil element would panic — both are filtered into +// a non-nil-only slice first (mirroring mecated/mecatui's Task 11/12 pattern). +// With both nil this returns nil, reproducing the byte-identical +// no-telemetry Sink posture exactly. +func productMetricsSink(obs observability) port.EventSink { + var sinks []port.EventSink + if obs.Sink != nil { + sinks = append(sinks, obs.Sink) + } + if obs.productMetrics.Sink != nil { + sinks = append(sinks, obs.productMetrics.Sink) + } + if len(sinks) == 0 { + return nil + } + return telemetry.NewSink(sinks...) +} + +// productMetricsRecorder fans obs.ToolCallRecorder together with +// obs.productMetrics.ToolCallRecorder via cliconfig.TeeToolCallRecorder — but +// ONLY when at least one is non-nil. TeeToolCallRecorder always returns a +// non-nil multiToolCallRecorder interface value even over an all-nil input +// (a typed-nil-slice wrapper, not a nil interface), which would break the +// byte-identical no-telemetry ToolCallRecorder-is-nil posture when both +// sources are off. With both nil this returns nil. +func productMetricsRecorder(obs observability) port.ToolCallRecorder { + if obs.ToolCallRecorder == nil && obs.productMetrics.ToolCallRecorder == nil { + return nil } + return cliconfig.TeeToolCallRecorder(obs.ToolCallRecorder, obs.productMetrics.ToolCallRecorder) } diff --git a/cmd/mecatequi/productmetrics_test.go b/cmd/mecatequi/productmetrics_test.go new file mode 100644 index 0000000000..b12fce45ee --- /dev/null +++ b/cmd/mecatequi/productmetrics_test.go @@ -0,0 +1,51 @@ +package main + +import ( + "flag" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins that mecatequi's +// heartbeat surfaces Guardrails/MCP/Provider from its own flags rather than +// leaving them at their zero values — an empty Provider would heartbeat as +// the invalid provider_configured{family=""}, outside the documented +// anthropic|openai|openrouter|other enum. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + mcpServers := cliconfig.RegisterMCPServerFlag(flag.NewFlagSet("test", flag.ContinueOnError), "") + if err := mcpServers.Set("example=https://mcp.example.com"); err != nil { + t.Fatalf("mcpServers.Set: %v", err) + } + if err := mcpServers.Finalize(); err != nil { + t.Fatalf("mcpServers.Finalize: %v", err) + } + + snap := productMetricsSnapshot(flags{ + guardrailsModel: "claude-haiku", + mcpServers: mcpServers, + defaultProvider: "openrouter/some-model", + }) + if !snap.Guardrails { + t.Error("Guardrails = false, want true (guardrailsModel configured)") + } + if !snap.MCP { + t.Error("MCP = false, want true (an mcp server is configured)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } + if snap.Mode != productmetrics.ModeHeadless { + t.Errorf("Mode = %q, want %q", snap.Mode, productmetrics.ModeHeadless) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value flags still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(flags{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/cmd/mecatequi/telemetry_test.go b/cmd/mecatequi/telemetry_test.go index 4fd88509b8..b51e23a067 100644 --- a/cmd/mecatequi/telemetry_test.go +++ b/cmd/mecatequi/telemetry_test.go @@ -136,7 +136,7 @@ func TestTelemetryDefaultIsNil(t *testing.T) { if err != nil { t.Fatalf("parseFlags: %v", err) } - obs, err := buildObservability(context.Background(), f) + obs, err := buildObservability(context.Background(), f, newDiagnostics(), io.Discard) if err != nil { t.Fatalf("buildObservability: %v", err) } diff --git a/cmd/mecatui/config.go b/cmd/mecatui/config.go index a9f675eb3f..33afd92731 100644 --- a/cmd/mecatui/config.go +++ b/cmd/mecatui/config.go @@ -157,6 +157,20 @@ type config struct { noSteer bool noSteerFlagSet bool + // productMetrics reports anonymous product-adoption metrics to Stacklok + // for the embedded server only (ignored under `mecatui connect`, which + // hosts no local engine). OPT-OUT: ON by default. See the + // --product-metrics flag help text. productMetricsFlagSet records an + // explicit --product-metrics so CLI out-ranks the operator-global + // settings.yaml telemetry.productMetrics.enabled: key (mirrors + // noSteerFlagSet). + productMetrics bool + productMetricsFlagSet bool + // productMetricsDryRun logs every would-be product-metrics observation + // via diag instead of exporting it over OTLP — an audit mode to verify + // the no-PII claim before trusting --product-metrics for real. + productMetricsDryRun bool + // resumeID and resumeLatest select an existing owned main chat for static // startup adoption. They are shared by embedded and connect modes and mutually // exclusive; the first prompt still owns all run-entry attachment/revalidation. @@ -460,6 +474,10 @@ func parseTransportFlags(mode transportMode, out io.Writer, args []string, brows fs.BoolVar(&cfg.noCommands, "no-commands", false, "embedded server only: disable slash-command expansion entirely") fs.StringVar(&cfg.skillsDir, "skills-dir", "", "embedded server only: directory of skill units (/SKILL.md); empty = the conventional dirs (e.g. .claude/skills)") fs.BoolVar(&cfg.noSkills, "no-skills", false, "embedded server only: disable skill discovery (the Skill tool) entirely") + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, MECATL_PRODUCT_METRICS=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") fs.BoolVar(&cfg.perf, "perf", false, "embedded server only: expose the private perf-observability admin surface (/metrics, /debug/pprof, /debug/vars, /debug/flightrecorder) and wire domain metrics into the engine. OFF by default. Empty --perf-addr uses a per-instance UNIX socket. SECURITY: UNAUTHENTICATED — its output can embed prompt text/file paths/goroutine stacks") fs.StringVar(&cfg.perfAddr, "perf-addr", "", "embedded server only: explicit loopback host:port for the --perf admin surface (empty = private per-instance UNIX socket, or ephemeral 127.0.0.1 TCP with --perf-mcp). Use 127.0.0.1:0 for explicit ephemeral TCP. Non-loopback addresses are refused. Only consulted with --perf") @@ -644,6 +662,11 @@ func recordExplicitFlag(f *flag.Flag, cfg *config) { case "no-steer": // Record an explicit --no-steer so CLI out-ranks the settings.yaml steer: key. cfg.noSteerFlagSet = true + case "product-metrics": + // Record an explicit --product-metrics so CLI out-ranks the settings.yaml + // telemetry.productMetrics.enabled: key (ResolveProductMetricsEnabled's + // highest-precedence input). + cfg.productMetricsFlagSet = true case "reasoning-effort": cfg.reasoningEffortFlagSet = true case "default-provider": diff --git a/cmd/mecatui/embed/embed.go b/cmd/mecatui/embed/embed.go index 8b15d6e79d..ab7cd5b29e 100644 --- a/cmd/mecatui/embed/embed.go +++ b/cmd/mecatui/embed/embed.go @@ -39,6 +39,7 @@ import ( "github.com/stacklok/mecatl/internal/adapter/slogdiag" "github.com/stacklok/mecatl/internal/adapter/telemetry" "github.com/stacklok/mecatl/internal/app" + "github.com/stacklok/mecatl/internal/cliconfig" ) // socketName is the fixed socket filename inside the per-process temp directory. @@ -547,9 +548,21 @@ func setupPerf(ctx context.Context, perf PerfConfig, cfg *app.Config, runtimeDir // It returns the slow-turn buffer (nil when the perf MCP server is off) for the // mcpperf Deps wiring. func wirePerfSinks(cfg *app.Config, metrics *telemetry.Metrics, tracing port.EventSink, mountMCP bool) *telemetry.SlowTurnBuffer { + // Capture whatever cfg.Sink/cfg.ToolCallRecorder ALREADY held before either + // field is reassigned below — the product-metrics tap main.go wired onto + // composition BEFORE Start (and thus before setupPerf/wirePerfSinks ran), + // when perf is also enabled. Folding it in here (rather than overwriting) + // keeps the tap alive alongside the perf metrics; a nil oldSink/ + // oldToolCallRecorder (perf-only, no product metrics) is the byte-identical + // prior behaviour. + oldSink := cfg.Sink + oldToolCallRecorder := cfg.ToolCallRecorder mainScoped := metrics.WithRole(telemetry.RoleMain) var slowTurns *telemetry.SlowTurnBuffer sinks := []port.EventSink{mainScoped, tracing} + if oldSink != nil { + sinks = append(sinks, oldSink) + } if mountMCP { // The ring stores scalars only (redaction by shape) and spawns no // goroutine — goleak-clean. Built only when the MCP server will read it. @@ -557,7 +570,7 @@ func wirePerfSinks(cfg *app.Config, metrics *telemetry.Metrics, tracing port.Eve sinks = append(sinks, slowTurns.WithRole(telemetry.RoleMain)) } cfg.Sink = telemetry.NewSink(sinks...) - cfg.ToolCallRecorder = mainScoped + cfg.ToolCallRecorder = cliconfig.TeeToolCallRecorder(mainScoped, oldToolCallRecorder) // Schedule metrics (issue #233, Phase 2b): wire the metrics callback over the // telemetry adapter's EmitSchedule, mirroring MetricsRoleScoper. Schedule // metrics are NOT a role-family; this is a separate schedule-lifecycle diff --git a/cmd/mecatui/embed/perf_internal_test.go b/cmd/mecatui/embed/perf_internal_test.go index 1cea183513..7d340b8de6 100644 --- a/cmd/mecatui/embed/perf_internal_test.go +++ b/cmd/mecatui/embed/perf_internal_test.go @@ -6,7 +6,9 @@ import ( "path/filepath" "strings" "testing" + "time" + "github.com/stacklok/mecatl/engine/session" "github.com/stacklok/mecatl/internal/app" ) @@ -36,6 +38,33 @@ func TestSetupPerfDisabledLeavesTelemetrySeamsNil(t *testing.T) { } } +// TestSetupPerfDisabledPreservesPreexistingTap pins the perf-off half of the +// Task 12 product-metrics-tap preservation contract: with PerfConfig.Enabled +// false (mecatui's default), setupPerf's early return must leave a +// preexisting cfg.Sink/cfg.ToolCallRecorder — set by main.go BEFORE embed.Start +// when the product-metrics pipeline is enabled — completely untouched, so it +// survives byte-identically into the running engine. +func TestSetupPerfDisabledPreservesPreexistingTap(t *testing.T) { + preexistingSink := fakeSink{emitted: new(bool)} + preexistingRecorder := fakeRecorder{recorded: new(bool)} + cfg := app.Config{ + Workspace: t.TempDir(), + Model: "mock", + Sink: preexistingSink, + ToolCallRecorder: preexistingRecorder, + } + + if _, err := setupPerf(context.Background(), PerfConfig{}, &cfg, t.TempDir()); err != nil { + t.Fatalf("setupPerf(disabled): %v", err) + } + if cfg.Sink != preexistingSink { + t.Errorf("perf-off cfg.Sink = %#v, want the untouched preexisting sink", cfg.Sink) + } + if cfg.ToolCallRecorder != preexistingRecorder { + t.Errorf("perf-off cfg.ToolCallRecorder = %#v, want the untouched preexisting recorder", cfg.ToolCallRecorder) + } +} + func TestListenPrivateUnixRejectsUnsafeCollisions(t *testing.T) { for _, kind := range []string{"file", "symlink"} { t.Run(kind, func(t *testing.T) { @@ -68,3 +97,68 @@ func TestSetupPerfRejectsNonLoopbackWithoutMCP(t *testing.T) { t.Fatalf("setupPerf error = %v, want non-loopback refusal", err) } } + +// fakeSink and fakeRecorder are minimal port.EventSink/port.ToolCallRecorder +// probes recording whether they were invoked. +type fakeSink struct{ emitted *bool } + +func (f fakeSink) Emit(context.Context, session.Event) { *f.emitted = true } + +type fakeRecorder struct{ recorded *bool } + +func (f fakeRecorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + *f.recorded = true +} + +// TestWirePerfSinksFoldsInPreexistingTap pins the product-metrics-tap +// preservation contract (Task 12): whatever cfg.Sink/cfg.ToolCallRecorder ALREADY +// held before setupPerf/wirePerfSinks ran (main.go wires the product-metrics +// pipeline onto composition BEFORE embed.Start, i.e. before this ever runs) MUST +// still receive every event/tool-call fan-out AFTER perf wiring — not be +// silently overwritten. Exercises the real setupPerf → wirePerfSinks path with +// PerfConfig.Enabled true (the perf-on case; the perf-off case is covered by +// TestSetupPerfDisabledLeavesTelemetrySeamsNil next to it, which asserts a +// preexisting cfg.Sink is untouched because setupPerf never mutates cfg at all). +func TestWirePerfSinksFoldsInPreexistingTap(t *testing.T) { + var preexistingEmitted, preexistingRecorded bool + preexistingSink := fakeSink{emitted: &preexistingEmitted} + preexistingRecorder := fakeRecorder{recorded: &preexistingRecorded} + + cfg := app.Config{ + Workspace: t.TempDir(), + Model: "mock", + Sink: preexistingSink, + ToolCallRecorder: preexistingRecorder, + } + + // A dedicated short-path temp dir for the admin unix socket: t.TempDir() + // embeds this test's (long) name in the path, which overflows the OS unix + // socket path length limit. + runtimeDir, err := os.MkdirTemp("", "embedperf") + if err != nil { + t.Fatalf("MkdirTemp: %v", err) + } + t.Cleanup(func() { _ = os.RemoveAll(runtimeDir) }) + + ps, err := setupPerf(context.Background(), PerfConfig{Enabled: true}, &cfg, runtimeDir) + if err != nil { + t.Fatalf("setupPerf(enabled): %v", err) + } + t.Cleanup(func() { ps.teardown(context.Background()) }) + + if cfg.Sink == nil { + t.Fatal("perf-on cfg.Sink is nil after wiring") + } + cfg.Sink.Emit(context.Background(), session.Event{}) + if !preexistingEmitted { + t.Error("perf-on wiring dropped the preexisting EventSink instead of folding it in") + } + + if cfg.ToolCallRecorder == nil { + t.Fatal("perf-on cfg.ToolCallRecorder is nil after wiring") + } + cfg.ToolCallRecorder.ToolCall(session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + if !preexistingRecorded { + t.Error("perf-on wiring dropped the preexisting ToolCallRecorder instead of teeing it in") + } +} diff --git a/cmd/mecatui/helpmeta.go b/cmd/mecatui/helpmeta.go index c7048503db..b1a0a29b9d 100644 --- a/cmd/mecatui/helpmeta.go +++ b/cmd/mecatui/helpmeta.go @@ -164,6 +164,8 @@ var flagApplicabilityByFlag = map[string]flagApplicability{ "perf-addr": {group: groupObservability, common: false, local: true, connect: false}, "perf-goroutine-warn-threshold": {group: groupObservability, common: false, local: true, connect: false}, "perf-mcp": {group: groupObservability, common: false, local: true, connect: false}, + "product-metrics": {group: groupObservability, common: true, local: true, connect: false}, + "product-metrics-dry-run": {group: groupObservability, common: false, local: true, connect: false}, // ── Info (meta-flags) ─────────────────────────────────────────────────── "help-all": {group: groupInfo, common: false, local: true, connect: true}, diff --git a/cmd/mecatui/main.go b/cmd/mecatui/main.go index 639d0ea426..a4ec6e3ad3 100644 --- a/cmd/mecatui/main.go +++ b/cmd/mecatui/main.go @@ -46,6 +46,8 @@ import ( "github.com/stacklok/mecatl/internal/adapter/clientauth" "github.com/stacklok/mecatl/internal/adapter/credentialstore" "github.com/stacklok/mecatl/internal/adapter/mcpauthority" + "github.com/stacklok/mecatl/internal/adapter/permconfig" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" "github.com/stacklok/mecatl/internal/adapter/slogdiag" "github.com/stacklok/mecatl/internal/adapter/xdgconfig" "github.com/stacklok/mecatl/internal/app" @@ -935,8 +937,20 @@ func resolveTransport(ctx context.Context, cfg config) (target string, dial clie cfg = applyTrustPrompt(cfg, diag) composition := embeddedConfig(cfg, diag) + // Product metrics (opt-out, Task 12, mirroring mecated's Task 11 wiring): + // resolve the effective enabled value and build the pipeline BEFORE + // embed.Start, so setupPerf/wirePerfSinks (perf is OFF by default) can see + // composition.Sink/ToolCallRecorder already populated and fold them in + // rather than overwrite them when perf IS also enabled. + pm, cancelHeartbeat := setupProductMetrics(ctx, cfg, diag) + if pm.Sink != nil { + composition.Sink = pm.Sink + } + composition.ToolCallRecorder = pm.ToolCallRecorder srv, err := embed.Start(ctx, composition, perfConfig(cfg, perfLogger)) if err != nil { + cancelHeartbeat() + shutdownProductMetrics(pm) _ = diagCloser.Close() return target, client.DialConfig{}, noop, fmt.Errorf("start embedded server: %w", err) } @@ -963,14 +977,94 @@ func resolveTransport(ctx context.Context, cfg config) (target string, dial clie } // The embedded server has no auth/TLS — it is a private UNIX socket dialled // plaintext, the same single-user loopback trust model mecated uses. Cleanup - // closes the server AND the diagnostics log file (a no-op closer for the - // discard/quiet paths), so a clean exit leaks no fd. + // closes the server, stops the product-metrics heartbeat/pipeline, AND closes + // the diagnostics log file (a no-op closer for the discard/quiet paths), so a + // clean exit leaks no fd/goroutine. return srv.Target(), client.DialConfig{Server: srv.Target()}, func() { _ = srv.Close() + cancelHeartbeat() + shutdownProductMetrics(pm) _ = diagCloser.Close() }, nil } +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports for the embedded server, from fields already +// resolved on cfg — never a model id/alias, only whether each feature is +// configured at all. mecatui has no dedicated flags for guardrails/MCP/the +// scheduler on its embedded-server config (those are resolved deeper inside +// app.Build from settings.yaml, not surfaced back to main.go), so +// Guardrails/MCP/Scheduling stay false — but Memory and Provider ARE +// available on cfg and must not be left at their zero values (an empty +// Provider heartbeats as the invalid provider_configured{family=""}, +// outside the documented anthropic|openai|openrouter|other enum). +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + return productmetrics.FeatureSnapshot{ + Memory: cfg.memoryDir != "", + Provider: cliconfig.ResolveProviderFamily(false, cfg.defaultProvider), + Mode: productmetrics.ModeInteractive, + } +} + +// setupProductMetrics resolves the opt-out product-metrics precedence and builds +// the pipeline (Task 12, mirroring mecated's Task 11 setupProductMetrics) for the +// EMBEDDED server only — `mecatui connect` hosts no local engine and never calls +// this. It reads the operator's telemetry.productMetrics.enabled setting via a +// THROWAWAY resolver mirroring embeddedConfig's own hardcoded discovery posture +// (Conventional/ImportClaude: true, no explicit files, no project trust — +// app.Build's own resolver is internal and never exposed back to main.go, the +// same mecated mcplogin.go precedent). settings.yaml is parsed twice at boot +// (once here, once inside app.Build via embed.Start → app.Build); an accepted, +// negligible boot-time cost. +// +// It logs its own build failure via diag (NEVER stderr — stderr would corrupt +// the Bubble Tea alt-screen once the TUI program starts). The first-run +// disclosure notice IS written to stderr — through a notify callback +// BuildProductMetrics itself invokes SYNCHRONOUSLY, before starting the +// heartbeat goroutine and before returning — because at the point this runs +// (resolveTransport, well before tea.NewProgram(...).Run() ever enters the +// alt-screen) stderr is still plain, unbuffered terminal output; a +// diag.Log-routed notice would instead land only in the diagnostics FILE +// (invisible, and dropped entirely under --quiet), defeating ADR 0338's +// visible-disclosure requirement. This also runs BEFORE embed.Start, not +// deferred to a check on the returned handles' FirstRun field after the +// embedded server has started (which left a window where a failed +// embed.Start could flush an already-recording pipeline via +// shutdownProductMetrics without the notice ever having been shown). The +// returned cancel func must be called/deferred unconditionally by the +// caller (Shutdown on the handles is always a safe no-op when +// disabled/errored). +func setupProductMetrics(ctx context.Context, cfg config, diag port.Diagnostics) (cliconfig.ProductMetricsHandles, func()) { + permResolver := permconfig.NewWithEnv(permconfig.Options{ + Conventional: true, + ImportClaude: true, + Diagnostics: diag, + }, xdgconfig.OSEnv) + productMetricsEnabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.productMetricsFlagSet, + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, cfg.productMetricsDryRun, + productmetrics.BinaryMecatui, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot(cfg), "" /* no install-id override: local-file mechanism */, diag, + func(notice string) { fmt.Fprint(os.Stderr, notice) }) + if err != nil { + diag.Log(ctx, port.LevelWarn, "mecatui: product metrics disabled: setup failed", "err", err.Error()) + } + return pm, cancelHeartbeat +} + +// shutdownProductMetrics bounds pm.Shutdown the same way mecated's run() bounds +// its own product-metrics Shutdown defer (5s), for use on every path +// resolveTransport returns (the error path and the success cleanup closure). +func shutdownProductMetrics(pm cliconfig.ProductMetricsHandles) { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = pm.Shutdown(shutdownCtx) +} + // applyTrustPrompt runs the pre-TUI first-encounter workspace-trust gate // (Workspace-Trust Phase 2c) and returns cfg with trustProject set when the // operator (or an already-existing trust grant) trusts the run. It builds the diff --git a/cmd/mecatui/productmetrics_test.go b/cmd/mecatui/productmetrics_test.go new file mode 100644 index 0000000000..9365f2f8df --- /dev/null +++ b/cmd/mecatui/productmetrics_test.go @@ -0,0 +1,37 @@ +package main + +import ( + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// TestProductMetricsSnapshotPopulatesConfiguredFeatures pins that mecatui's +// embedded-server heartbeat surfaces the fields it CAN see on cfg (Memory, +// Provider) rather than leaving them at their zero values — an empty +// Provider would heartbeat as the invalid provider_configured{family=""}, +// outside the documented anthropic|openai|openrouter|other enum. +func TestProductMetricsSnapshotPopulatesConfiguredFeatures(t *testing.T) { + snap := productMetricsSnapshot(config{ + memoryDir: "/tmp/memory", + defaultProvider: "openrouter/some-model", + }) + if !snap.Memory { + t.Error("Memory = false, want true (memoryDir configured)") + } + if snap.Provider != productmetrics.ProviderOpenRouter { + t.Errorf("Provider = %q, want %q", snap.Provider, productmetrics.ProviderOpenRouter) + } + if snap.Mode != productmetrics.ModeInteractive { + t.Errorf("Mode = %q, want %q", snap.Mode, productmetrics.ModeInteractive) + } +} + +// TestProductMetricsSnapshotProviderNeverEmptyByDefault pins that an +// unconfigured, zero-value config still resolves Provider to a real enum +// member (the Anthropic default), never the invalid empty string. +func TestProductMetricsSnapshotProviderNeverEmptyByDefault(t *testing.T) { + if got := productMetricsSnapshot(config{}).Provider; got != productmetrics.ProviderAnthropic { + t.Errorf("Provider = %q, want %q (the default when nothing is configured)", got, productmetrics.ProviderAnthropic) + } +} diff --git a/deploy/helm/mecak8s/chart_test.go b/deploy/helm/mecak8s/chart_test.go index 33b3ebf13c..bfaae11106 100644 --- a/deploy/helm/mecak8s/chart_test.go +++ b/deploy/helm/mecak8s/chart_test.go @@ -101,12 +101,43 @@ func deploymentFromRender(t *testing.T, rendered string) *appsv1.Deployment { return nil } -func workloadEnv(container corev1.Container) []corev1.EnvVar { - return slices.DeleteFunc(slices.Clone(container.Env), func(env corev1.EnvVar) bool { - return env.Name == "MECATL_INSTALLATION_ID" +// installIDEnvVar is the chart-provisioned product-metrics install id +// (install-id-configmap.yaml). It is UNCONDITIONAL — mecak8s is storage-free +// (ADR 0048), so the local-file mechanism the other binaries use would mint a +// fresh id on every pod restart — which means every env-shape assertion below +// is about the OTHER, opt-in variables. appEnv drops it so those assertions +// keep saying exactly what they said before it existed. +const installIDEnvVar = "MECATL_PRODUCT_METRICS_INSTALL_ID" + +// installationIDEnvVar is the OPERATOR-facing `mecatl.installation.id` OTel +// resource attribute (telemetry-configmap.yaml) — a different id, for a +// different audience, than installIDEnvVar above. It is unconditional for the +// same reason, so appEnv drops it too. +const installationIDEnvVar = "MECATL_INSTALLATION_ID" + +// chartOwnedEnvVars are the two unconditional, chart-provisioned environment +// variables every container carries. Both are dropped by appEnv/workloadEnv so +// the pre-existing env-shape assertions below keep asserting exactly what they +// asserted before either id existed. +var chartOwnedEnvVars = []string{installIDEnvVar, installationIDEnvVar} + +// maskInstallID neutralises the per-render uuidv4 so two renders can be +// compared for equality everywhere ELSE. See install-id-configmap.yaml. +func maskInstallID(rendered string) string { + return regexp.MustCompile(`(?m)^ installId: .*$`).ReplaceAllString(rendered, " installId: MASKED") +} + +func appEnv(env []corev1.EnvVar) []corev1.EnvVar { + return slices.DeleteFunc(slices.Clone(env), func(e corev1.EnvVar) bool { + return slices.Contains(chartOwnedEnvVars, e.Name) }) } +// workloadEnv is the container-taking form of appEnv. +func workloadEnv(container corev1.Container) []corev1.EnvVar { + return appEnv(container.Env) +} + func TestMecak8sHelmChart_InstallationID(t *testing.T) { const explicit = "123e4567-e89b-12d3-a456-426614174000" @@ -1431,7 +1462,10 @@ func TestMecak8sHelmChart_ServerTLS(t *testing.T) { if err != nil { t.Fatalf("render tls.enabled=false production values: %v", err) } - if defaultRender != falseRender { + // The chart-owned install id is a fresh uuidv4 on every client-side render + // (lookup finds no cluster ConfigMap), so it differs between two otherwise + // identical renders by design — mask it rather than comparing it. + if maskInstallID(defaultRender) != maskInstallID(falseRender) { t.Fatal("tls.enabled=false changed the default production render") } for _, forbidden := range []string{"--tls-cert", "--tls-key", "/var/run/secrets/tls", "name: tls", "scheme: HTTPS"} { @@ -1628,13 +1662,73 @@ func TestMecak8sHelmChart_KindLiveProviderDisablesMock(t *testing.T) { } } +// TestMecak8sHelmChart_ProductMetricsInstallID pins the storage-free install-id +// contract (ADR 0048): mecak8s keeps no local state, so a per-pod install-id +// file would mint a fresh, never-reused id on every restart. The chart instead +// provisions ONE id per release in a ConfigMap and mounts it as an env var, so +// both halves — the generated value and the reference to it — must render +// unconditionally, on the bare default values as well as a production fixture. +func TestMecak8sHelmChart_ProductMetricsInstallID(t *testing.T) { + // A bare v4 UUID: nothing machine- or user-derived may appear here. + uuidV4 := regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`) + + for _, tc := range []struct { + name, release string + args []string + }{ + {name: "production", release: "production", args: productionArgs()}, + {name: "kind", release: "kind", args: kindFixtureArgs()}, + } { + t.Run(tc.name, func(t *testing.T) { + rendered, err := helm(t, tc.args...) + if err != nil { + t.Fatalf("render: %v", err) + } + + name := tc.release + "-mecak8s-install-id" + cm := configMapFromRender(t, rendered, name) + id := cm.Data["installId"] + if !uuidV4.MatchString(id) { + t.Fatalf("ConfigMap %q installId = %q, want a v4 UUID", name, id) + } + if cm.Labels["app.kubernetes.io/name"] != "mecak8s" { + t.Fatalf("ConfigMap %q labels = %#v, want the chart's standard labels", name, cm.Labels) + } + + container := deploymentFromRender(t, rendered).Spec.Template.Spec.Containers[0] + var got *corev1.EnvVar + for i := range container.Env { + if container.Env[i].Name == installIDEnvVar { + got = &container.Env[i] + } + } + if got == nil { + t.Fatalf("container environment = %#v, want %s", container.Env, installIDEnvVar) + } + // Referenced, never inlined: the id must come from the ConfigMap at + // pod start, so a `helm upgrade` that reuses the existing ConfigMap + // cannot be undone by a stale literal baked into the Deployment. + ref := got.ValueFrom + if got.Value != "" || ref == nil || ref.ConfigMapKeyRef == nil || + ref.ConfigMapKeyRef.Name != name || ref.ConfigMapKeyRef.Key != "installId" { + t.Fatalf("%s = %#v, want a configMapKeyRef to %q/installId", installIDEnvVar, *got, name) + } + }) + } +} + func TestMecak8sHelmChart_ExtraEnv(t *testing.T) { rendered, err := helm(t, productionArgs()...) if err != nil { t.Fatalf("render production values: %v", err) } - if !strings.Contains(rendered, "name: MECATL_INSTALLATION_ID") { - t.Fatal("default render missing chart-managed installation ID environment") + if env := appEnv(deploymentFromRender(t, rendered).Spec.Template.Spec.Containers[0].Env); len(env) != 0 { + t.Fatalf("default render (extraEnv unset) environment = %#v, want only the chart-owned ids", env) + } + for _, want := range []string{"name: " + installIDEnvVar, "name: " + installationIDEnvVar} { + if !strings.Contains(rendered, want) { + t.Fatalf("default render missing chart-managed environment %q", want) + } } args := append(productionArgs(), @@ -1969,12 +2063,15 @@ mcp: env[0].ValueFrom.SecretKeyRef.Key != "bearer-token" { t.Fatalf("static bearer environment = %#v", env) } + // The install-id and telemetry ConfigMaps are both unconditional, so the + // OAuth-profile probe names the MCP settings ConfigMap specifically rather + // than any ConfigMap. if strings.Contains(rendered, "production-mecak8s-mcp") || strings.Contains(rendered, "--permission-config=/etc/mecatl-mcp/settings.yaml") { t.Fatal("static/no-auth MCP render unexpectedly created an OAuth profile") } } -func TestMecak8sHelmChart_MCPNoAuthDoesNotRenderEnv(t *testing.T) { +func TestMecak8sHelmChart_MCPNoAuthRendersNoTokenEnv(t *testing.T) { rendered, err := renderMCPValues(t, ` mcp: servers: diff --git a/deploy/helm/mecak8s/templates/deployment.yaml b/deploy/helm/mecak8s/templates/deployment.yaml index 807b2aa979..034eadbf39 100644 --- a/deploy/helm/mecak8s/templates/deployment.yaml +++ b/deploy/helm/mecak8s/templates/deployment.yaml @@ -202,6 +202,16 @@ spec: - --permission-config=/etc/mecatl-mcp/settings.yaml {{- end }} env: + {{- /* The chart-provisioned, stable-per-release product-metrics + install id. Always present: mecak8s is storage-free (ADR + 0048), so the local-file mechanism the other binaries use + would mint a fresh id on every pod restart. See + install-id-configmap.yaml. */}} + - name: MECATL_PRODUCT_METRICS_INSTALL_ID + valueFrom: + configMapKeyRef: + name: {{ include "mecak8s.fullname" . }}-install-id + key: installId - name: MECATL_INSTALLATION_ID valueFrom: configMapKeyRef: diff --git a/deploy/helm/mecak8s/templates/install-id-configmap.yaml b/deploy/helm/mecak8s/templates/install-id-configmap.yaml new file mode 100644 index 0000000000..67813206e3 --- /dev/null +++ b/deploy/helm/mecak8s/templates/install-id-configmap.yaml @@ -0,0 +1,29 @@ +{{/* +Generates ONE stable install-id for this Helm release, shared by every replica +and preserved across every `helm upgrade` — unlike a per-pod local file, which +mecak8s cannot use at all: it runs storage-free, no PVC (ADR 0048), so every pod +restart would otherwise mint a fresh, never-reused id — the worst-case +cardinality pattern for the product-metrics pipeline this feeds. + +The `lookup` guard is the standard Helm idiom for "generate once, keep stable on +upgrade": when a ConfigMap of this name already exists in the release namespace +its EXISTING value is reused verbatim, so only a genuinely first `helm install` +(or a deliberately deleted ConfigMap) mints a new id. `lookup` returns an empty +map for a client-side render (`helm template`, `--dry-run`), so those renders +show a throwaway id rather than the cluster's — expected, and never written. +*/}} +{{- $existing := lookup "v1" "ConfigMap" .Release.Namespace (printf "%s-install-id" (include "mecak8s.fullname" .)) }} +{{- $installID := "" }} +{{- if and $existing $existing.data }} +{{- $installID = index $existing.data "installId" }} +{{- end }} +{{- if not $installID }} +{{- $installID = uuidv4 }} +{{- end }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "mecak8s.fullname" . }}-install-id + labels: {{- include "mecak8s.labels" . | nindent 4 }} +data: + installId: {{ $installID | quote }} diff --git a/docs/adr/0338-product-metrics.md b/docs/adr/0338-product-metrics.md new file mode 100644 index 0000000000..30e4d4a169 --- /dev/null +++ b/docs/adr/0338-product-metrics.md @@ -0,0 +1,437 @@ +# ADR 0338 — Product (adoption) metrics over OTLP + +- Status: Accepted +- Date: 2026-09-09 +- Scope: `internal/adapter/productmetrics` (new), `internal/cliconfig`, `internal/adapter/permconfig` (new `telemetry:` operator section), `engine/port` (new `RunAwareToolCallRecorder`), `engine/agent` (dispatch wiring), `cmd/mecated`, `cmd/mecatui`, `cmd/mecatequi`, `cmd/mecak8s`, `deploy/helm/mecak8s` +- Supersedes: — +- Superseded by: — + +## Context + +mecatl had no visibility into community adoption: no install counts, no +feature-adoption signal, no aggregate usage depth. Stacklok's infra team +stood up a dedicated, internet-facing OTLP/HTTP metrics ingest at +`https://mecatl.metrics.stacklok.com/v1/metrics` specifically for mecatl binaries +running on infrastructure Stacklok does not control (`stacklok/infra#5604`): +API-key-gated at the edge (`x-mecatl-metrics-key` header, stripped before the +collector) and server-side filtered to accept only metric names matching +`^mecatl\..*`. + +This is a genuinely different concern from mecatl's existing operator-facing +observability (ADR 0018/0045/0098, and the three-channel taxonomy in +ADR 0020: `port.Diagnostics`, `port.ToolCallRecorder`, `port.EventSink`→ +`internal/adapter/telemetry`). That pipeline exists so an *operator* can +observe *their own* deployment, pointed at *their own* collector. Product +metrics is Stacklok observing aggregate, anonymous community adoption across +every install — a different audience, a different destination, and a +different consent model. Reusing any part of the existing three-channel +pipeline as a transport for this data would be a category error: an +operator's own `--otlp-endpoint` must have zero effect on what does or +doesn't reach Stacklok, and enabling/disabling product metrics must have +zero effect on what the operator's own collector receives. The two cannot +be allowed to share a `MeterProvider`, a struct, or a destination, or that +guarantee becomes an implementation accident instead of a structural fact. + +`toolhive-core` (already an mecatl dependency) ships `telemetry/providers`: a +small, already-reviewed OTel SDK-wiring layer (`providers.NewCompositeProvider`) +that builds a `metric.MeterProvider` from an options struct (endpoint, +headers, service name/version, custom resource attributes) without ever +installing it as the process-global provider — the natural building block +for this pipeline's OTLP/HTTP exporter, so mecatl does not hand-roll a third +OTLP wiring implementation next to the two it already has +(`internal/adapter/telemetry/otlp.go`'s inline construction, and this one). + +## Decision + +### A fully independent adapter: `internal/adapter/productmetrics` + +Zero import relationship with `internal/adapter/telemetry`. It owns: + +- Its own `metric.MeterProvider` (`Provider`, wrapping + `toolhive-core/telemetry/providers.CompositeProvider`), built against a + **hardcoded** endpoint (`https://mecatl.metrics.stacklok.com/v1/metrics`) and a + **hardcoded** header key baked into the binary at build time via + `-X …productmetrics.bakedKey=…` (`Taskfile.yml`'s `BUILD_LDFLAGS`) — neither + is operator-configurable, and there is exactly one place this data can go. + An empty baked key (every local/dev/CI-test build) makes `NewProvider` + refuse to construct at all: a non-release build can never accidentally + phone home. +- Its own `Recorder`, implementing `port.EventSink` (`Emit`) and + `port.ToolCallRecorder` (`ToolCall`) — the same two seams + `internal/adapter/telemetry` taps, but a type whose public API cannot + accept a bare free-text `string` anywhere (see "Privacy guard" below). +- Its own heartbeat ticker (`RunHeartbeat`): fires once immediately, then + every `DefaultHeartbeatInterval` (24h) for long-running processes + (`mecated`, `mecatui`, `mecak8s`); a single fire with no ticker + (`interval<=0`) plus flush-before-exit for the short-lived `mecatequi`, + mirroring the OTLP-push-with-flush precedent ADR 0098 already established + for that binary's shape. +- Its own local install-identity file (`installid.go`) — a first-run marker + whose UUID value is now also attached as the `mecatl.install.id` resource + attribute (see "A per-install identifier IS attached" below). + +Composition combines the two independent sinks with a trivial fan-out helper +in `internal/cliconfig` (`BuildProductMetrics`, `TeeToolCallRecorder`) — the +existing Rule-of-Three home for cross-binary telemetry wiring per ADR 0098. +`internal/app` stays import-free of `productmetrics`, exactly as it is of +`telemetry` today. Each `cmd/*/main.go` builds its existing operator +telemetry pipeline unchanged, and — only when product metrics are enabled — +separately constructs a `productmetrics.Recorder` (or, under +`--product-metrics-dry-run`, a `DryRunRecorder`) and tees it in alongside via +`TeeToolCallRecorder` / `internal/adapter/telemetry.NewSink`'s existing +`EventSink` fan-out. + +An operator who disables their own OTLP export still has product metrics +flow (if enabled) to Stacklok; an operator who fully disables product +metrics has zero effect on their own OTLP/Prometheus pipeline. The two +cannot leak into each other because they share no struct, provider, +registry, or destination — only the same two read-only observation points +every consumer of those ports already receives independently, per +composition's existing fan-out discipline. + +### The shipped `mecatl.product.*` catalog + +All instrument names are namespaced under `mecatl.product.*` — passes the +collector's `^mecatl\..*` filter, and is visually/query-wise distinct from +the operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family, so nobody +looking at either series family can mistake one for the other. + +**Resource attributes** (set once per process via `productmetrics.Config`, +not per-metric labels): `service.name=mecatl`, `service.version`, +`mecatl.install.id` (a per-install random UUID — see below), +`mecatl.binary` (one of the closed `Binary` set: +`mecated`/`mecatui`/`mecatequi`/`mecak8s`). + +**A per-install identifier IS attached, after being sized and accepted.** +This pipeline's destination is a Prometheus-remote-write backend +(`stacklok/infra#5604`), where every resource attribute becomes a permanent +label on *every* instrument's time series. A random per-install UUID was +initially left out over exactly this concern — it multiplies active series +by (installs × instrument count), an unbounded-cardinality cost that grows +with adoption. It was reinstated after sizing that cost against the actual +backend (Amazon Managed Service for Prometheus, per `stacklok/infra#5604`): +roughly $1,930/month at 100,000 installs under a worst-case assumption (every +instrument in this catalog, 24/7 uptime), and roughly $650/month under a +more realistic assumption (an interactive CLI tool, ~8h/day active use) — +both well within what the adoption signal an exact per-install breakdown +enables (activation rate, time-to-first-value, weekly retention) is worth to +the product team, and a defensible ceiling rather than a runaway. +`installid.go` still persists a local UUID; its value is now threaded into +`Config.InstallID` and attached as the `mecatl.install.id` resource +attribute. `mecak8s` — storage-free, no PVC (ADR 0048) — cannot use that +same local-file mechanism (every pod restart would mint a fresh, never-reused +id, the worst-case cardinality pattern this pipeline could hit); its Helm +chart instead provisions one stable id per release via a Kubernetes +ConfigMap (see "mecak8s's install-id: a Helm ConfigMap, not a local file" +below). + +**Heartbeat** (`metrics.go`/`heartbeat.go` — on start, then every ~24h for +long-running processes; single fire for `mecatequi`): + +| Instrument | Kind | Attributes | Notes | +|---|---|---|---| +| `mecatl.product.heartbeat` | counter, +1 per fire | none | install/liveness signal | +| `mecatl.product.feature_enabled` | counter, +1 per enabled feature per heartbeat | `feature` | closed `Feature` set — see refinement below | +| `mecatl.product.provider_configured` | counter | `family` | closed `ProviderFamily`: `anthropic`/`openai`/`openrouter`/`other` — never a model id/alias | +| `mecatl.product.deployment_mode` | counter | `mode` | closed `DeploymentMode`: `interactive`/`headless`/`k8s` | + +**Coarse usage** (`metrics.go`/`toolcall.go` — derived from the +`port.EventSink`/`port.ToolCallRecorder` tap, exported on the provider's +normal periodic-reader cadence since these are cumulative counters): + +| Instrument | Kind | Attributes | Fires on | +|---|---|---|---| +| `mecatl.product.sessions_started` | counter | none | `EvSessionInit` | +| `mecatl.product.runs_completed` | counter | `stop` (reuses `session.StopReason`), `had_tool_call` (`true`/`false`: at least one SUCCESSFUL tool call in the run — an errored-only run is `false`) | `EvResult` | +| `mecatl.product.tool_calls` | counter | `category` (a built-in tool's own name from a closed allowlist, `mcp` for anything MCP-server-provided via the structural `mcp__` name prefix, or `other` for anything unrecognized — **never a raw MCP server/tool name**), `outcome` (`success`/`error`) | every `ToolCallRecorder.ToolCall`/`RunAwareToolCallRecorder.ToolCallForRun` | +| `mecatl.product.tokens` | counter | `kind` (`input`/`output`/`cache_read`/`cache_write`/`reasoning`) | `EvResult`'s `Usage` | +| `mecatl.product.subagent_used` | counter | none | `EvSubagentStart` | +| `mecatl.product.team_used` | counter | none | `EvTeamStart` | +| `mecatl.product.run_duration` | histogram (seconds) | none | `EvResult`, when this process observed the run's `EvSessionInit` | +| `mecatl.product.tool_calls_per_run` | histogram (count) | none | every `EvResult` (a tool-less run contributes an honest 0) | +| `mecatl.product.time_to_first_value` | histogram (seconds) | none | at most once per install, ever, on the first run with `had_tool_call=true` and `stop=StopEndTurn` | + + +**Deliberate scope refinement from the original design spec.** The spec's +illustrative heartbeat catalog listed `teams`/`subagents`/`learning` as +feature-enabled flags alongside `memory`/`guardrails`/`mcp`/`scheduling`. +During implementation those three were dropped from the heartbeat: there is +no reliable per-CLI-flag boolean signal for "is Team/Subagent/learning +enabled" the way there is for a settings toggle like memory or guardrails — +those features are always *available*, not gated by a single config +boolean, so a `feature_enabled{feature="teams"}` counter would either always +fire (uninformative) or require inventing a proxy signal (dishonest). The +shipped `productmetrics.Feature` closed set is exactly four values: +`memory`, `guardrails`, `mcp`, `scheduling` (`config.go`). Team/Subagent +adoption is instead captured honestly via the coarse-usage event tap — +`mecatl.product.subagent_used`/`team_used` fire once per run the first time +that delegation family is actually invoked — which is a truer adoption +signal than a static capability flag. This ADR records the *shipped* set; +readers should treat the design spec (`docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md`) +as historical context, not the current catalog. + +Nothing here is free text, a session/run/model identifier, an MCP server/tool +name, a file path, a prompt, or an output. `category` is the one attribute +that emits a name verbatim — but only a built-in tool's own name, drawn from +a maintained closed allowlist in `toolcall.go` (`builtinToolCategories`), with +`"mcp"`/`"other"` as safe catch-alls; every other label value is drawn from a +closed Go-level enum defined in `internal/adapter/productmetrics/config.go` +or reused from `engine/session` (`StopReason`). + +### Opt-out precedence and the operator-tier-only settings gate + +Product metrics are **enabled by default** (opt-out). `internal/cliconfig.ResolveProductMetricsEnabled` +(`productmetrics_config.go`) folds five inputs, highest precedence first: + +1. An explicit CLI flag: `--product-metrics=false` (all four binaries). +2. The `MECATL_PRODUCT_METRICS` environment variable, when it parses as a + `strconv.ParseBool` boolean (`1`/`true`/`0`/`false`, case-insensitive, and + their variants) — a mecatl-specific override, checked BEFORE the generic + `DO_NOT_TRACK` convention so it can win in either direction (e.g. opt + mecatl back IN despite an ambient `DO_NOT_TRACK=1` set for other tools, or + opt mecatl OUT specifically without touching `DO_NOT_TRACK`). Named to + match `--product-metrics`/`telemetry.productMetrics.enabled` exactly — + deliberately NOT `*_DO_NOT_TRACK` or `*_TELEMETRY`: this package's own + `internal/adapter/telemetry` already means something else (the unrelated, + opt-in operator OTLP/Prometheus pipeline), so a same-flavored name here + would misleadingly suggest this variable also touches that pipeline. An + unset/empty/unparseable value falls through to the next tier. +3. The `DO_NOT_TRACK` environment variable set to a truthy value (`""`/`"0"`/ + `"false"`, case-insensitive, are NOT an opt-out) — the cross-ecosystem + convention (donottrack.sh), so the one env var that already opts CI + fleets and dev machines out of *other* tools' telemetry covers mecatl + too, with no mecatl-specific variable to remember. (Originally the ONLY + env-var signal, on the reasoning that "one standard signal beats two + overlapping ones" — `MECATL_PRODUCT_METRICS` was added afterward for + users who want mecatl-specific control independent of their `DO_NOT_TRACK` + setting; the two are complementary, not redundant, since one is a + cross-tool convention and the other is a same-named override matching + this package's own flag/settings vocabulary.) +4. `telemetry.productMetrics.enabled: false` in the **operator-tier** + settings file (`~/.config/mecatl/settings.yaml` + CLI-loaded equivalents) + — `permconfig.Resolver.OperatorProductMetricsEnabled()`. +5. Default: enabled. + +The settings toggle is **operator-tier only**, the same trust boundary as +`guardrails:`/`openrouter:` (AGENTS.md's existing operator-tier-only +precedent). A project-tier `.mecatl/settings.yaml` can neither enable nor +disable it for a user — parsed with the same WARN-and-ignore discipline as +the other operator-only subtrees (`internal/adapter/permconfig/resolve.go`'s +`captureTelemetry` + the project-tier ignore WARN). This direction matters +specifically because it is the *reverse* of most tighten-only project gates: +a project silently overriding a user's own telemetry opt-out (in *either* +direction — forcing it on, or forcing it off to hide activity from an +otherwise-informed operator) would itself be a trust violation, so the +whole subtree is simply inadmissible from a project file. + +### Privacy guard test discipline + +A reflect-based guard test (mirroring the existing `attrRole`/`attrStop` +bounded-label discipline elsewhere in the codebase, and the +`engine/port/diagnostics_imports_test.go` import-tripwire pattern) asserts +that `Recorder`'s entire public API accepts no bare free-text `string` +parameter — only bounded enum types (Go string-alias types with a small +closed value set) and counts/durations. This makes "no PII can flow through +this type" a property CI checks on every future change to the package, not +just a code-review norm that erodes over time. `DryRunRecorder` mirrors +`Recorder`'s restraint field-for-field (it logs the exact same bounded set +`Recorder` would have recorded, via `port.Diagnostics`, never more). + +Additional structural safeguards: exporter failures are silent to the app +(lazy-dial exporter, matching the existing OTLP exporter pattern — a dead +`mecatl.metrics.stacklok.com` never blocks or slows a session); `Shutdown` is +bounded so a hung network path can never delay process exit; the +`MeterProvider` is never installed as the process-global provider (mirrors +`internal/adapter/telemetry`'s own discipline), so it structurally cannot +collide with, or be mistaken for, an operator's own OTel setup. + +### The dry-run audit mode + +`--product-metrics-dry-run` (all four binaries) builds a +`productmetrics.DryRunRecorder` in place of the real OTLP `Recorder` +(`BuildProductMetrics` in `internal/cliconfig/productmetrics.go`): no +install-id read/write, no real provider construction, no real heartbeat +ticker — it fires exactly one representative `Heartbeat` in its own +goroutine (a dry run only needs to demonstrate one sample, not simulate the +full ~24h cadence) and logs every would-be observation through the injected +`port.Diagnostics` instead of exporting it. It logs precisely the same +bounded fields the real `Recorder` would have read (event type, stop +reason, token counts by kind, feature/provider/mode enum values) — never +more. This exists so a skeptical operator can verify the "no PII, here is +exactly what leaves this process" claim by running the binary once, rather +than trusting this document. + +### Why opt-out is defensible here + +Opt-out telemetry is usually a trust liability. Four things together make it +defensible in this specific case, and all four are load-bearing — remove any +one and the balance shifts back toward requiring opt-in: + +1. **A visible, non-blocking first-run disclosure.** The first time a + binary is about to actually send product metrics (telemetry enabled, and + this install's telemetry-id file did not yet exist), it prints + `cliconfig.ProductMetricsDisclosureNotice` once to stderr — what is + collected, that it is on by default, and the exact four ways to turn it + off. It never blocks. An opt-out default with no visible disclosure is + the pattern that burns community trust; this is the whole of that + disclosure, and it is not optional or hidden in a man page. +2. **`DO_NOT_TRACK` support**, so the opt-out is not mecatl-specific + knowledge — anyone who already opts every other tool in their environment + out of telemetry gets mecatl covered for free, with zero new config to + learn. +3. **A reviewable, tested, closed catalog.** Every metric this pipeline can + ever emit is enumerated in this document and enforced by the privacy + guard test above — an operator (or a contributor reading this ADR) does + not have to trust a claim; they can read the closed `Feature`/ + `ProviderFamily`/`DeploymentMode`/token-kind enums and the guard test that + pins them, and know that is the entire surface, mechanically, not just as + of today. +4. **The dry-run self-verification path.** `--product-metrics-dry-run` lets + anyone confirm the catalog claim empirically on their own machine before + trusting `--product-metrics` for real, rather than trusting either the + docs or the code review that produced them. + +### Correlating a tool call to its run: `port.RunAwareToolCallRecorder` + +`had_tool_call`, `tool_calls_per_run`, and `time_to_first_value` all need to +know which *run* a given tool call belongs to. The existing +`port.ToolCallRecorder.ToolCall(id session.SessionID, ...)` only carries a +`SessionID` — a session can span many sequential runs over its lifetime, so a +`SessionID` alone cannot answer "did this run make a tool call". The engine's +other event-sourced callback, `port.EventSink.Emit(ctx, session.Event)`, has +the opposite problem: `session.Event.RunID` is present, but `Emit` never sees +a tool call at all. + +The fix is a new, standalone, OPTIONAL port interface — +`port.RunAwareToolCallRecorder` — added to `engine/port/log.go`, mirroring the +existing `HookApprovalLearner` precedent (`engine/port/hookrunner.go`) exactly: + +```go +type RunAwareToolCallRecorder interface { + ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, + result session.ToolResult, queued, took time.Duration) +} +``` + +`engine/agent/dispatch.go`'s one `ToolCallRecorder` call site type-asserts for +this richer interface and prefers it (passing the enclosing `Run`'s own +`RunID()`, already in scope — no new parameter threading needed anywhere in +the call chain) when a recorder implements it, falling back to the plain +`ToolCall` otherwise. This is purely additive: no existing `ToolCallRecorder` +implementer (the operator telemetry pipeline, `jsonlstore`, `redisstore`) is +affected, and `engine/CHANGELOG.md` records it as `Added` (minor) per +`engine/COMPATIBILITY.md`. + +`internal/adapter/productmetrics.Recorder` implements +`RunAwareToolCallRecorder` (`toolcall.go`): `ToolCallForRun` records the +bounded `category`/`outcome` attributes on `tool_calls` and tallies a +per-run state (`hadToolCall`, `toolCallCount`, `startedAt`) keyed by `RunID` +in a mutex-guarded `perRunTracker`, cleared at the run's `EvResult`. The +category derivation is a closed-set projection, `toolCategory` +(`toolcall.go`): the `mcp__` structural name prefix (`internal/adapter/mcp`'s +`"mcp__" + server + "__" + toolName` construction) buckets every MCP-server +tool under the single literal `"mcp"` with no allowlist needed; every other +name is checked against a maintained `builtinToolCategories` allowlist +(mecatl's own fixed tool catalog), falling back to the literal `"other"` for +anything unrecognized. A hand-maintained allowlist for non-MCP tools was not +the original design (a pure structural-prefix rule was) — it became necessary +because this package's existing privacy-guard test (`bounded_test.go`) +correctly rejects any tool name reaching an attribute value verbatim, and a +prefix-only rule would let an agent-def name, a learned-skill name, or a +future extension-seam name (all potentially operator- or model-derived free +text) leak straight onto an exported counter. The allowlist trades "zero +maintenance for new built-ins" for "structurally impossible to leak" — a new +built-in tool shows up as `"other"` until a line is added, which is visible +and harmless, never a leak. + +### mecak8s's install-id: a Helm ConfigMap, not a local file + +`mecak8s` runs storage-free with no PVC (ADR 0048) — the local-file mechanism +`installid.go` uses for the other three binaries would mint a fresh, +never-reused install id on every pod restart, the worst-case cardinality +pattern this pipeline could hit (every replica of every deployment counted as +a distinct, ever-churning "install"). Its Helm chart +(`deploy/helm/mecak8s/templates/install-id-configmap.yaml`) instead +provisions ONE stable id per release into a `-mecak8s-install-id` +ConfigMap, using Helm's standard `lookup`+`uuidv4` "generate once, keep stable +on upgrade" idiom: a `lookup` against the release namespace for an existing +ConfigMap of that name reuses its `installId` value verbatim on every `helm +upgrade` (and safely re-mints a fresh id if the lookup finds no usable value — +a nil `.data` map or a missing key degrade to "mint a new one", never an +error or an empty string); only a genuinely first `helm install` mints a new +id. `lookup` runs with the Helm client's own credentials at render time, not +the pod's ServiceAccount at runtime, so no RBAC grant was needed. The id is +threaded into the container via a `MECATL_PRODUCT_METRICS_INSTALL_ID` +environment variable (`configMapKeyRef`), which `cmd/mecak8s/observability.go` +reads and passes to `internal/cliconfig.BuildProductMetrics`'s new +`installIDOverride` parameter — non-empty skips `LoadOrCreateInstallIDDefault` +entirely and never reports `FirstRun` (the chart, not the process, owns the +id's lifecycle). The other three binaries pass `""` and keep the local-file +behavior unchanged. Deleting the ConfigMap resets the id, the same as +deleting the local file does for the other three binaries. + +## Consequences + +- A new direct dependency surface: `github.com/stacklok/toolhive-core/telemetry/providers` + (already present at `v0.0.43`) is now also used by `internal/adapter/productmetrics`, + in addition to `internal/adapter/telemetry`'s own inline OTLP construction — + two independent OTLP wiring call sites in the same binary, by design, never + merged. +- A new operator-tier-only YAML subtree (`telemetry.productMetrics.enabled`) + and its own strict-decode schema (`permconfig.TelemetrySection`/ + `ProductMetricsSection`) — a project-tier `telemetry:` block is parsed and + then discarded with a WARN, never silently accepted. +- Four new CLI flags across four binaries (`--product-metrics`, + `--product-metrics-dry-run`), all defaulting to the byte-identical + disabled-pipeline posture when a non-release build carries no baked ingest + key (`bakedKey == ""` refuses `NewProvider` outright) — a local `go build`/ + `go test`/CI build can never phone home regardless of flag state. +- Two small persisted files per install, both under `$XDG_STATE_HOME/mecatl/`: + `telemetry-id` (a bare random v4 UUID, now attached as the + `mecatl.install.id` resource attribute — see the reinstatement rationale + above) and `first-value-recorded` (a bare marker recording whether + `time_to_first_value` has already been sampled). Both are trivially reset + by deleting them, carry no machine or user information, and (on + `mecak8s`) are replaced entirely by the Helm-provisioned ConfigMap — see + "mecak8s's install-id" above. +- ADR-0027 List-1 (resource inventory) is NOT extended: the heartbeat + ticker's lifetime matches the process (owned by the caller's + `heartbeatCtx`, cancelled on shutdown alongside the rest of composition's + shutdown sequence) and the `Provider`/`Recorder` pair holds no cross-run + state a restart would need to rehydrate — this pipeline is stateless + across restart by design, the same reasoning ADR 0098 used for its own + periodic-reader rows. +- Going forward, any NEW metric added to this package must (a) be added to + the catalog table above, (b) use only an existing or newly-defined closed + enum type (never a bare `string`), and (c) pass the privacy guard test + unchanged — the test is the enforcement mechanism, this ADR is the record. +- A per-run cost/dollar metric and any richer per-feature usage counters + (per-MCP-transport type, per-model-family latency) remain explicitly out + of scope, mirroring the deferred #192 cost-metric gap in the operator + pipeline — start narrow, revisit only if a concrete adoption question the + current catalog cannot answer comes up. + +## See also + +- [ADR 0098 — Telemetry for the headless binaries](./0098-headless-telemetry.md) + — the operator-facing OTLP/Prometheus pipeline and the + `internal/cliconfig` Rule-of-Three home this ADR's composition helper + (`BuildProductMetrics`) reuses structurally, but shares NO runtime state + with. Prior art deliberately not reused: this is a new, separate channel, + not a fourth use of the existing telemetry pipeline. +- [ADR 0020 — Diagnostics, audit, and the global-slog ban](./0020-diagnostics.md) + — the three-channel observability taxonomy (`port.Diagnostics`, + `port.ToolCallRecorder`, `port.EventSink`→`internal/adapter/telemetry`) + this decision deliberately does NOT add a fourth row to; product metrics + taps the same two ports (`EventSink`, `ToolCallRecorder`) every existing + consumer already receives independently, but through its own entirely + separate adapter, `MeterProvider`, and destination. +- [ADR 0018 — Performance & observability](./0018-perf-observability.md) +- `user-docs/building/what-you-get/observability.md` — the "Product / + adoption metrics (opt-out)" section added alongside the existing + operator-facing channels. +- `docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md` — the + original design spec this ADR records the shipped outcome of (see the + scope-refinement note above for the one deliberate deviation). +- `stacklok/infra#5604` — the dedicated public OTLP ingest this pipeline + reports to. diff --git a/docs/adr/README.md b/docs/adr/README.md index 2a24e8ccce..103ad68e0d 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -207,6 +207,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0020 — Diagnostics](./0020-diagnostics.md) - [0045 — Explicit-bucket latency histograms (zero-config quantiles on `/metrics`)](./0045-explicit-bucket-latency-histograms.md) - [0098 — Telemetry for the headless binaries (mecatequi, mecak8s)](./0098-headless-telemetry.md) +- [0338 — Product (adoption) metrics over OTLP](./0338-product-metrics.md) ### Governance & trust - [0241 — Canonical untrusted-content fences live in governance](./0241-governance-fence-ownership.md) diff --git a/docs/superpowers/plans/2026-09-08-product-metrics-otel.md b/docs/superpowers/plans/2026-09-08-product-metrics-otel.md new file mode 100644 index 0000000000..a29f4713b4 --- /dev/null +++ b/docs/superpowers/plans/2026-09-08-product-metrics-otel.md @@ -0,0 +1,2389 @@ +# Product Metrics (Opt-Out Adoption Telemetry) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship a fully independent, opt-out-by-default OTLP pipeline that reports bounded adoption/usage counters (install heartbeat, feature flags, coarse session/run/tool-call/token counts) to Stacklok's public metrics collector, with zero coupling to mecatl's existing operator-facing telemetry. + +**Architecture:** A new `internal/adapter/productmetrics` package owns its own OTel `MeterProvider` (built via the already-vendored `toolhive-core/telemetry/providers`, pointed at a hardcoded `https://metrics.stacklok.com/v1/metrics` endpoint with a build-time-baked header key) and its own `port.EventSink`/`port.ToolCallRecorder` implementation that extracts only bounded, closed-enum counts — no tool/session/model names, no free text. Composition (`internal/cliconfig`, then each of the four `cmd/*` mains) tees this second sink alongside the existing operator telemetry sink; the two share no struct, registry, or destination. + +**Tech Stack:** Go 1.26, `go.opentelemetry.io/otel/{metric,sdk/metric}`, `github.com/stacklok/toolhive-core/telemetry/providers` (already a dependency), `github.com/google/uuid` (already an indirect dependency, promoted to direct). + +**Spec:** `docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md` — read it alongside this plan; the plan implements it with one deliberate scope refinement (see Global Constraints). + +## Global Constraints + +- **Opt-out, enabled by default.** Precedence, highest first: CLI flag `--product-metrics` (explicit) > `DO_NOT_TRACK` env var (any non-empty value disables) > operator `settings.yaml` `telemetry.productMetrics.enabled` > default `true`. +- **Operator-tier only.** A project-tier `.mecatl/settings.yaml` `telemetry:` block is parsed but IGNORED with a WARN — mirrors the existing `guardrails:`/`openrouter:`/`posture:` discipline exactly. +- **Zero import relationship** between `internal/adapter/productmetrics` and `internal/adapter/telemetry`. Combined only via a fan-out at the composition edge. +- **No free text ever becomes a metric attribute.** Every label value is a bounded Go closed-set type defined in this package, or an already-existing bounded enum (`session.StopReason`, the token-kind strings). +- **Metric names are namespaced `mecatl.adoption.*`** (passes the collector's `^mecatl\..*` server-side filter; visually distinct from the operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family). +- **Scope refinement vs the spec:** the spec's illustrative catalog listed `teams`/`subagents`/`learning` as heartbeat *feature flags*. This plan drops them from the heartbeat (no reliable, already-verified boolean signal exists at CLI-flag level for "learning enabled", and "teams"/"subagents" are core engine capabilities, not togglable features) and instead captures Subagent/Team **adoption** via the event tap (`mecatl.adoption.subagent_used`/`team_used`, already in the spec's coarse-usage catalog) — the more honest signal. The heartbeat's `feature_enabled` set for this plan is `memory`, `guardrails`, `mcp`, `scheduling` — every one backed by a real, already-verified `cmd/mecated` config field (see Task 11). +- **Exporter failures must never affect the app.** Lazy-dial exporter (matches the existing OTLP exporters in `internal/adapter/telemetry/otlp.go`); bounded `Shutdown` (~3s). +- **`task lint && task test`** must stay green after every task. + +--- + +### Task 1: `productmetrics` package skeleton — closed enums, `FeatureSnapshot`, `Config` + +**Files:** +- Create: `internal/adapter/productmetrics/config.go` +- Test: `internal/adapter/productmetrics/config_test.go` + +**Interfaces:** +- Produces: `type Feature string` + consts `FeatureMemory`, `FeatureGuardrails`, `FeatureMCP`, `FeatureScheduling`; `type ProviderFamily string` + consts `ProviderAnthropic`, `ProviderOpenAI`, `ProviderOpenRouter`, `ProviderOther`; `type DeploymentMode string` + consts `ModeInteractive`, `ModeHeadless`, `ModeK8s`; `type Binary string` + consts `BinaryMecated`, `BinaryMecatui`, `BinaryMecatequi`, `BinaryMecak8s`; `type FeatureSnapshot struct{ Memory, Guardrails, MCP, Scheduling bool; Provider ProviderFamily; Mode DeploymentMode }` with method `func (s FeatureSnapshot) enabled() map[Feature]bool`; `type Config struct{ Binary Binary; Version string; InstallID string }`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import "testing" + +func TestFeatureSnapshotEnabledIsClosedAndBounded(t *testing.T) { + snap := FeatureSnapshot{Memory: true, MCP: true, Provider: ProviderAnthropic, Mode: ModeInteractive} + got := snap.enabled() + + want := map[Feature]bool{ + FeatureMemory: true, + FeatureGuardrails: false, + FeatureMCP: true, + FeatureScheduling: false, + } + if len(got) != len(want) { + t.Fatalf("enabled() returned %d entries, want %d (%v)", len(got), len(want), got) + } + for f, v := range want { + if got[f] != v { + t.Errorf("enabled()[%q] = %v, want %v", f, got[f], v) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestFeatureSnapshotEnabledIsClosedAndBounded -v` +Expected: FAIL — package doesn't exist yet / `enabled` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +// Package productmetrics is a fully independent, opt-out-by-default OTel +// metrics adapter reporting bounded adoption/usage counters to Stacklok's +// public metrics collector. It shares no import, struct, MeterProvider, or +// destination with internal/adapter/telemetry (mecatl's operator-facing +// observability pipeline) — the two are combined only at the composition +// edge (internal/cliconfig), by fanning both into the engine's +// port.EventSink/port.ToolCallRecorder seams. +// +// Every exported type in this package that can become a metric attribute is +// a closed Go string-alias enum. Nothing here carries a session id, model +// id/alias, tool or MCP-server name, file path, or free text. +package productmetrics + +// Feature is the closed set of major toggleable features reported at +// heartbeat time. Never a def/model/tool name — only these four values. +type Feature string + +const ( + FeatureMemory Feature = "memory" + FeatureGuardrails Feature = "guardrails" + FeatureMCP Feature = "mcp" + FeatureScheduling Feature = "scheduling" +) + +// ProviderFamily is the closed set of configured LLM provider families. +// Never a model id or alias. +type ProviderFamily string + +const ( + ProviderAnthropic ProviderFamily = "anthropic" + ProviderOpenAI ProviderFamily = "openai" + ProviderOpenRouter ProviderFamily = "openrouter" + ProviderOther ProviderFamily = "other" +) + +// DeploymentMode is the closed set of process shapes. +type DeploymentMode string + +const ( + ModeInteractive DeploymentMode = "interactive" + ModeHeadless DeploymentMode = "headless" + ModeK8s DeploymentMode = "k8s" +) + +// Binary is the closed set of the four mecatl entry points. +type Binary string + +const ( + BinaryMecated Binary = "mecated" + BinaryMecatui Binary = "mecatui" + BinaryMecatequi Binary = "mecatequi" + BinaryMecak8s Binary = "mecak8s" +) + +// FeatureSnapshot is a closed-shape, read-only snapshot of which major +// features are enabled and which provider family / deployment mode this +// process runs as. It carries no free text and no model id/alias. +type FeatureSnapshot struct { + Memory bool + Guardrails bool + MCP bool + Scheduling bool + Provider ProviderFamily + Mode DeploymentMode +} + +// enabled returns every Feature mapped to whether this snapshot reports it +// enabled. It is the single place Heartbeat iterates, so adding a Feature +// const without adding it here is caught by the exhaustiveness this map +// documents (and by TestFeatureSnapshotEnabledIsClosedAndBounded above). +func (s FeatureSnapshot) enabled() map[Feature]bool { + return map[Feature]bool{ + FeatureMemory: s.Memory, + FeatureGuardrails: s.Guardrails, + FeatureMCP: s.MCP, + FeatureScheduling: s.Scheduling, + } +} + +// Config configures a Provider/Recorder pair for one process. +type Config struct { + // Binary identifies which of the four entry points this process is. + Binary Binary + // Version is the mecatl build version (resource attribute service.version). + Version string + // InstallID is this process's persisted anonymous install identifier. + InstallID string +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestFeatureSnapshotEnabledIsClosedAndBounded -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add internal/adapter/productmetrics/config.go internal/adapter/productmetrics/config_test.go +git commit -m "feat(productmetrics): add closed enums, FeatureSnapshot, Config" +``` + +--- + +### Task 2: Install identity persistence + +**Files:** +- Create: `internal/adapter/productmetrics/installid.go` +- Test: `internal/adapter/productmetrics/installid_test.go` + +**Interfaces:** +- Consumes: `xdgconfig.ResolveEnv`, `xdgconfig.UserStateDir(env)` (`internal/adapter/xdgconfig`, already read: `func UserStateDir(env ResolveEnv) string`). +- Produces: `func LoadOrCreateInstallID(env xdgconfig.ResolveEnv, readFile func(string) ([]byte, error), writeFile func(string, []byte, os.FileMode) error, mkdirAll func(string, os.FileMode) error) (id string, firstRun bool, err error)` and a default-wiring convenience `func LoadOrCreateInstallIDDefault() (id string, firstRun bool, err error)`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "errors" + "os" + "testing" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func TestLoadOrCreateInstallIDCreatesOnFirstRun(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + written := map[string][]byte{} + readFile := func(path string) ([]byte, error) { + data, ok := written[path] + if !ok { + return nil, os.ErrNotExist + } + return data, nil + } + writeFile := func(path string, data []byte, _ os.FileMode) error { + written[path] = data + return nil + } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on an empty store, want true") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + + // Second call reads back the SAME id and reports firstRun=false. + id2, firstRun2, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateInstallID: %v", err) + } + if firstRun2 { + t.Error("firstRun = true on second call, want false") + } + if id2 != id { + t.Errorf("second call returned id %q, want %q (unchanged)", id2, id) + } +} + +func TestLoadOrCreateInstallIDRegeneratesOnCorruptFile(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + readFile := func(string) ([]byte, error) { return []byte("not-a-uuid"), nil } + var gotWrite []byte + writeFile := func(_ string, data []byte, _ os.FileMode) error { gotWrite = data; return nil } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on a corrupt file, want true (treated as absent)") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + if string(gotWrite) != id { + t.Errorf("written content %q != returned id %q", gotWrite, id) + } +} + +func TestLoadOrCreateInstallIDFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + _, _, err := LoadOrCreateInstallID(env, nil, nil, nil) + if err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestLoadOrCreateInstallID -v` +Expected: FAIL — `LoadOrCreateInstallID` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// installIDRelPath is the state-dir-relative path to the persisted anonymous +// install identifier — machine-written runtime state, not human config, so +// it lives under XDG_STATE_HOME (mirroring mecatui's +// $XDG_STATE_HOME/mecatl/mecatui.log precedent), not XDG_CONFIG_HOME. +const installIDRelPath = "mecatl/telemetry-id" + +// LoadOrCreateInstallID reads the persisted install UUID, creating one if +// absent or unparseable. The id is a bare random v4 UUID: it carries no +// machine or user information, and is trivially reset by deleting the file +// (the next opt-in mints a new one). firstRun is true whenever a new id was +// just minted — the caller uses it to decide whether to print the one-time +// disclosure notice. readFile/writeFile/mkdirAll are injected for testing; +// LoadOrCreateInstallIDDefault binds the real filesystem. +func LoadOrCreateInstallID( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (id string, firstRun bool, err error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return "", false, fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + path := filepath.Join(base, installIDRelPath) + + if readFile != nil { + if data, rerr := readFile(path); rerr == nil { + if existing := strings.TrimSpace(string(data)); existing != "" { + if _, perr := uuid.Parse(existing); perr == nil { + return existing, false, nil + } + // Corrupt file: fall through and regenerate. + } + } + } + + fresh := uuid.NewString() + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return "", false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte(fresh), 0o600); werr != nil { + return "", false, fmt.Errorf("productmetrics: write install id: %w", werr) + } + } + return fresh, true, nil +} + +// LoadOrCreateInstallIDDefault binds LoadOrCreateInstallID to the real +// process environment and filesystem. +func LoadOrCreateInstallIDDefault() (id string, firstRun bool, err error) { + return LoadOrCreateInstallID(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestLoadOrCreateInstallID -v` +Expected: PASS (all three tests) + +- [ ] **Step 5: Commit** + +```bash +git add internal/adapter/productmetrics/installid.go internal/adapter/productmetrics/installid_test.go +git commit -m "feat(productmetrics): persist an anonymous random install id" +``` + +--- + +### Task 3: OTLP provider construction via toolhive-core + +**Files:** +- Create: `internal/adapter/productmetrics/provider.go` +- Test: `internal/adapter/productmetrics/provider_test.go` +- Modify: `Taskfile.yml:46` (extend `BUILD_LDFLAGS` with the baked-key `-X` flag) + +**Interfaces:** +- Consumes: `providers.NewCompositeProvider(ctx, ...ProviderOption) (*providers.CompositeProvider, error)`, `providers.WithServiceName`, `WithServiceVersion`, `WithOTLPEndpoint`, `WithMetricsEnabled`, `WithHeaders`, `WithCustomAttributes` (`github.com/stacklok/toolhive-core/telemetry/providers`, already verified); `Config` from Task 1. +- Produces: `type Provider struct{...}`, `func NewProvider(ctx context.Context, cfg Config) (*Provider, error)`, `func (p *Provider) Meter() metric.MeterProvider`, `func (p *Provider) Shutdown(ctx context.Context) error`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "context" + "net/http" + "net/http/httptest" + "testing" +) + +func TestNewProviderFailsClosedWithNoBakedKey(t *testing.T) { + orig := bakedKey + bakedKey = "" + defer func() { bakedKey = orig }() + + _, err := NewProvider(context.Background(), Config{Binary: BinaryMecated, Version: "test"}) + if err == nil { + t.Fatal("expected an error when no ingest key is baked into the build, got nil") + } +} + +func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { + origKey := bakedKey + bakedKey = "test-key" + defer func() { bakedKey = origKey }() + + var gotHeader string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotHeader = r.Header.Get(headerKeyName) + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + origEndpoint := endpoint + endpoint = srv.URL + defer func() { endpoint = origEndpoint }() + + p, err := NewProvider(context.Background(), Config{ + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", + }) + if err != nil { + t.Fatalf("NewProvider: %v", err) + } + defer p.Shutdown(context.Background()) + + meter := p.Meter().Meter("test") + counter, cerr := meter.Int64Counter("mecatl.adoption.test") + if cerr != nil { + t.Fatalf("Int64Counter: %v", cerr) + } + counter.Add(context.Background(), 1) + if err := p.Shutdown(context.Background()); err != nil { + t.Fatalf("Shutdown: %v", err) + } + if gotHeader != "test-key" { + t.Errorf("collector received %s=%q, want %q", headerKeyName, gotHeader, "test-key") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestNewProvider -v` +Expected: FAIL — `NewProvider`/`bakedKey`/`endpoint`/`headerKeyName` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/toolhive-core/telemetry/providers" +) + +// endpoint and headerKeyName are the ONE destination this pipeline can ever +// send to (stacklok/infra#5604): a dedicated, internet-facing OTLP/HTTP +// ingest at metrics.stacklok.com, gated by a single shared key baked into +// the binary. Neither is operator-configurable — an operator's own +// --otlp-endpoint has zero effect on this path, and this path has zero +// effect on the operator's own OTLP/Prometheus pipeline (a completely +// separate MeterProvider, never installed as global). endpoint is a var +// (not a const) so tests can point it at an httptest server. +var ( + endpoint = "https://metrics.stacklok.com/v1/metrics" + headerKeyName = "x-mecatl-metrics-key" +) + +// bakedKey is the shared ingest key baked into the binary at build time via +// `-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey=…` +// (see Taskfile.yml's BUILD_LDFLAGS). An empty key — every local/dev/CI-test +// build that does not set the ldflag — disables the pipeline entirely: +// NewProvider refuses to construct, so a non-release build can never +// accidentally phone home with an invalid or absent key. +var bakedKey = "" + +// Provider wraps the toolhive-core OTLP metrics provider. Its MeterProvider +// is NEVER installed as the process-global provider (mirrors +// internal/adapter/telemetry's own discipline in otlp.go), so it cannot +// collide with an operator's own OTel setup. +type Provider struct { + composite *providers.CompositeProvider +} + +// NewProvider builds the product-metrics MeterProvider for one process. A +// network-unreachable endpoint is NOT an error here — the OTLP/HTTP +// exporter dials lazily on first export, matching the existing exporters in +// internal/adapter/telemetry/otlp.go. +func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { + if bakedKey == "" { + return nil, fmt.Errorf("productmetrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") + } + composite, err := providers.NewCompositeProvider(ctx, + providers.WithServiceName("mecatl"), + providers.WithServiceVersion(cfg.Version), + providers.WithOTLPEndpoint(endpoint), + providers.WithMetricsEnabled(true), + providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), + providers.WithCustomAttributes(map[string]string{ + "mecatl.install.id": cfg.InstallID, + "mecatl.binary": string(cfg.Binary), + }), + ) + if err != nil { + return nil, fmt.Errorf("productmetrics: build provider: %w", err) + } + return &Provider{composite: composite}, nil +} + +// Meter returns the underlying metric.MeterProvider for instrument construction. +func (p *Provider) Meter() metric.MeterProvider { return p.composite.MeterProvider() } + +// Shutdown flushes and stops the provider, bounded by the caller's ctx. +func (p *Provider) Shutdown(ctx context.Context) error { return p.composite.Shutdown(ctx) } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestNewProvider -v` +Expected: PASS (both tests) + +- [ ] **Step 5: Extend the release build's ldflags** + +In `Taskfile.yml`, change line 46 from: + +```yaml + BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}}' +``` + +to: + +```yaml + BUILD_LDFLAGS: '-X github.com/stacklok/mecatl/internal/buildinfo.BuildID={{.BUILD_ID}} -X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey={{.MECATL_METRICS_KEY}}' +``` + +`MECATL_METRICS_KEY` is an env-driven Taskfile variable (empty for every local/dev build — the byte-identical "disabled" posture from Step 3 — set only by the release CI job from a repo secret). Add near the top of `Taskfile.yml` alongside the existing `vars:` block: + +```yaml + MECATL_METRICS_KEY: '{{.MECATL_METRICS_KEY | default ""}}' +``` + +- [ ] **Step 6: Run the full build to confirm it still compiles with an empty key** + +Run: `task build` +Expected: succeeds; `bin/mecated` etc. are built with `bakedKey=""` (unchanged local-dev behavior). + +- [ ] **Step 7: Commit** + +```bash +git add internal/adapter/productmetrics/provider.go internal/adapter/productmetrics/provider_test.go Taskfile.yml +git commit -m "feat(productmetrics): build the OTLP provider via toolhive-core, baked-key gated" +``` + +--- + +### Task 4: `Recorder` — `port.EventSink` (sessions/runs/tokens) + +**Files:** +- Create: `internal/adapter/productmetrics/metrics.go` +- Test: `internal/adapter/productmetrics/metrics_test.go` + +**Interfaces:** +- Consumes: `metric.MeterProvider` (Task 3's `Provider.Meter()`), `session.Event`/`session.EventType`/`session.ResultPayload`/`session.StopReason`/`session.Usage` (`engine/session`, already verified), `port.EventSink` (`engine/port`). +- Produces: `type Recorder struct{...}`, `func NewRecorder(mp metric.MeterProvider) (*Recorder, error)`, `func (r *Recorder) Emit(ctx context.Context, ev session.Event)` (satisfies `port.EventSink`). + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "context" + "testing" + + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +func newTestRecorder(t *testing.T) (*Recorder, *sdkmetric.ManualReader) { + t.Helper() + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + r, err := NewRecorder(mp) + if err != nil { + t.Fatalf("NewRecorder: %v", err) + } + return r, reader +} + +func collect(t *testing.T, reader *sdkmetric.ManualReader) map[string]metricdata.Aggregation { + t.Helper() + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + out := make(map[string]metricdata.Aggregation) + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + out[md.Name] = md.Data + } + } + return out +} + +func sumValue(t *testing.T, agg metricdata.Aggregation) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + var total int64 + for _, dp := range sum.DataPoints { + total += dp.Value + } + return total +} + +func sumPoint(t *testing.T, agg metricdata.Aggregation, key, value string) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + for _, dp := range sum.DataPoints { + if v, present := dp.Attributes.Value(attribute.Key(key)); present && v.AsString() == value { + return dp.Value + } + } + t.Fatalf("no data point with %s=%q", key, value) + return 0 +} + +func TestRecorderEmitSessionsStarted(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + + agg, ok := collect(t, reader)["mecatl.adoption.sessions_started"] + if !ok { + t.Fatal("mecatl.adoption.sessions_started missing") + } + if got := sumValue(t, agg); got != 2 { + t.Errorf("sessions_started = %d, want 2", got) + } +} + +func TestRecorderEmitRunsCompletedByStopReason(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.adoption.runs_completed"] + if got := sumPoint(t, agg, "stop", "end_turn"); got != 1 { + t.Errorf("runs_completed{stop=end_turn} = %d, want 1", got) + } + if got := sumPoint(t, agg, "stop", "error"); got != 1 { + t.Errorf("runs_completed{stop=error} = %d, want 1", got) + } +} + +func TestRecorderEmitTokensByKind(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopEndTurn, + Usage: session.Usage{ + InputTokens: 100, + OutputTokens: 50, + CacheReadTokens: 20, + CacheWriteTokens: 5, + ReasoningTokens: 10, + }, + }, + }) + + agg := collect(t, reader)["mecatl.adoption.tokens"] + cases := map[string]int64{"input": 100, "output": 50, "cache_read": 20, "cache_write": 5, "reasoning": 10} + for kind, want := range cases { + if got := sumPoint(t, agg, "kind", kind); got != want { + t.Errorf("tokens{kind=%s} = %d, want %d", kind, got, want) + } + } +} + +func TestRecorderEmitSubagentAndTeamUsed(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.adoption.subagent_used"]); got != 1 { + t.Errorf("subagent_used = %d, want 1", got) + } + if got := sumValue(t, collected["mecatl.adoption.team_used"]); got != 1 { + t.Errorf("team_used = %d, want 1", got) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderEmit -v` +Expected: FAIL — `Recorder`/`NewRecorder` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// meterName is the instrumentation scope name for this package's meter. +const meterName = "github.com/stacklok/mecatl/internal/adapter/productmetrics" + +// Attribute keys. Every value ever attached under these keys is drawn from a +// bounded closed set (session.StopReason, the fixed token-kind strings, or +// this package's own Feature/ProviderFamily/DeploymentMode enums) — never a +// session id, model id, tool name, or free text. +const ( + attrStop = "stop" + attrKind = "kind" + attrFeature = "feature" + attrProvider = "family" + attrMode = "mode" +) + +// Recorder is the product-metrics adapter: it implements port.EventSink +// (this file) and port.ToolCallRecorder (toolcall.go), deriving ONLY the +// bounded counts in the design's catalog. It never reads a tool name, +// session id, model id, or any free-text field. +type Recorder struct { + heartbeat metric.Int64Counter + featureEnabled metric.Int64Counter + providerConfig metric.Int64Counter + deploymentMode metric.Int64Counter + sessionsStarted metric.Int64Counter + runsCompleted metric.Int64Counter + toolCalls metric.Int64Counter + tokens metric.Int64Counter + subagentUsed metric.Int64Counter + teamUsed metric.Int64Counter +} + +// Compile-time interface checks. +var ( + _ port.EventSink = (*Recorder)(nil) + _ port.ToolCallRecorder = (*Recorder)(nil) +) + +// NewRecorder constructs every instrument from the given MeterProvider. It +// returns an error if any instrument fails to construct — the OTel meter API +// is fallible. +func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { + meter := mp.Meter(meterName) + r := &Recorder{} + var err error + + if r.heartbeat, err = meter.Int64Counter("mecatl.adoption.heartbeat", + metric.WithDescription("Process liveness heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: heartbeat counter: %w", err) + } + if r.featureEnabled, err = meter.Int64Counter("mecatl.adoption.feature_enabled", + metric.WithDescription("Major feature enabled, by closed feature name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: feature_enabled counter: %w", err) + } + if r.providerConfig, err = meter.Int64Counter("mecatl.adoption.provider_configured", + metric.WithDescription("Configured LLM provider family, by closed family name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: provider_configured counter: %w", err) + } + if r.deploymentMode, err = meter.Int64Counter("mecatl.adoption.deployment_mode", + metric.WithDescription("Process deployment mode, by closed mode name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: deployment_mode counter: %w", err) + } + if r.sessionsStarted, err = meter.Int64Counter("mecatl.adoption.sessions_started", + metric.WithDescription("Total sessions started.")); err != nil { + return nil, fmt.Errorf("productmetrics: sessions_started counter: %w", err) + } + if r.runsCompleted, err = meter.Int64Counter("mecatl.adoption.runs_completed", + metric.WithDescription("Total runs completed, by bounded stop reason.")); err != nil { + return nil, fmt.Errorf("productmetrics: runs_completed counter: %w", err) + } + if r.toolCalls, err = meter.Int64Counter("mecatl.adoption.tool_calls", + metric.WithDescription("Total tool calls executed (no tool identity attached).")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls counter: %w", err) + } + if r.tokens, err = meter.Int64Counter("mecatl.adoption.tokens", + metric.WithDescription("Total tokens accounted, by bounded kind."), + metric.WithUnit("{token}")); err != nil { + return nil, fmt.Errorf("productmetrics: tokens counter: %w", err) + } + if r.subagentUsed, err = meter.Int64Counter("mecatl.adoption.subagent_used", + metric.WithDescription("Runs that used the Subagent delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: subagent_used counter: %w", err) + } + if r.teamUsed, err = meter.Int64Counter("mecatl.adoption.team_used", + metric.WithDescription("Runs that used the Team delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: team_used counter: %w", err) + } + return r, nil +} + +// Emit derives coarse, bounded counts from a single domain Event. It reads +// ONLY ev.Type, ev.Result.Stop, and ev.Result.Usage — never a session id, +// model id/alias, tool name, or any free-text field (ev.Result.Text/Error +// are never touched). +func (r *Recorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + r.sessionsStarted.Add(ctx, 1) + case session.EvResult: + r.recordResult(ctx, ev.Result) + case session.EvSubagentStart: + r.subagentUsed.Add(ctx, 1) + case session.EvTeamStart: + r.teamUsed.Add(ctx, 1) + } +} + +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload) { + if res == nil { + r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(session.StopNone)))) + return + } + r.runsCompleted.Add(ctx, 1, metric.WithAttributes(attribute.String(attrStop, string(res.Stop)))) + u := res.Usage + r.tokens.Add(ctx, int64(u.InputTokens), metric.WithAttributes(attribute.String(attrKind, "input"))) + r.tokens.Add(ctx, int64(u.OutputTokens), metric.WithAttributes(attribute.String(attrKind, "output"))) + r.tokens.Add(ctx, int64(u.CacheReadTokens), metric.WithAttributes(attribute.String(attrKind, "cache_read"))) + r.tokens.Add(ctx, int64(u.CacheWriteTokens), metric.WithAttributes(attribute.String(attrKind, "cache_write"))) + r.tokens.Add(ctx, int64(u.ReasoningTokens), metric.WithAttributes(attribute.String(attrKind, "reasoning"))) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderEmit -v` +Expected: PASS (all four tests) + +- [ ] **Step 5: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/metrics_test.go +git commit -m "feat(productmetrics): Recorder.Emit derives bounded session/run/token counts" +``` + +--- + +### Task 5: `Recorder` — `port.ToolCallRecorder` and heartbeat + +**Files:** +- Create: `internal/adapter/productmetrics/toolcall.go` +- Create: `internal/adapter/productmetrics/heartbeat.go` +- Test: `internal/adapter/productmetrics/toolcall_test.go` +- Test: `internal/adapter/productmetrics/heartbeat_test.go` + +**Interfaces:** +- Consumes: `port.ToolCallRecorder.ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration)` (verified signature); `Recorder` from Task 4; `FeatureSnapshot` from Task 1. +- Produces: `func (r *Recorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration)`; `func (r *Recorder) Heartbeat(snap FeatureSnapshot)`; `const DefaultHeartbeatInterval = 24 * time.Hour`; `func RunHeartbeat(ctx context.Context, r *Recorder, interval time.Duration, snap FeatureSnapshot)`. + +- [ ] **Step 1: Write the failing test (ToolCall)** + +```go +package productmetrics + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +func TestRecorderToolCallCountsWithoutIdentity(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCall( + session.SessionID("sensitive-session-id"), + session.ToolCall{Name: "read_secret_file"}, + session.ToolResult{Content: "super secret content", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.ToolCall(session.SessionID("other"), session.ToolCall{Name: "another_tool"}, session.ToolResult{}, 0, 0) + + agg := collect(t, reader)["mecatl.adoption.tool_calls"] + if got := sumValue(t, agg); got != 2 { + t.Errorf("tool_calls = %d, want 2", got) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderToolCallCountsWithoutIdentity -v` +Expected: FAIL — `Recorder.ToolCall` undefined. + +- [ ] **Step 3: Write minimal implementation (toolcall.go)** + +```go +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +// ToolCall records ONLY that a tool call happened — no tool name, no +// session id, no result content, no duration. It satisfies +// port.ToolCallRecorder. The three typed parameters it ignores (id, call, +// result) are accepted only because the port's signature requires them; not +// one of their fields is ever read. +func (r *Recorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + r.toolCalls.Add(context.Background(), 1) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderToolCallCountsWithoutIdentity -v` +Expected: PASS + +- [ ] **Step 5: Write the failing test (Heartbeat)** + +```go +package productmetrics + +import ( + "context" + "testing" + "time" +) + +func TestRecorderHeartbeatRecordsClosedLabelsOnly(t *testing.T) { + r, reader := newTestRecorder(t) + r.Heartbeat(FeatureSnapshot{ + Memory: true, MCP: true, + Provider: ProviderAnthropic, Mode: ModeInteractive, + }) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.adoption.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want 1", got) + } + featureAgg := collected["mecatl.adoption.feature_enabled"] + if got := sumPoint(t, featureAgg, "feature", "memory"); got != 1 { + t.Errorf("feature_enabled{feature=memory} = %d, want 1", got) + } + if got := sumPoint(t, featureAgg, "feature", "mcp"); got != 1 { + t.Errorf("feature_enabled{feature=mcp} = %d, want 1", got) + } + // guardrails/scheduling were false in the snapshot: TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent + // (Task 6) is the exhaustive "no other data point exists" check; this test + // only asserts the enabled ones are present with the right value. + if got := sumPoint(t, collected["mecatl.adoption.provider_configured"], "family", "anthropic"); got != 1 { + t.Errorf("provider_configured{family=anthropic} = %d, want 1", got) + } + if got := sumPoint(t, collected["mecatl.adoption.deployment_mode"], "mode", "interactive"); got != 1 { + t.Errorf("deployment_mode{mode=interactive} = %d, want 1", got) + } +} + +func TestRunHeartbeatFiresImmediatelyThenStopsOnCtxDone(t *testing.T) { + r, reader := newTestRecorder(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancelled BEFORE RunHeartbeat: only the immediate fire happens. + + RunHeartbeat(ctx, r, time.Hour, FeatureSnapshot{Mode: ModeHeadless}) + + if got := sumValue(t, collect(t, reader)["mecatl.adoption.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want exactly 1 (immediate fire only)", got) + } +} +``` + +- [ ] **Step 6: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderHeartbeat -v` and `-run TestRunHeartbeat` +Expected: FAIL — `Heartbeat`/`RunHeartbeat` undefined. + +- [ ] **Step 7: Write minimal implementation (heartbeat.go)** + +```go +package productmetrics + +import ( + "context" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// DefaultHeartbeatInterval is the steady-state heartbeat cadence for +// long-running processes (mecated, mecatui, mecak8s). mecatequi (short-lived) +// passes 0 — a single immediate fire only, no ticker. +const DefaultHeartbeatInterval = 24 * time.Hour + +// Heartbeat records the periodic liveness + feature/provider/mode signal. +// Every attribute value comes from the closed Feature/ProviderFamily/ +// DeploymentMode enums — never a def/model/tool name. +func (r *Recorder) Heartbeat(snap FeatureSnapshot) { + ctx := context.Background() + r.heartbeat.Add(ctx, 1) + for feature, on := range snap.enabled() { + if on { + r.featureEnabled.Add(ctx, 1, metric.WithAttributes(attribute.String(attrFeature, string(feature)))) + } + } + r.providerConfig.Add(ctx, 1, metric.WithAttributes(attribute.String(attrProvider, string(snap.Provider)))) + r.deploymentMode.Add(ctx, 1, metric.WithAttributes(attribute.String(attrMode, string(snap.Mode)))) +} + +// RunHeartbeat fires one heartbeat immediately, then one every interval, +// until ctx is done. interval<=0 disables the ticker (a single fire only — +// mecatequi's shape). Meant to run in its own goroutine, owned by the +// caller (composition), which cancels ctx on shutdown. +func RunHeartbeat(ctx context.Context, r *Recorder, interval time.Duration, snap FeatureSnapshot) { + r.Heartbeat(snap) + if interval <= 0 { + return + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + r.Heartbeat(snap) + } + } +} +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd internal/adapter/productmetrics && go test ./... -v` +Expected: PASS (every test in the package so far) + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/toolcall.go internal/adapter/productmetrics/heartbeat.go \ + internal/adapter/productmetrics/toolcall_test.go internal/adapter/productmetrics/heartbeat_test.go +git commit -m "feat(productmetrics): ToolCall counting and the heartbeat ticker" +``` + +--- + +### Task 6: Privacy guard test — exhaustive bounded-attribute check + +**Files:** +- Create: `internal/adapter/productmetrics/bounded_test.go` + +**Interfaces:** +- Consumes: everything from Tasks 1, 4, 5 (`Recorder`, `FeatureSnapshot`, closed enums). +- Produces: no new production code — a test-only safety net that fails CI the moment a future change attaches an unbounded attribute or an unlisted key. + +This test is the automated version of the design doc's privacy invariant: it drives the Recorder with values chosen to look like they'd leak something sensitive if the code were wrong (a "secret"-looking session id, a suspicious tool name, free-text tool output, an odd stop reason), then asserts across *every* collected data point of *every* instrument that (a) every attribute key is in a fixed allowlist, and (b) no attribute value or metric name contains any of the injected "sensitive" substrings. + +- [ ] **Step 1: Write the test** + +```go +package productmetrics + +import ( + "context" + "strings" + "testing" + "time" + + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +// allowedAttributeKeys is the COMPLETE set of attribute keys any instrument +// in this package may ever carry. A future change that attaches a new label +// must add it here explicitly — the same "closed set is a reviewed +// decision" discipline as internal/adapter/telemetry's attrRole. +var allowedAttributeKeys = map[string]bool{ + attrStop: true, + attrKind: true, + attrFeature: true, + attrProvider: true, + attrMode: true, +} + +// sensitiveMarkers are strings injected into every field the Recorder must +// NEVER read. If any of these ever shows up in a collected metric name or +// attribute value, something started reading a field it shouldn't. +var sensitiveMarkers = []string{ + "sensitive-session-id-marker", + "secret-tool-name-marker", + "secret-tool-content-marker", + "secret-error-text-marker", +} + +func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T) { + r, reader := newTestRecorder(t) + + // Drive every observation path with deliberately sensitive-looking data. + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopError, + Text: "sensitive-session-id-marker should never be read", + Error: "secret-error-text-marker: connection to 10.0.0.5 failed", + Usage: session.Usage{InputTokens: 1, OutputTokens: 1}, + }, + }) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + r.ToolCall( + session.SessionID("sensitive-session-id-marker"), + session.ToolCall{Name: "secret-tool-name-marker"}, + session.ToolResult{Content: "secret-tool-content-marker", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.Heartbeat(FeatureSnapshot{ + Memory: true, Guardrails: true, MCP: true, Scheduling: true, + Provider: ProviderOther, Mode: ModeK8s, + }) + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + assertNoSensitiveSubstring(t, md.Name) + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("metric %s: aggregation is %T, want Sum[int64]", md.Name, md.Data) + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + key := string(kv.Key) + if !allowedAttributeKeys[key] { + t.Errorf("metric %s carries attribute key %q, not in allowedAttributeKeys", md.Name, key) + } + assertNoSensitiveSubstring(t, kv.Value.AsString()) + } + } + // mecatl.adoption.tool_calls carries NO attributes at all — the + // strongest form of "no tool identity ever attaches." + if md.Name == "mecatl.adoption.tool_calls" { + for _, dp := range sum.DataPoints { + if dp.Attributes.Len() != 0 { + t.Errorf("mecatl.adoption.tool_calls data point carries %d attributes, want 0: %v", + dp.Attributes.Len(), dp.Attributes) + } + } + } + } + } +} + +func assertNoSensitiveSubstring(t *testing.T, s string) { + t.Helper() + for _, marker := range sensitiveMarkers { + if strings.Contains(s, marker) { + t.Errorf("value %q contains sensitive marker %q", s, marker) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails or passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent -v` +Expected: PASS immediately (Tasks 4-5's implementation already satisfies it) — this task adds no production code, only the safety net. If it fails, the failure output names exactly which instrument/attribute leaked; fix `metrics.go`/`toolcall.go`/`heartbeat.go` until it passes, never loosen this test. + +- [ ] **Step 3: Commit** + +```bash +git add internal/adapter/productmetrics/bounded_test.go +git commit -m "test(productmetrics): exhaustive guard against unbounded/sensitive attributes" +``` + +--- + +### Task 7: `permconfig` schema — `telemetry:` operator subtree + +**Files:** +- Modify: `internal/adapter/permconfig/schema.go` (add `Telemetry *TelemetrySection` field to the top-level `Config` struct near line 154's `OpenRouter` field, plus the new `TelemetrySection`/`ProductMetricsSection` types near line 1202's `OpenRouterSection`) +- Test: `internal/adapter/permconfig/telemetry_schema_test.go` + +**Interfaces:** +- Produces: `type TelemetrySection struct{ ProductMetrics *ProductMetricsSection }`, `type ProductMetricsSection struct{ Enabled *bool }`, both with strict `UnmarshalYAML` (mirroring `OpenRouterSection`, already verified). + +- [ ] **Step 1: Write the failing test** + +```go +package permconfig + +import "testing" + +func TestParseYAMLTelemetryProductMetricsEnabled(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enabled: false\n") + cfg, err := parseYAML(data) + if err != nil { + t.Fatalf("parseYAML: %v", err) + } + if cfg.Telemetry == nil || cfg.Telemetry.ProductMetrics == nil { + t.Fatal("Telemetry.ProductMetrics is nil") + } + if cfg.Telemetry.ProductMetrics.Enabled == nil || *cfg.Telemetry.ProductMetrics.Enabled != false { + t.Errorf("Enabled = %v, want explicit false", cfg.Telemetry.ProductMetrics.Enabled) + } +} + +func TestParseYAMLTelemetryUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productmetric:\n enabled: false\n") // typo: productmetric + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown telemetry.productmetric key, got nil") + } +} + +func TestParseYAMLTelemetryProductMetricsUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enable: false\n") // typo: enable + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown enable key, got nil") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestParseYAMLTelemetry -v` +Expected: FAIL — `Config.Telemetry` undefined. + +- [ ] **Step 3: Add the field to the top-level `Config` struct** + +In `internal/adapter/permconfig/schema.go`, immediately after the `OpenRouter *OpenRouterSection` field (the one ending around line 154), add: + +```go + // Telemetry holds the OPERATOR-TIER `telemetry:` subtree (opt-out product/ + // adoption metrics). Like OpenRouter/Guardrails/Posture it is honoured ONLY + // from the user-global + CLI tiers; a project-tier file's telemetry: block + // is IGNORED with a WARN (a project repo cannot flip a user's own telemetry + // choice in either direction). Parsed STRICTLY (unknown keys error). A nil + // Telemetry means the key was absent — composition then falls through the + // DO_NOT_TRACK env var and finally defaults to enabled. + Telemetry *TelemetrySection `yaml:"telemetry"` +``` + +- [ ] **Step 4: Add the new section types** + +In `internal/adapter/permconfig/schema.go`, immediately after the `OpenRouterSection`/`OpenRouterModelRoute` block (after the code ending around line 1249), add: + +```go +// TelemetrySection is the `telemetry:` operator-tier YAML subtree: the opt-out +// switch for community/adoption product metrics. Parsed STRICTLY (unknown +// keys error), mirroring OpenRouterSection/GuardrailsSection. +type TelemetrySection struct { + // ProductMetrics is the opt-out product/adoption metrics config. + ProductMetrics *ProductMetricsSection `yaml:"productMetrics"` +} + +func (s *TelemetrySection) strictFields() map[string]any { + return map[string]any{ + "productMetrics": &s.ProductMetrics, + } +} + +// UnmarshalYAML decodes the telemetry: mapping STRICTLY: an unknown key +// (e.g. a typo'd product-metrics:) is a parse error, same discipline as +// openrouter:/guardrails:. +func (s *TelemetrySection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry", s.strictFields()) +} + +// ProductMetricsSection is the `telemetry.productMetrics:` subtree. +type ProductMetricsSection struct { + // Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit + // false: nil = absent (composition falls through to DO_NOT_TRACK then the + // enabled-by-default posture); a non-nil value is honoured exactly. + Enabled *bool `yaml:"enabled"` +} + +func (s *ProductMetricsSection) strictFields() map[string]any { + return map[string]any{ + "enabled": newPermconfigNodePointer(&s.Enabled), + } +} + +// UnmarshalYAML decodes the productMetrics: mapping STRICTLY. +func (s *ProductMetricsSection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry.productMetrics", s.strictFields()) +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestParseYAMLTelemetry -v` +Expected: PASS (all three tests) + +- [ ] **Step 6: Commit** + +```bash +git add internal/adapter/permconfig/schema.go internal/adapter/permconfig/telemetry_schema_test.go +git commit -m "feat(permconfig): add the strict telemetry.productMetrics: operator schema" +``` + +--- + +### Task 8: `permconfig` resolver — operator-tier-only capture + project-tier WARN-ignore + +**Files:** +- Modify: `internal/adapter/permconfig/resolve.go` (Resolver struct field near `operatorOpenRouter` (~line 197), capture method near `captureOpenRouter` (~line 1082), accessor near `OperatorOpenRouter` (~line 388), call sites at both `loadUserRules` locations (~line 907 and ~line 946), and the project-tier WARN block inside `loadProjectRules` (~line 672)) +- Test: `internal/adapter/permconfig/telemetry_resolve_test.go` + +**Interfaces:** +- Produces: `func (r *Resolver) OperatorProductMetricsEnabled() *bool` (nil = absent — the SOLE accessor composition uses). + +- [ ] **Step 1: Write the failing test** + +```go +package permconfig + +import ( + "context" + "testing" + + "github.com/stacklok/mecatl/engine/adapter/memfs" + "github.com/stacklok/mecatl/engine/tool" +) + +func TestOperatorProductMetricsEnabledFromUserGlobal(t *testing.T) { + env := fakeEnv(t, map[string]string{ + "mecatl/settings.yaml": "telemetry:\n productMetrics:\n enabled: false\n", + }) + r := NewResolver(Options{Conventional: true}, env, nil) + got := r.OperatorProductMetricsEnabled() + if got == nil || *got != false { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want explicit false", got) + } +} + +func TestOperatorProductMetricsEnabledAbsentIsNil(t *testing.T) { + env := fakeEnv(t, map[string]string{}) + r := NewResolver(Options{Conventional: true}, env, nil) + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want nil (absent)", got) + } +} + +func TestProjectTierTelemetryBlockIsIgnoredWithWarn(t *testing.T) { + ws := memfs.NewWorkspace(t.TempDir()) + if err := ws.Write(context.Background(), ".mecatl/settings.yaml", + []byte("telemetry:\n productMetrics:\n enabled: false\n")); err != nil { + t.Fatalf("seed project file: %v", err) + } + r := NewResolver(Options{Conventional: true}, fakeEnv(t, nil), nil) + if _, _, err := r.Rules(context.Background(), ws.(tool.WorkspaceReader)); err != nil { + t.Fatalf("Rules: %v", err) + } + // A project-tier telemetry: block must NEVER be captured as operator config. + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Errorf("OperatorProductMetricsEnabled() = %v after a PROJECT-tier block, want nil (project-tier is ignored)", got) + } +} +``` + +Note: `fakeEnv`/`NewResolver`/`r.Rules` signatures above must match this package's existing test helpers exactly — before writing this test, read one existing resolver test (e.g. the file containing `TestOperatorOpenRouterFromUserGlobal`-shaped tests, likely `internal/adapter/permconfig/resolve_test.go` or an `openrouter_test.go` sibling) and copy its exact `fakeEnv`/construction idiom rather than inventing a new one. + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestOperatorProductMetricsEnabled -v` and `-run TestProjectTierTelemetryBlockIsIgnoredWithWarn` +Expected: FAIL — `OperatorProductMetricsEnabled` undefined. + +- [ ] **Step 3: Add the Resolver field** + +In `internal/adapter/permconfig/resolve.go`, immediately after the `operatorOpenRouter *OpenRouterSection` field, add: + +```go + // operatorTelemetry is the OPERATOR-TIER telemetry: subtree, read ONCE at + // construction from the user-global + CLI tiers ONLY (the SOLE capture path + // is captureTelemetry from loadUserRules — mirroring captureOpenRouter). A + // project-tier file's telemetry: block is IGNORED with a WARN in + // loadProjectRules. nil when no operator-tier file carried a telemetry: + // section. CLI (explicit files) out-ranks user-global (first-non-nil keeps + // CLI). + operatorTelemetry *TelemetrySection +``` + +- [ ] **Step 4: Add the capture method** + +Immediately after `captureOpenRouter`, add: + +```go +// captureTelemetry records the FIRST operator-tier telemetry: block seen +// during loadUserRules (CLI files out-rank user-global, so first-non-nil +// keeps CLI). Mirrors captureOpenRouter. +func (r *Resolver) captureTelemetry(s *TelemetrySection) { + if s == nil || r.operatorTelemetry != nil { + return + } + r.operatorTelemetry = s +} +``` + +- [ ] **Step 5: Add the accessor** + +Immediately after `OperatorOpenRouter`, add: + +```go +// OperatorProductMetricsEnabled returns the operator-tier +// telemetry.productMetrics.enabled: value (user-global + CLI only), or nil +// when none was configured. It is the SOLE accessor composition uses to +// read the product-metrics opt-out from config — by construction it never +// returns a project-tier value (a project telemetry: block is ignored with +// a WARN in loadProjectRules). nil-safe. Mirrors OperatorOpenRouter(). +func (r *Resolver) OperatorProductMetricsEnabled() *bool { + if r == nil || r.operatorTelemetry == nil || r.operatorTelemetry.ProductMetrics == nil { + return nil + } + return r.operatorTelemetry.ProductMetrics.Enabled +} +``` + +- [ ] **Step 6: Wire the two `loadUserRules` capture call sites** + +At both locations identified in the Files list (immediately after each existing `r.captureOpenRouter(cfg.OpenRouter)` line — one in the explicit-CLI-files branch, one in the user-global-YAML branch), add: + +```go + // Operator-tier telemetry (opt-out product metrics): same + // first-non-nil-keeps-CLI discipline as openrouter. + r.captureTelemetry(cfg.Telemetry) +``` + +- [ ] **Step 7: Wire the project-tier WARN-ignore in `loadProjectRules`** + +Immediately after the existing `if cfg.OpenRouter != nil { ... }` WARN block inside `loadProjectRules`, add: + +```go + if cfg.Telemetry != nil { + r.diag.Log(context.Background(), port.LevelWarn, + "telemetry: IGNORING a project-tier telemetry: block (operator-tier only — a project repo cannot change a user's own product-metrics opt-out in either direction; set telemetry in your user-global settings.yaml)", + "file", src.path, "root", ws.Root()) + } +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd internal/adapter/permconfig && go test ./... -run TestOperatorProductMetricsEnabled -v` and `-run TestProjectTierTelemetryBlockIsIgnoredWithWarn` +Expected: PASS + +- [ ] **Step 9: Run the full permconfig suite to catch any regression** + +Run: `cd internal/adapter/permconfig && go test ./...` +Expected: PASS + +- [ ] **Step 10: Commit** + +```bash +git add internal/adapter/permconfig/resolve.go internal/adapter/permconfig/telemetry_resolve_test.go +git commit -m "feat(permconfig): resolve the operator-tier telemetry.productMetrics opt-out" +``` + +--- + +### Task 9: `cliconfig` — opt-out precedence + `ToolCallRecorder` fan-out + +**Files:** +- Create: `internal/cliconfig/productmetrics_config.go` +- Test: `internal/cliconfig/productmetrics_config_test.go` + +**Interfaces:** +- Consumes: `port.EventSink`, `port.ToolCallRecorder`, `session.SessionID`/`session.ToolCall`/`session.ToolResult` (`engine/port`, `engine/session`). +- Produces: `type ProductMetricsPrecedence struct{ FlagSet, FlagValue bool; Getenv func(string) string; SettingsEnabled *bool }`, `func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool`, `func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecorder`. + +- [ ] **Step 1: Write the failing test** + +```go +package cliconfig + +import "testing" + +func boolPtr(b bool) *bool { return &b } + +func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { + getenvSet := func(string) string { return "1" } + getenvUnset := func(string) string { return "" } + + cases := []struct { + name string + p ProductMetricsPrecedence + want bool + }{ + {"flag true wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: true, Getenv: getenvSet, SettingsEnabled: boolPtr(false)}, true}, + {"flag false wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: false, Getenv: getenvUnset, SettingsEnabled: boolPtr(true)}, false}, + {"DO_NOT_TRACK disables when no flag", ProductMetricsPrecedence{Getenv: getenvSet, SettingsEnabled: boolPtr(true)}, false}, + {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: boolPtr(false)}, false}, + {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: getenvUnset, SettingsEnabled: nil}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := ResolveProductMetricsEnabled(tc.p); got != tc.want { + t.Errorf("ResolveProductMetricsEnabled(%+v) = %v, want %v", tc.p, got, tc.want) + } + }) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/cliconfig && go test ./... -run TestResolveProductMetricsEnabledPrecedence -v` +Expected: FAIL — `ProductMetricsPrecedence`/`ResolveProductMetricsEnabled` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package cliconfig + +import ( + "os" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// ProductMetricsPrecedence carries the opt-out inputs +// ResolveProductMetricsEnabled folds, highest precedence first: an explicit +// CLI flag, then the DO_NOT_TRACK env var convention (consoledonottrack.com), +// then the operator settings.yaml value, then default-enabled. +type ProductMetricsPrecedence struct { + // FlagSet/FlagValue report whether --product-metrics was explicitly + // passed on the command line and its value. + FlagSet bool + FlagValue bool + // Getenv abstracts os.Getenv for DO_NOT_TRACK / testing. Defaults to + // os.Getenv when nil. + Getenv func(string) string + // SettingsEnabled is permconfig.Resolver.OperatorProductMetricsEnabled() + // — nil when the operator set no telemetry.productMetrics.enabled value. + SettingsEnabled *bool +} + +// ResolveProductMetricsEnabled applies the opt-out precedence documented on +// ProductMetricsPrecedence. Default (nothing set anywhere) is true — product +// metrics are OPT-OUT, not opt-in. +func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { + if p.FlagSet { + return p.FlagValue + } + getenv := p.Getenv + if getenv == nil { + getenv = os.Getenv + } + if getenv("DO_NOT_TRACK") != "" { + return false + } + if p.SettingsEnabled != nil { + return *p.SettingsEnabled + } + return true +} + +// TeeToolCallRecorder combines multiple ToolCallRecorders into one — the +// ToolCallRecorder twin of internal/adapter/telemetry.NewSink's EventSink +// fan-out (no such helper existed before product metrics, because until now +// only one ToolCallRecorder ever observed a given engine). nil entries are +// skipped, so a caller can pass an always-present operator recorder +// alongside an optional product-metrics one without a conditional slice +// build. +func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecorder { + var non []port.ToolCallRecorder + for _, r := range recorders { + if r != nil { + non = append(non, r) + } + } + return multiToolCallRecorder(non) +} + +type multiToolCallRecorder []port.ToolCallRecorder + +func (m multiToolCallRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + for _, r := range m { + r.ToolCall(id, call, result, queued, took) + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/cliconfig && go test ./... -run TestResolveProductMetricsEnabledPrecedence -v` +Expected: PASS (all five subtests) + +- [ ] **Step 5: Add a small test for `TeeToolCallRecorder`** + +```go +func TestTeeToolCallRecorderCallsEveryNonNilRecorder(t *testing.T) { + var calls []string + rec := func(name string) port.ToolCallRecorder { + return recorderFunc(func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + calls = append(calls, name) + }) + } + tee := TeeToolCallRecorder(rec("a"), nil, rec("b")) + tee.ToolCall(session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if len(calls) != 2 || calls[0] != "a" || calls[1] != "b" { + t.Errorf("calls = %v, want [a b] (nil skipped, order preserved)", calls) + } +} + +// recorderFunc adapts a plain func to port.ToolCallRecorder for this test. +type recorderFunc func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) + +func (f recorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f(id, call, result, queued, took) +} +``` + +Add the necessary `"github.com/stacklok/mecatl/engine/port"`, `"github.com/stacklok/mecatl/engine/session"`, and `"time"` imports to the test file if not already present from Step 1. + +- [ ] **Step 6: Run test to verify it passes** + +Run: `cd internal/cliconfig && go test ./... -run TestTeeToolCallRecorderCallsEveryNonNilRecorder -v` +Expected: PASS + +- [ ] **Step 7: Commit** + +```bash +git add internal/cliconfig/productmetrics_config.go internal/cliconfig/productmetrics_config_test.go +git commit -m "feat(cliconfig): product-metrics opt-out precedence + ToolCallRecorder fan-out" +``` + +--- + +### Task 10: `cliconfig` — the `BuildProductMetrics` composition helper + +**Files:** +- Create: `internal/cliconfig/productmetrics.go` +- Test: `internal/cliconfig/productmetrics_test.go` + +**Interfaces:** +- Consumes: everything from `internal/adapter/productmetrics` (Tasks 1-5) and `ResolveProductMetricsEnabled`/`TeeToolCallRecorder` (Task 9). +- Produces: `type ProductMetricsHandles struct{ Sink port.EventSink; ToolCallRecorder port.ToolCallRecorder; Shutdown func(context.Context) error; FirstRun bool }`, `func BuildProductMetrics(ctx, heartbeatCtx context.Context, enabled bool, binary productmetrics.Binary, version string, heartbeatInterval time.Duration, snap productmetrics.FeatureSnapshot) (ProductMetricsHandles, error)`, `const ProductMetricsDisclosureNotice = "..."`. + +- [ ] **Step 1: Write the failing test** + +```go +package cliconfig + +import ( + "context" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=false): %v", err) + } + if h.Sink != nil || h.ToolCallRecorder != nil { + t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder: %+v", h) + } + if h.Shutdown == nil { + t.Fatal("Shutdown must be non-nil even when disabled (a no-op)") + } + if err := h.Shutdown(context.Background()); err != nil { + t.Errorf("no-op Shutdown returned an error: %v", err) + } +} + +func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { + // bakedKey is empty in every non-release build/test — enabling must + // surface the error rather than silently disabling, so a caller notices + // its release build is missing the ldflag. + _, err := BuildProductMetrics(context.Background(), context.Background(), true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}) + if err == nil { + t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/cliconfig && go test ./... -run TestBuildProductMetrics -v` +Expected: FAIL — `BuildProductMetrics` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package cliconfig + +import ( + "context" + "fmt" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// ProductMetricsDisclosureNotice is printed ONCE — the first run after +// product metrics were enabled and this install's telemetry id did not yet +// exist — to stderr, non-blockingly, before any pipeline is built. Opt-out +// telemetry without a visible disclosure is the pattern that burns +// community trust; this is the whole of that disclosure. +const ProductMetricsDisclosureNotice = `mecatl reports anonymous product-adoption metrics (version, OS/arch, which +major features you have enabled, and coarse session/run/tool-call counts — +never a prompt, file path, tool name, or model id) to help Stacklok understand +community adoption. This is on by default. To opt out: pass +--product-metrics=false, set DO_NOT_TRACK=1, or set +telemetry.productMetrics.enabled: false in your settings.yaml. Details: +. +` + +// ProductMetricsHandles bundles the handles a cmd main threads into its +// EventSink/ToolCallRecorder fan-out (via TeeToolCallRecorder / +// internal/adapter/telemetry.NewSink alongside the operator sink) and its +// shutdown defer. Every field is zero-valued when telemetry is disabled. +type ProductMetricsHandles struct { + Sink port.EventSink + ToolCallRecorder port.ToolCallRecorder + // Shutdown flushes + stops the provider. Always non-nil (a no-op when + // disabled), so a caller can defer it unconditionally. + Shutdown func(context.Context) error + // FirstRun is true the first time this install's telemetry id was just + // minted — the caller prints ProductMetricsDisclosureNotice when true. + FirstRun bool +} + +// BuildProductMetrics constructs the full opt-out product-metrics pipeline +// when enabled is true; when false it returns zero handles (the +// byte-identical disabled posture) and no error. heartbeatInterval is +// productmetrics.DefaultHeartbeatInterval for long-running processes, or 0 +// for a single-fire-only short-lived process (mecatequi). heartbeatCtx is +// cancelled by the caller on shutdown to stop the periodic ticker goroutine +// this starts. +func BuildProductMetrics( + ctx, heartbeatCtx context.Context, + enabled bool, + binary productmetrics.Binary, + version string, + heartbeatInterval time.Duration, + snap productmetrics.FeatureSnapshot, +) (ProductMetricsHandles, error) { + noop := func(context.Context) error { return nil } + if !enabled { + return ProductMetricsHandles{Shutdown: noop}, nil + } + + installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + InstallID: installID, + }) + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: provider: %w", err) + } + + recorder, err := productmetrics.NewRecorder(provider.Meter()) + if err != nil { + _ = provider.Shutdown(ctx) + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: recorder: %w", err) + } + + go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) + + return ProductMetricsHandles{ + Sink: recorder, + ToolCallRecorder: recorder, + Shutdown: provider.Shutdown, + FirstRun: firstRun, + }, nil +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/cliconfig && go test ./... -run TestBuildProductMetrics -v` +Expected: PASS (both tests) + +- [ ] **Step 5: Commit** + +```bash +git add internal/cliconfig/productmetrics.go internal/cliconfig/productmetrics_test.go +git commit -m "feat(cliconfig): BuildProductMetrics composition helper + disclosure notice" +``` + +--- + +### Task 11: Wire into `cmd/mecated` + +**Files:** +- Modify: `cmd/mecated/main.go` + +**Interfaces:** +- Consumes: `cliconfig.ResolveProductMetricsEnabled`, `cliconfig.BuildProductMetrics`, `cliconfig.TeeToolCallRecorder` (Tasks 9-10); `internal/adapter/permconfig` resolver's `OperatorProductMetricsEnabled()` (Task 8) — mecated already builds a `permconfig.Resolver` for its permission rules; thread its `OperatorProductMetricsEnabled()` value through to this wiring. +- Produces: a new `--product-metrics` flag, defaulting `true`; the fanned-in sink/recorder feeding the existing `sink`/`mainScoped` variables at lines ~927/949. + +- [ ] **Step 1: Add the flag** + +Near the existing `fs.StringVar(&cfg.otlpEndpoint, "otlp-endpoint", ...)` registration (line 1610), add a new `cfg` field and flag: + +```go + fs.BoolVar(&cfg.productMetrics, "product-metrics", true, + "report anonymous product-adoption metrics to Stacklok (version, OS/arch, enabled features, coarse session/run/tool-call counts — never a prompt, file path, tool name, or model id). ON by default; opt out with --product-metrics=false, DO_NOT_TRACK=1, or telemetry.productMetrics.enabled: false in settings.yaml") +``` + +Add `productMetrics bool` to the `config` struct near the existing `otlpEndpoint string` field. + +- [ ] **Step 2: Compute `cliExplicit["product-metrics"]` (already-established mechanism)** + +No new code needed here: `cmd/mecated/main.go`'s existing `fs.Visit(func(f *flag.Flag) { cfg.cliExplicit[f.Name] = true })` (around line 1877-1879) already marks every explicitly-passed flag by name, so `cfg.cliExplicit["product-metrics"]` is populated for free once Step 1's flag is registered. + +- [ ] **Step 3: Resolve the effective enabled value and build the handles** + +In `run()`, immediately after `obs, err := setupObservability(ctx, cfg, diag)` (line 899) and before its `defer` block, add: + +```go + productMetricsEnabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: cfg.cliExplicit["product-metrics"], + FlagValue: cfg.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + defer cancelHeartbeat() + pm, err := cliconfig.BuildProductMetrics(ctx, heartbeatCtx, productMetricsEnabled, + productmetrics.BinaryMecated, buildinfo.BuildID, productmetrics.DefaultHeartbeatInterval, + productMetricsSnapshot(cfg)) + if err != nil { + slog.Warn("product metrics disabled: setup failed", "err", err) + } else { + defer func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if serr := pm.Shutdown(shutdownCtx); serr != nil { + slog.Warn("product metrics shutdown", "err", serr) + } + }() + if pm.FirstRun { + fmt.Fprint(os.Stderr, cliconfig.ProductMetricsDisclosureNotice) + } + } +``` + +Replace `permResolver` above with whatever local variable name `cmd/mecated/main.go`'s `run()` already binds its constructed `*permconfig.Resolver` to (read the surrounding ~50 lines of `run()` to find the exact name before writing this — it is used to build the engine's permission policy earlier in the same function). + +- [ ] **Step 4: Fan the new sink/recorder into the existing pipeline** + +Change the existing: + +```go + sinks := []port.EventSink{mainScoped, tracing} +``` + +to: + +```go + sinks := []port.EventSink{mainScoped, tracing} + if pm.Sink != nil { + sinks = append(sinks, pm.Sink) + } +``` + +and change the existing: + +```go + composition := appConfig(cfg, sink, mainScoped, roleScoper, obs.metrics, diag) +``` + +to: + +```go + composition := appConfig(cfg, sink, cliconfig.TeeToolCallRecorder(mainScoped, pm.ToolCallRecorder), roleScoper, obs.metrics, diag) +``` + +- [ ] **Step 5: Write `productMetricsSnapshot`** + +Add this helper (near `appConfig` or `setupObservability`): + +```go +// productMetricsSnapshot derives the closed-set FeatureSnapshot the product- +// metrics heartbeat reports, from fields already resolved on cfg — never a +// model id/alias, only whether each feature is configured at all. +func productMetricsSnapshot(cfg config) productmetrics.FeatureSnapshot { + provider := productmetrics.ProviderOther + switch { + case cfg.useOpenAI: + provider = productmetrics.ProviderOpenAI + case strings.Contains(strings.ToLower(cfg.defaultProvider), "openrouter"): + provider = productmetrics.ProviderOpenRouter + case strings.Contains(strings.ToLower(cfg.defaultProvider), "openai"): + provider = productmetrics.ProviderOpenAI + case cfg.defaultProvider == "" || strings.Contains(strings.ToLower(cfg.defaultProvider), "anthropic"): + provider = productmetrics.ProviderAnthropic + } + return productmetrics.FeatureSnapshot{ + Memory: cfg.memoryDir != "", + Guardrails: cfg.guardrailsModel != "", + MCP: cfg.mcpServers != nil && len(cfg.mcpServers.Servers()) > 0, + Scheduling: !cfg.noScheduler, + Provider: provider, + Mode: productmetrics.ModeInteractive, + } +} +``` + +Add the `"strings"` import if not already present, and `"github.com/stacklok/mecatl/internal/adapter/productmetrics"` + `"github.com/stacklok/mecatl/internal/buildinfo"` (for `buildinfo.BuildID`, already used elsewhere in this file per the `BUILD_LDFLAGS` reference) to the import block. + +- [ ] **Step 6: Build and run the existing test suite** + +Run: `task build && cd cmd/mecated && go test ./... -v` +Expected: builds and existing tests pass (this task changes wiring only, no new mecated-level tests are required beyond what already exercises flag parsing/appConfig — if `cmd/mecated` has a flag-parsing golden test, verify it still passes with the new `--product-metrics` flag appearing in its help output). + +- [ ] **Step 7: Manually verify the disabled/default posture** + +Run: `go run ./cmd/mecademo` (unaffected — mecademo doesn't wire telemetry) and `bin/mecated --help 2>&1 | grep product-metrics` to confirm the flag is registered and its help text is legible. + +- [ ] **Step 8: Commit** + +```bash +git add cmd/mecated/main.go +git commit -m "feat(mecated): wire opt-out product metrics alongside operator telemetry" +``` + +--- + +### Task 12: Wire into `cmd/mecatui` + +**Files:** +- Modify: `cmd/mecatui/embed/embed.go` (around the `wirePerfSinks` function, line ~549, and its `telemetry.Setup` call at line ~400) +- Modify: `cmd/mecatui/config.go` or `cmd/mecatui/main.go` (wherever mecatui's top-level flags are registered — read the file first to find the exact flag-registration function name, mirroring Task 11 Step 1's `--product-metrics` flag) + +**Interfaces:** +- Consumes: the same `cliconfig.BuildProductMetrics`/`ResolveProductMetricsEnabled`/`TeeToolCallRecorder` as Task 11. + +- [ ] **Step 1: Read the exact flag-registration and `wirePerfSinks` call site** + +Before writing code, read `cmd/mecatui/embed/embed.go` lines 373-560 (already excerpted above) and the file that registers mecatui's top-level CLI flags (find it via `grep -n "flag.NewFlagSet\|RegisterProviderFlags" cmd/mecatui/*.go`) to confirm the exact local variable/function names this task's diff must anchor to — mecatui's structure was not fully read during planning; it mirrors mecated's shape closely (same `sinks := []port.EventSink{mainScoped, tracing}` idiom at line 552) but names may differ slightly. + +- [ ] **Step 2: Add the `--product-metrics` flag** + +Mirror Task 11 Step 1 exactly, in whichever file registers mecatui's top-level flags. + +- [ ] **Step 3: Build and thread the handles** + +In `wirePerfSinks` (or its caller — whichever holds `cfg *app.Config` and constructs `sinks`), mirror Task 11 Steps 3-4: resolve `productMetricsEnabled`, call `cliconfig.BuildProductMetrics` with `productmetrics.BinaryMecatui` and `productmetrics.ModeInteractive`, append `pm.Sink` to `sinks`, and wrap `cfg.ToolCallRecorder` with `cliconfig.TeeToolCallRecorder`. + +- [ ] **Step 4: Print the disclosure notice** + +Wherever mecatui prints its own existing privacy warning (`docs/usage.md:282` references one — find its call site via `grep -rn "privacy warning" cmd/mecatui/`), print `cliconfig.ProductMetricsDisclosureNotice` alongside it when `pm.FirstRun` is true, so the two disclosures appear together rather than as two unrelated startup messages. + +- [ ] **Step 5: Build and run mecatui's existing tests** + +Run: `task build && cd cmd/mecatui && go test ./... -v` +Expected: builds and passes. + +- [ ] **Step 6: Commit** + +```bash +git add cmd/mecatui/ +git commit -m "feat(mecatui): wire opt-out product metrics alongside operator telemetry" +``` + +--- + +### Task 13: Wire into `cmd/mecatequi` and `cmd/mecak8s` + +**Files:** +- Modify: `cmd/mecatequi/observability.go`, `cmd/mecatequi/flags.go` +- Modify: `cmd/mecak8s/observability.go`, `cmd/mecak8s/flags.go` + +**Interfaces:** +- Consumes: `cliconfig.BuildProductMetrics`/`ResolveProductMetricsEnabled`/`TeeToolCallRecorder`. +- Produces: extends the existing `observability` struct in both files with the product-metrics handles. + +- [ ] **Step 1: Add the `--product-metrics` flag to both `flags.go` files** + +Mirror Task 11 Step 1 in each binary's flag registration (both already register `--otlp-endpoint` etc. per the `HeadlessTelemetryConfig` fields read earlier — add `--product-metrics` alongside them). + +- [ ] **Step 2: Extend `cmd/mecatequi/observability.go`** + +```go +package main + +import ( + "context" + "fmt" + "io" + "time" + + "github.com/stacklok/mecatl/internal/adapter/productmetrics" + "github.com/stacklok/mecatl/internal/cliconfig" +) + +// observability carries the telemetry handles realMain threads into appConfig, +// plus the flush-on-exit Shutdown the main owns. +type observability struct { + cliconfig.HeadlessTelemetryHandles + productMetrics cliconfig.ProductMetricsHandles +} + +func buildObservability(ctx context.Context, f flags, permResolver telemetryOperatorSource) (observability, error) { + h, err := cliconfig.HeadlessTelemetry(ctx, cliconfig.HeadlessTelemetryConfig{ + ServiceName: "mecatequi", + OTLPTraceEndpoint: f.otlpEndpoint, + OTLPTraceProtocol: f.otlpProtocol, + OTLPTraceInsecure: f.otlpInsecure, + OTLPMetricsEndpoint: f.otlpMetricsEndpoint, + OTLPMetricsProtocol: f.otlpMetricsProtocol, + OTLPMetricsInsecure: f.otlpInsecure, + }) + if err != nil { + return observability{}, err + } + + enabled := cliconfig.ResolveProductMetricsEnabled(cliconfig.ProductMetricsPrecedence{ + FlagSet: f.productMetricsSet, + FlagValue: f.productMetrics, + SettingsEnabled: permResolver.OperatorProductMetricsEnabled(), + }) + pm, pmErr := cliconfig.BuildProductMetrics(ctx, ctx, enabled, + productmetrics.BinaryMecatequi, f.version, 0, /* single fire, short-lived */ + productmetrics.FeatureSnapshot{Mode: productmetrics.ModeHeadless}) + if pmErr != nil { + // Mirror the existing telemetry-setup-failure posture: a warning, never + // a fatal error — product metrics are best-effort and must not block a + // CI run. Logged by the caller (realMain already has a diag/logger in + // scope) rather than here, to keep this function's error return + // meaningful for the OTLP half only. + pm = cliconfig.ProductMetricsHandles{Shutdown: func(context.Context) error { return nil }} + } + + return observability{HeadlessTelemetryHandles: h, productMetrics: pm}, nil +} + +func flushTelemetry(stderr io.Writer, obs observability, timeout time.Duration) { + ctx := context.Background() + if timeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, timeout) + defer cancel() + } + if obs.Shutdown != nil { + if err := obs.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: telemetry flush: %v\n", err) + } + } + if obs.productMetrics.Shutdown != nil { + if err := obs.productMetrics.Shutdown(ctx); err != nil { + _, _ = fmt.Fprintf(stderr, "mecatequi: product metrics flush: %v\n", err) + } + } + if obs.productMetrics.FirstRun { + _, _ = fmt.Fprint(stderr, cliconfig.ProductMetricsDisclosureNotice) + } +} +``` + +`telemetryOperatorSource` is a one-method interface (`OperatorProductMetricsEnabled() *bool`) — define it in this file so `observability.go` doesn't need to import `permconfig` directly; pass mecatequi's already-constructed `*permconfig.Resolver` as the argument at the call site (find it via `grep -n "buildObservability(" cmd/mecatequi/*.go` and thread the resolver variable already in scope there). + +- [ ] **Step 3: Thread the product-metrics handles into `appConfig`** + +At mecatequi's `appConfig`-equivalent assembly point (wherever `HeadlessTelemetryHandles.Sink`/`ToolCallRecorder` currently feed `app.Config.Sink`/`ToolCallRecorder`), fan in the product-metrics handles the same way as Task 11 Step 4: + +```go + sink := telemetry.NewSink(obs.Sink, obs.productMetrics.Sink) // telemetry.NewSink already skips nothing — verify it tolerates a nil element, or filter nils first + recorder := cliconfig.TeeToolCallRecorder(obs.ToolCallRecorder, obs.productMetrics.ToolCallRecorder) +``` + +`internal/adapter/telemetry.NewSink`'s `fanOut.Emit` calls every wrapped sink unconditionally — a nil `port.EventSink` element would panic on `.Emit`. Guard it: build the slice with a nil check before calling `NewSink`, exactly like Task 11 Step 4's `if pm.Sink != nil { sinks = append(...) }` pattern, rather than passing a possibly-nil element directly. + +- [ ] **Step 4: Repeat Steps 2-3 for `cmd/mecak8s/observability.go`**, using `productmetrics.BinaryMecak8s`, `productmetrics.ModeK8s`, and `productmetrics.DefaultHeartbeatInterval` (mecak8s is long-running, unlike mecatequi) with `heartbeatCtx` tied to the server's shutdown context rather than the short-lived `ctx`. + +- [ ] **Step 5: Add `productMetrics`/`productMetricsSet` fields + flag registration to both `flags.go` files** + +Mirror Task 11 Step 1 in each. + +- [ ] **Step 6: Build and run both binaries' existing tests** + +Run: `task build && cd cmd/mecatequi && go test ./... -v && cd ../mecak8s && go test ./... -v` +Expected: builds and passes. + +- [ ] **Step 7: Commit** + +```bash +git add cmd/mecatequi/ cmd/mecak8s/ +git commit -m "feat(mecatequi,mecak8s): wire opt-out product metrics via cliconfig" +``` + +--- + +### Task 14: Dry-run / audit recorder + +**Files:** +- Create: `internal/adapter/productmetrics/dryrun.go` +- Test: `internal/adapter/productmetrics/dryrun_test.go` +- Modify: `internal/cliconfig/productmetrics.go` (thread a `dryRun bool` parameter into `BuildProductMetrics`) +- Modify: all four `cmd/*` flag-registration sites from Tasks 11-13 (add `--product-metrics-dry-run`) + +**Interfaces:** +- Produces: `type DryRunRecorder struct{...}` implementing `port.EventSink` + `port.ToolCallRecorder`, logging every would-be observation via `port.Diagnostics` instead of exporting it — `func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder`. + +- [ ] **Step 1: Write the failing test** + +```go +package productmetrics + +import ( + "context" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +type capturingDiag struct { + lines []string +} + +func (c *capturingDiag) Log(_ context.Context, _ port.Level, msg string, args ...any) { + c.lines = append(c.lines, msg) + _ = args +} +func (c *capturingDiag) With(...any) port.Diagnostics { return c } + +func TestDryRunRecorderLogsInsteadOfExporting(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "sensitive-name"}, session.ToolResult{Content: "sensitive-content"}, 0, time.Millisecond) + + if len(diag.lines) != 2 { + t.Fatalf("got %d logged lines, want 2: %v", len(diag.lines), diag.lines) + } + for _, line := range diag.lines { + if contains(line, "sensitive") { + t.Errorf("dry-run log line leaked sensitive content: %q", line) + } + } +} + +func contains(s, substr string) bool { + return len(s) >= len(substr) && (func() bool { + for i := 0; i+len(substr) <= len(s); i++ { + if s[i:i+len(substr)] == substr { + return true + } + } + return false + })() +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestDryRunRecorderLogsInsteadOfExporting -v` +Expected: FAIL — `NewDryRunRecorder` undefined. + +- [ ] **Step 3: Write minimal implementation** + +```go +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// DryRunRecorder implements the same two ports as Recorder but logs every +// would-be observation via port.Diagnostics instead of exporting it over +// OTLP — the --product-metrics-dry-run audit path, so a skeptical operator +// can see exactly what this pipeline would have sent without trusting the +// docs. It logs ONLY the same bounded fields Recorder ever reads (event +// type, stop reason, token kind/amounts) — never a tool name, session id, +// or result content, mirroring Recorder's own restraint exactly. +type DryRunRecorder struct { + diag port.Diagnostics +} + +var ( + _ port.EventSink = (*DryRunRecorder)(nil) + _ port.ToolCallRecorder = (*DryRunRecorder)(nil) +) + +// NewDryRunRecorder builds a DryRunRecorder over the given Diagnostics sink. +func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder { + return &DryRunRecorder{diag: diag} +} + +// Emit logs the bounded event type (and, for EvResult, the stop reason and +// token counts by kind) — the exact same fields Recorder.Emit reads. +func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") + case session.EvResult: + if ev.Result == nil { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record runs_completed{stop=\"\"}+1") + return + } + u := ev.Result.Usage + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record run + tokens", + "stop", string(ev.Result.Stop), + "input_tokens", u.InputTokens, "output_tokens", u.OutputTokens, + "cache_read_tokens", u.CacheReadTokens, "cache_write_tokens", u.CacheWriteTokens, + "reasoning_tokens", u.ReasoningTokens) + case session.EvSubagentStart: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") + case session.EvTeamStart: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record team_used+1") + } +} + +// ToolCall logs only that a call happened — no name, no content, matching +// Recorder.ToolCall's restraint exactly. +func (d *DryRunRecorder) ToolCall(_ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1") +} + +// Heartbeat logs the closed-enum feature/provider/mode signal, matching +// Recorder.Heartbeat's fields exactly. +func (d *DryRunRecorder) Heartbeat(snap FeatureSnapshot) { + enabled := make([]string, 0, 4) + for f, on := range snap.enabled() { + if on { + enabled = append(enabled, string(f)) + } + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record heartbeat", + "features_enabled", enabled, "provider_family", string(snap.Provider), "deployment_mode", string(snap.Mode)) +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestDryRunRecorderLogsInsteadOfExporting -v` +Expected: PASS + +- [ ] **Step 5: Thread a `dryRun` option through `BuildProductMetrics`** + +In `internal/cliconfig/productmetrics.go`, change `BuildProductMetrics`'s signature to accept a `dryRun bool` parameter (after `enabled`), and branch near the top of the function body: + +```go +func BuildProductMetrics( + ctx, heartbeatCtx context.Context, + enabled, dryRun bool, + binary productmetrics.Binary, + version string, + heartbeatInterval time.Duration, + snap productmetrics.FeatureSnapshot, + diag port.Diagnostics, +) (ProductMetricsHandles, error) { + noop := func(context.Context) error { return nil } + if !enabled { + return ProductMetricsHandles{Shutdown: noop}, nil + } + if dryRun { + rec := productmetrics.NewDryRunRecorder(diag) + go func() { + rec.Heartbeat(snap) + // A dry run never persists an install id or starts a real ticker — + // it exists to show ONE representative sample, not to simulate the + // full 24h cadence. + }() + return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil + } + // ... existing enabled, non-dry-run body unchanged below this point ... +``` + +Update the two callers from Task 10's test file and every `cmd/*` call site from Tasks 11-13 to pass `false` for `dryRun` (or the resolved flag value) and a `port.Diagnostics` (each `cmd/*/main.go` already constructs one — thread the existing `diag` variable through). + +- [ ] **Step 6: Add `--product-metrics-dry-run` to all four binaries' flags** + +Mirror Task 11 Step 1's flag pattern in each of the four flag-registration sites touched in Tasks 11-13: + +```go + fs.BoolVar(&cfg.productMetricsDryRun, "product-metrics-dry-run", false, + "print every product-metrics observation to stderr instead of sending it — verify the no-PII claim yourself before enabling --product-metrics for real") +``` + +- [ ] **Step 7: Run the full `productmetrics` and `cliconfig` suites** + +Run: `cd internal/adapter/productmetrics && go test ./... -v && cd ../../cliconfig && go test ./... -v` +Expected: PASS + +- [ ] **Step 8: Run `task build` to confirm every binary still compiles with the new parameter threaded through** + +Run: `task build` +Expected: succeeds. + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/dryrun.go internal/adapter/productmetrics/dryrun_test.go \ + internal/cliconfig/productmetrics.go cmd/mecated/main.go cmd/mecatui/ cmd/mecatequi/ cmd/mecak8s/ +git commit -m "feat(productmetrics): --product-metrics-dry-run audit mode across all four binaries" +``` + +--- + +### Task 15: ADR + user-docs + +**Files:** +- Create: `docs/adr/0317-product-metrics.md` (confirm 0317 is still the next free number by running `ls docs/adr | grep -oE '^[0-9]+' | sort -n | tail -1` immediately before creating the file — another PR may have landed a higher number since this plan was written) +- Create: `user-docs/building/what-you-get/product-metrics.md` (confirm the exact directory naming convention by listing `user-docs/building/what-you-get/` first — follow its existing file-naming/frontmatter pattern exactly) +- Modify: `internal/cliconfig/productmetrics.go` (fill in `ProductMetricsDisclosureNotice`'s doc link placeholder with the real path) + +**Interfaces:** none — documentation only. + +- [ ] **Step 1: Confirm the next free ADR number** + +Run: `ls docs/adr | grep -oE '^[0-9]+' | sort -n | tail -1` +Use ` + 1` as the filename prefix (0317 as of this plan's writing). + +- [ ] **Step 2: Write the ADR** + +Create `docs/adr/0317-product-metrics.md` (or whatever number Step 1 resolved) following `docs/adr/template.md`'s structure (header with `- Status: Accepted` / `- Date:` / `- Scope:` / `- Supersedes: —` / `- Superseded by: —`, then Context/Decision/Consequences/See also, mirroring ADR 0098's structure read during planning). Content: summarize this plan's design — the fully independent adapter, the exact `mecatl.adoption.*` catalog (Tasks 4-5), the opt-out precedence (Task 9), the operator-tier-only settings gate (Tasks 7-8), the privacy guard test (Task 6), the dry-run audit mode (Task 14), and the "why opt-out is defensible here" rationale (disclosure notice + `DO_NOT_TRACK` + a reviewable, tested catalog). Cross-reference ADR 0098 (headless telemetry) and ADR 0020 (diagnostics/the three-channel table this is a deliberate fourth, separate channel from) as prior art it deliberately does NOT reuse. + +- [ ] **Step 3: Write the user-docs page** + +List `user-docs/building/what-you-get/` first to match its existing frontmatter/heading conventions, then add a short page: what's collected (link the ADR's catalog), the opt-out mechanisms (flag, `DO_NOT_TRACK`, settings.yaml), and how to self-verify via `--product-metrics-dry-run`. + +- [ ] **Step 4: Fill in the disclosure notice's doc link** + +In `internal/cliconfig/productmetrics.go`, replace `` in `ProductMetricsDisclosureNotice` with the real path/URL to the page written in Step 3. + +- [ ] **Step 5: Run the docs gates** + +Run: `task docs` (regenerates the configuration reference + runs the strict link gate) and `task site:build` (Docusaurus build, catches a broken link before CI does). +Expected: both succeed with no broken links/anchors. + +- [ ] **Step 6: Run `task lint && task test` one final time across the whole feature** + +Run: `task lint && task test` +Expected: green. + +- [ ] **Step 7: Commit** + +```bash +git add docs/adr/ user-docs/building/what-you-get/ internal/cliconfig/productmetrics.go +git commit -m "docs: add ADR and user-docs for opt-out product/adoption metrics" +``` diff --git a/docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md b/docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md new file mode 100644 index 0000000000..d00975b26f --- /dev/null +++ b/docs/superpowers/plans/2026-09-10-product-metrics-activation-extension.md @@ -0,0 +1,1320 @@ +# Product Metrics — Activation/Retention/Reliability Extension Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extend the already-shipped `internal/adapter/productmetrics` pipeline (PR #1278) to answer the product team's activation/time-to-value/retention/reliability questions: reinstate a per-install identity (accepting the cardinality cost, now that it's been sized and understood), and add `had_tool_call`, tool category/outcome, `run_duration`, `tool_calls_per_run`, and `time_to_first_value`. + +**Architecture:** A small, additive `engine/port` capability (`RunAwareToolCallRecorder`, mirroring the existing `HookApprovalLearner` optional-interface precedent) lets the already-shipped `Recorder` correlate a tool call to the run that made it — the one piece of engine-layer work this requires. Everything else is confined to `internal/adapter/productmetrics`, `internal/cliconfig`, and a new Helm template for `mecak8s`'s install-id provisioning. + +**Tech Stack:** Go 1.26, the existing `internal/adapter/productmetrics` package (Tasks 1-15 of the prior plan, already merged), Helm (`deploy/helm/mecak8s/`). + +**Spec:** This plan's own context section below records every design decision reached in conversation; there is no separate written design doc for this increment — the conversation that produced it is the record. + +## Global Constraints + +- **No new free-text/PII surface.** Tool category comes from a structural `mcp__` prefix check (confirmed via `internal/adapter/mcp/tool.go:89`'s `"mcp__" + server + "__" + toolName` construction) — never a maintained allowlist, never the raw MCP server/tool name. Everything else stays a bounded enum or a count/duration, per the existing package-wide guard test discipline (Task 6 of the prior plan). +- **`install.id` is now a deliberate, accepted exception** to "every attribute is bounded" — reinstate it as a resource attribute (undoing the earlier removal), now that its cardinality cost has been sized (~$1,930/month at 100K installs, worst-case 24/7 uptime, full catalog, on the actual AMP pricing model) and accepted. +- **The engine port change must be purely additive.** `RunAwareToolCallRecorder` is a NEW, STANDALONE interface (not embedding `ToolCallRecorder`), type-asserted at the one dispatch call site — mirrors `engine/port/hookrunner.go`'s `HookApprovalLearner` exactly. No existing `ToolCallRecorder` implementer (the operator `internal/adapter/telemetry.Metrics`/`RoleMetrics`, `jsonlstore`, `redisstore`, etc.) needs to change at all. This is an `Added` (minor) change per `engine/COMPATIBILITY.md` — run `task api:update` and add an `engine/CHANGELOG.md` entry. +- **`task lint && task test` must stay green after every task.** `task api:check` (part of `task test`) must pass after the engine port change. + +--- + +### Task 1: Engine port extension — `RunAwareToolCallRecorder` + +**Files:** +- Modify: `engine/port/log.go` (add the new interface, do NOT touch `ToolCallRecorder`) +- Modify: `engine/agent/dispatch.go` (the one call site, `execute`, currently around line 1293) +- Test: `engine/agent/run_aware_tool_call_recorder_test.go` (new) +- Modify: `engine/CHANGELOG.md`, run `task api:update` to regenerate `engine/api/*.txt` + +**Interfaces:** +- Produces: `type RunAwareToolCallRecorder interface { ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) }` in `engine/port`. + +- [ ] **Step 1: Write the failing test** + +```go +package agent + +import ( + "context" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/session" +) + +// runAwareFakeRecorder implements BOTH port.ToolCallRecorder and the new +// port.RunAwareToolCallRecorder, recording which method the dispatcher chose. +type runAwareFakeRecorder struct { + plainCalls int + runAwareCalls int + lastRunID string +} + +func (f *runAwareFakeRecorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + f.plainCalls++ +} + +func (f *runAwareFakeRecorder) ToolCallForRun(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + f.runAwareCalls++ + f.lastRunID = runID +} + +// TestExecutePrefersRunAwareToolCallRecorderWhenImplemented pins that the +// dispatcher, at its one ToolCallRecorder call site, calls ToolCallForRun +// (never both) when the injected recorder implements it, passing the SAME +// RunID the enclosing Run already carries — and falls back to the plain +// ToolCall for a recorder that does not implement the richer interface +// (every existing ToolCallRecorder implementer is unaffected). +func TestExecutePrefersRunAwareToolCallRecorderWhenImplemented(t *testing.T) { + rec := &runAwareFakeRecorder{} + eng, sess := newTestEngineWithToolCallRecorder(t, rec) // see Step 3 for this test helper's real signature, sourced from an existing dispatch_test.go helper + runID := driveOneToolCallingTurn(t, eng, sess) // helper: drives a turn that calls a tool at least once + + if rec.plainCalls != 0 { + t.Errorf("plainCalls = %d, want 0 (RunAwareToolCallRecorder must be preferred)", rec.plainCalls) + } + if rec.runAwareCalls == 0 { + t.Fatal("runAwareCalls = 0, want at least 1") + } + if rec.lastRunID != runID { + t.Errorf("lastRunID = %q, want %q (the enclosing Run's own id)", rec.lastRunID, runID) + } +} + +func TestExecuteFallsBackToPlainToolCallRecorder(t *testing.T) { + // A recorder implementing ONLY port.ToolCallRecorder (not the richer + // interface) must keep working exactly as before — confirmed via the + // EXISTING plain-ToolCallRecorder test fixture already in this package + // (find it by name in dispatch_test.go and reuse it directly rather than + // inventing a new one). +} +``` + +Note to implementer: `newTestEngineWithToolCallRecorder`/`driveOneToolCallingTurn` are placeholder helper NAMES — before writing this file, grep `engine/agent/*_test.go` for the EXISTING test harness this package already uses to build a test `*Engine` and drive a tool-calling turn (there is one; every dispatch test in this package uses it), and write these two tests using the REAL existing helpers/fixtures, not new ones. `TestExecuteFallsBackToPlainToolCallRecorder`'s body is intentionally left for you to fill in using that same real harness with a recorder implementing only the base interface — assert its existing plain-`ToolCall` path still fires exactly as it does today (this is a regression guard, not new behavior). + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd engine && go test ./agent/... -run TestExecutePrefersRunAwareToolCallRecorderWhenImplemented -v` +Expected: FAIL — `port.RunAwareToolCallRecorder` undefined, or the dispatcher doesn't yet type-assert for it. + +- [ ] **Step 3: Add the port interface** + +In `engine/port/log.go`, immediately after the existing `ToolCallRecorder` interface, add: + +```go +// RunAwareToolCallRecorder is an OPTIONAL capability a ToolCallRecorder may +// ALSO implement to additionally receive the RunID of the run that made the +// call (the same opaque per-run correlation id carried on session.Event.RunID, +// ADR 0249) — the one thing ToolCall's signature cannot express, since a +// SessionID can span many sequential runs over a session's lifetime and +// ToolCall alone gives no way to tell which run a given call belongs to. +// +// The engine TYPE-ASSERTS this interface on Deps.ToolCallRecorder and calls +// ToolCallForRun INSTEAD OF ToolCall (never both) when implemented — so a +// recorder that implements only the base ToolCallRecorder is wholly +// unaffected (no method added to ToolCallRecorder: that would be a breaking +// change, mirroring the HookApprovalLearner precedent in hookrunner.go). +type RunAwareToolCallRecorder interface { + // ToolCallForRun is ToolCall's signature plus the leading runID — the + // same value the enclosing Run stamps onto every session.Event.RunID it + // emits. Consumers that need to correlate a tool call to the run that + // made it (e.g. "did this run have at least one successful tool call") + // use this instead of ToolCall. + ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) +} +``` + +- [ ] **Step 4: Wire the type-assertion at the dispatch call site** + +In `engine/agent/dispatch.go`, replace: + +```go + if e.deps.ToolCallRecorder != nil { + e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + } +``` + +with: + +```go + if e.deps.ToolCallRecorder != nil { + if aware, ok := e.deps.ToolCallRecorder.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(r.RunID(), sess.ID, c, res, queued, dur) + } else { + e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + } + } +``` + +(`r` is the enclosing `execute` method's existing `*Run` parameter — already in scope two lines below this call site where `e.emit(r, ...)` is called; `port` is already imported in this file — confirm, and add the import if for some reason it is not.) + +- [ ] **Step 5: Run test to verify it passes** + +Run: `cd engine && go test ./agent/... -run 'TestExecutePrefersRunAwareToolCallRecorderWhenImplemented|TestExecuteFallsBackToPlainToolCallRecorder' -v` +Expected: PASS (both tests) + +- [ ] **Step 6: Run the full engine test suite to catch any regression** + +Run: `cd engine && go test ./... -race` +Expected: PASS, no regressions (this is an additive interface change; every existing `ToolCallRecorder` consumer's tests should be untouched). + +- [ ] **Step 7: Update the engine API compatibility surface** + +Run: `task api:update` (regenerates `engine/api/*.txt`). Then add an entry to `engine/CHANGELOG.md` classified as `Added` (minor) per `engine/COMPATIBILITY.md`'s rules, e.g.: + +```markdown +## Unreleased +### Added +- `port.RunAwareToolCallRecorder`: an optional `ToolCallRecorder` extension + that additionally receives the calling run's `RunID`, letting a consumer + correlate a tool call to the run that made it. Purely additive — no + existing `ToolCallRecorder` implementer is affected. +``` + +- [ ] **Step 8: Run `task api:check` and repo-wide `task lint`** + +Run: `task api:check` (should now pass against the regenerated `engine/api/*.txt`) and `task lint` (0 issues expected). + +- [ ] **Step 9: Commit** + +```bash +git add engine/port/log.go engine/agent/dispatch.go engine/agent/run_aware_tool_call_recorder_test.go engine/CHANGELOG.md engine/api/ +git commit -m "feat(engine): add the optional RunAwareToolCallRecorder port capability" +``` + +--- + +### Task 2: Reinstate `install.id` + +**Files:** +- Modify: `internal/adapter/productmetrics/config.go` (restore `Config.InstallID`) +- Modify: `internal/adapter/productmetrics/provider.go` (restore the resource attribute) +- Modify: `internal/adapter/productmetrics/provider_test.go` (restore the test's `InstallID` field) +- Modify: `internal/cliconfig/productmetrics.go` (thread `installID` back into `Config`) +- Modify: `internal/adapter/productmetrics/installid.go` (doc comment: remove the "deliberately never threaded" language — it's threaded again now) +- Test: existing `installid_test.go` unaffected (the persistence mechanism itself never changed) + +**Interfaces:** +- Produces: `Config.InstallID string` restored; `NewProvider`'s resource attributes include `"mecatl.install.id": cfg.InstallID` again. + +- [ ] **Step 1: Restore `Config.InstallID`** + +In `internal/adapter/productmetrics/config.go`, restore the field: + +```go +// Config configures a Provider/Recorder pair for one process. +type Config struct { + // Binary identifies which of the four entry points this process is. + Binary Binary + // Version is the mecatl build version (resource attribute service.version). + Version string + // InstallID is this process's persisted (or externally-provisioned, for + // mecak8s — see Task 7) anonymous install identifier. Reinstated as a + // resource attribute after being sized and accepted: ~$1,930/month at + // 100K installs under worst-case 24/7 uptime on the actual AMP pricing + // model (see the ADR's updated cost-analysis section, Task 8). + InstallID string +} +``` + +- [ ] **Step 2: Write the failing test (provider_test.go)** + +Restore the `InstallID` field to the existing `TestNewProviderExportsToConfiguredEndpoint` test's `Config{...}` literal: + +```go + p, err := NewProvider(context.Background(), Config{ + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", + }) +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go build ./...` +Expected: FAIL — `unknown field InstallID` (Config doesn't have it back yet if you did Step 2 before Step 1 — do Step 1 first; this ordering note exists so the two steps are both concrete, not because there is a real red-green gap here — `Config`'s field addition and the test asserting it stay in the SAME commit). + +- [ ] **Step 4: Restore the resource attribute in `provider.go`** + +In `internal/adapter/productmetrics/provider.go`, restore: + +```go + composite, err := providers.NewCompositeProvider(ctx, + providers.WithServiceName("mecatl"), + providers.WithServiceVersion(cfg.Version), + providers.WithOTLPEndpoint(endpoint), + providers.WithMetricsEnabled(true), + providers.WithInsecure(strings.HasPrefix(endpoint, "http://")), + providers.WithHeaders(map[string]string{headerKeyName: bakedKey}), + providers.WithCustomAttributes(map[string]string{ + "mecatl.install.id": cfg.InstallID, + "mecatl.binary": string(cfg.Binary), + }), + ) +``` + +removing the "Deliberately NOT included" doc comment above it (or rewriting it — see Step 5). + +- [ ] **Step 5: Rewrite the doc comment explaining the reinstated decision** + +Replace the comment block above the `NewCompositeProvider` call with: + +```go + // mecatl.install.id is a per-install random UUID, deliberately attached + // as a resource attribute (so it flattens onto every instrument this + // provider exports). This was removed once (see git history) over + // unbounded-cardinality concerns on the Prometheus-remote-write + // destination (stacklok/infra#5604), then reinstated after the actual + // cost was sized against real AMP pricing and accepted — see the ADR's + // cost-analysis section for the numbers. mecak8s provisions this value + // differently (a stable per-Helm-release ConfigMap, not this package's + // local install-id file — see internal/cliconfig's mecak8s wiring and + // deploy/helm/mecak8s/templates/install-id-configmap.yaml), since a + // pod-local file would mint a new id on every pod restart. +``` + +- [ ] **Step 6: Restore threading in `internal/cliconfig/productmetrics.go`** + +Change: + +```go + // LoadOrCreateInstallIDDefault still runs (and persists its file) purely + // to detect first-run for the disclosure notice below — the returned id + // value itself is deliberately discarded, never threaded to NewProvider: + // see provider.go's doc comment on why a per-install identifier must + // never become a Prometheus-remote-write label. + _, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + }) +``` + +to: + +```go + // LoadOrCreateInstallIDDefault persists (or reads back) this process's + // local install-id file and reports firstRun for the disclosure notice + // below. Reinstated as a real, exported resource attribute (see + // provider.go's doc comment) after its cardinality cost was sized and + // accepted. + installID, firstRun, err := productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + InstallID: installID, + }) +``` + +Note: `BuildProductMetrics`'s signature does NOT change in this task — mecak8s's alternate provisioning (Task 7) overrides `installID` BEFORE calling `BuildProductMetrics` by having its OWN caller read the env-var-provided id and pass it through a new, distinct code path added in Task 7; this task only restores the DEFAULT (local-file) path all four binaries currently share. + +- [ ] **Step 7: Update `installid.go`'s doc comment** + +In `internal/adapter/productmetrics/installid.go`, remove the paragraph beginning "The returned id is deliberately never threaded into any exported metric attribute..." (added when `install.id` was removed) — replace with: + +```go +// The returned id IS threaded into an exported resource attribute (see +// provider.go) — this package makes no attempt to keep the id local-only; +// that was a prior, now-reverted design (see git history / the ADR's +// cost-analysis section for why it was reinstated). +``` + +- [ ] **Step 8: Run tests to verify they pass** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` and `cd ../../cliconfig && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/config.go internal/adapter/productmetrics/provider.go \ + internal/adapter/productmetrics/provider_test.go internal/adapter/productmetrics/installid.go \ + internal/cliconfig/productmetrics.go +git commit -m "feat(productmetrics): reinstate mecatl.install.id after sizing its cardinality cost" +``` + +--- + +### Task 3: Per-run tracking — `had_tool_call`, tool category + outcome + +**Files:** +- Modify: `internal/adapter/productmetrics/metrics.go` (unify per-run tracking; add `had_tool_call` attribute; add the `attrCategory`/`attrOutcome` keys) +- Modify: `internal/adapter/productmetrics/toolcall.go` (implement `ToolCallForRun`, category/outcome derivation, per-run tallying) +- Modify: `internal/adapter/productmetrics/metrics_test.go`, `toolcall_test.go` +- Modify: `internal/adapter/productmetrics/bounded_test.go` (extend the allowlist + drive `ToolCallForRun` with sensitive markers) + +**Interfaces:** +- Consumes: `port.RunAwareToolCallRecorder` (Task 1). +- Produces: `Recorder.ToolCallForRun(...)` (satisfies the new interface); `mecatl.product.runs_completed`'s new `had_tool_call` attribute; `mecatl.product.tool_calls`'s new `category`/`outcome` attributes. + +- [ ] **Step 1: Write the failing tests** + +```go +// In toolcall_test.go, alongside the existing TestRecorderToolCallCountsWithoutIdentity: + +func TestRecorderToolCallForRunCategorizesBuiltinsByName(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, "category", "Bash"); got != 1 { + t.Errorf("tool_calls{category=Bash} = %d, want 1", got) + } + if got := sumPoint(t, agg, "category", "Read"); got != 1 { + t.Errorf("tool_calls{category=Read} = %d, want 1", got) + } +} + +func TestRecorderToolCallForRunBucketsMCPToolsUnderOneCategory(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__github__list_issues"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__slack__post_message"}, session.ToolResult{}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, "category", "mcp"); got != 2 { + t.Errorf("tool_calls{category=mcp} = %d, want 2 (both MCP-server tools bucketed together)", got) + } + // The real server/tool names must never appear as an attribute value. + var rm metricdata.ResourceMetrics + _ = reader.Collect(context.Background(), &rm) + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + continue + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + if kv.Value.AsString() == "github" || kv.Value.AsString() == "list_issues" { + t.Fatalf("MCP server/tool name leaked as an attribute value: %s=%s", kv.Key, kv.Value.AsString()) + } + } + } + } + } +} + +func TestRecorderToolCallForRunRecordsOutcome(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, "outcome", "success"); got != 1 { + t.Errorf("tool_calls{outcome=success} = %d, want 1", got) + } + if got := sumPoint(t, agg, "outcome", "error"); got != 1 { + t.Errorf("tool_calls{outcome=error} = %d, want 1", got) + } +} +``` + +```go +// In metrics_test.go, alongside the existing runs_completed tests: + +func TestRecorderRunsCompletedHadToolCallTrueWhenASuccessfulToolCallOccurred(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1", got) + } +} + +func TestRecorderRunsCompletedHadToolCallFalseWithNoToolCall(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-2", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1", got) + } +} + +func TestRecorderRunsCompletedHadToolCallFalseWhenOnlyToolCallErrored(t *testing.T) { + // A tool call that ERRORED does not count toward had_tool_call — the + // product definition requires at least one SUCCESSFUL tool/action. + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-3", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-3", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1 (the only tool call errored)", got) + } +} + +func TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns(t *testing.T) { + // Two runs interleaved (a real possibility: Team/Parallel fan-out, or + // two concurrent client sessions on one process) must not leak state + // into each other. + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-a", session.SessionID("s1"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-b", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-a", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "had_tool_call", "false"); got != 1 { + t.Errorf("run-b's had_tool_call = %d points at false, want exactly 1 (run-a's tool call must not leak into run-b)", got) + } + if got := sumPoint(t, agg, "had_tool_call", "true"); got != 1 { + t.Errorf("run-a's had_tool_call = %d points at true, want exactly 1", got) + } +} +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd internal/adapter/productmetrics && go build ./...` +Expected: FAIL — `ToolCallForRun` undefined, `had_tool_call`/`category`/`outcome` attributes don't exist yet. + +- [ ] **Step 3: Unify per-run tracking in `metrics.go`** + +Replace the existing `runFamiliesUsed`/`usedFamilies` per-run tracking (added in the prior plan's review-fix commit) with a single, richer per-run record covering everything this task and Task 4 need: + +```go +// perRunState tracks, per LIVE run (keyed by session.Event.RunID / the same +// id ToolCallForRun receives), the bounded facts this package derives across +// the Emit/ToolCallForRun boundary. Cleared on EvResult so it stays bounded +// to concurrently-live runs, never growing across a process's lifetime. +type perRunState struct { + subagentSeen bool + teamSeen bool + hadToolCall bool + toolCallCount int64 + startedAt time.Time +} + +// runs guards concurrent access to the live-run map — the SAME discipline +// the prior subagent/team dedup fix already established, now extended to +// cover had_tool_call/tool_calls_per_run/run_duration too. +type runs struct { + mu sync.Mutex + byRun map[string]*perRunState +} + +func newRuns() *runs { return &runs{byRun: make(map[string]*perRunState)} } + +// get returns (creating if absent) the live perRunState for runID. An empty +// runID (no run context) returns a throwaway, never-shared state — matching +// the prior code's "always counts" fallback for the no-run-context case. +func (r *runs) get(runID string) *perRunState { + if runID == "" { + return &perRunState{} + } + r.mu.Lock() + defer r.mu.Unlock() + st, ok := r.byRun[runID] + if !ok { + st = &perRunState{} + r.byRun[runID] = st + } + return st +} + +// clear drops runID's live state at EvResult, returning the state that was +// there (or a zero-value one if none existed — e.g. a run with no tool +// calls and no delegation family use). +func (r *runs) clear(runID string) *perRunState { + if runID == "" { + return &perRunState{} + } + r.mu.Lock() + defer r.mu.Unlock() + st, ok := r.byRun[runID] + if !ok { + return &perRunState{} + } + delete(r.byRun, runID) + return st +} +``` + +Replace the `Recorder` struct's `mu sync.Mutex` + `runFamiliesUsed map[string]usedFamilies` fields with a single `perRun *runs` field, and update `NewRecorder` to initialize it: `r.perRun = newRuns()`. + +Update `firstInRun`/`clearRun` (the prior plan's helpers) to use `perRun.get(runID)`/`perRun.clear(runID)` instead of the old map directly — e.g.: + +```go +func (r *Recorder) firstInRun(runID string, family delegationFamily) bool { + st := r.perRun.get(runID) + r.perRun.mu.Lock() + defer r.perRun.mu.Unlock() + switch family { + case familySubagent: + if st.subagentSeen { + return false + } + st.subagentSeen = true + case familyTeam: + if st.teamSeen { + return false + } + st.teamSeen = true + } + return true +} +``` + +(Adjust exact lock placement so `runs.get`'s own internal lock and this method's use of the returned pointer don't double-lock or race — the simplest correct shape is for ALL mutation of a `*perRunState`'s fields to happen while holding `r.perRun.mu`, so restructure `get`/`clear` to return the state WITHOUT unlocking around field access, or have every state-mutating method take the lock itself around both the map lookup AND the field mutation as one critical section. Get this right and prove it with `-race` in Step 8 — this is the one place in this task worth extra care.) + +- [ ] **Step 4: Add the `had_tool_call` attribute to `recordResult`** + +```go +const attrHadToolCall = "had_tool_call" + +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, runID string) { + st := r.perRun.clear(runID) + hadToolCall := "false" + if st.hadToolCall { + hadToolCall = "true" + } + if res == nil { + r.runsCompleted.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrStop, string(session.StopNone)), + attribute.String(attrHadToolCall, hadToolCall))) + return + } + r.runsCompleted.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrStop, string(res.Stop)), + attribute.String(attrHadToolCall, hadToolCall))) + u := res.Usage + // ... existing tokens.Add(...) calls unchanged ... +} +``` + +Update `Emit`'s `case session.EvResult:` arm to pass `ev.RunID`: `r.recordResult(ctx, ev.Result, ev.RunID)`. + +- [ ] **Step 5: Implement `ToolCallForRun` in `toolcall.go`** + +```go +package productmetrics + +import ( + "context" + "strings" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +const ( + attrCategory = "category" + attrOutcome = "outcome" +) + +// mcpToolPrefix is the STRUCTURAL naming convention every client MCP tool is +// registered under (internal/adapter/mcp/tool.go: `"mcp__" + server + "__" + +// toolName`) — checking this prefix, rather than maintaining a built-in-tool +// allowlist, means (a) a new built-in tool is automatically and correctly +// categorized by its own real name with no allowlist to keep in sync, and +// (b) an MCP server/tool name can never leak, structurally, regardless of +// what any future MCP integration is named. +const mcpToolPrefix = "mcp__" + +// toolCategory derives the bounded category attribute for a tool call: the +// tool's own name for a built-in (never sensitive — mecatl's own fixed +// catalog), or the single literal "mcp" for anything MCP-server-provided +// (never the specific server/tool name). +func toolCategory(name string) string { + if strings.HasPrefix(name, mcpToolPrefix) { + return "mcp" + } + return name +} + +// ToolCall satisfies port.ToolCallRecorder for a caller that does not +// implement/use the richer port.RunAwareToolCallRecorder path — it records +// with no run correlation (runID ""), matching this package's pre-existing, +// always-counts behavior for the no-run-context case. +func (r *Recorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + r.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder. It records the +// bounded category/outcome attributes and tallies the per-run state Task 4's +// run_duration and this task's had_tool_call/tool_calls_per_run all read at +// EvResult time. It never reads call.Name/result.Content beyond the bounded +// category derivation above — no free text, no session id, no MCP +// server/tool name. +func (r *Recorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + ctx := context.Background() + outcome := "success" + if result.IsError { + outcome = "error" + } + r.toolCalls.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrCategory, toolCategory(call.Name)), + attribute.String(attrOutcome, outcome))) + + st := r.perRun.get(runID) + r.perRun.mu.Lock() + st.toolCallCount++ + if !result.IsError { + st.hadToolCall = true + } + r.perRun.mu.Unlock() +} + +// Compile-time interface checks. +var ( + _ port.ToolCallRecorder = (*Recorder)(nil) + _ port.RunAwareToolCallRecorder = (*Recorder)(nil) +) +``` + +(Remove the OLD `ToolCall` implementation this replaces — the prior plan's version that ignored every parameter and just bumped `r.toolCalls.Add(ctx, 1)` with no attributes.) + +- [ ] **Step 6: Update `metrics.go`'s `tool_calls` instrument description** + +The `mecatl.adoption.tool_calls`/`mecatl.product.tool_calls` counter's `metric.WithDescription(...)` string currently says "no tool identity attached" — update it: + +```go + if r.toolCalls, err = meter.Int64Counter("mecatl.product.tool_calls", + metric.WithDescription("Total tool calls executed, by bounded category (a built-in tool's own name, or the single value \"mcp\" for any MCP-server tool) and outcome.")); err != nil { +``` + +- [ ] **Step 7: Extend `bounded_test.go`** + +Add `attrCategory`, `attrOutcome`, `attrHadToolCall` to `allowedAttributeKeys`. Extend the existing `TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent` to ALSO drive `ToolCallForRun("run-x", ..., session.ToolCall{Name: "mcp__evilserver__leak_this_name"}, ...)` and assert `"evilserver"`/`"leak_this_name"` never appear as an attribute value anywhere in the collected output — this is the guard test's whole job, extend it rather than adding a separate one. + +- [ ] **Step 8: Run tests with `-race` to verify correctness and no data races** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` +Expected: PASS, all tests including the new ones, `-race` clean (this task adds concurrent map + struct-field access — `-race` is not optional here). + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/toolcall.go \ + internal/adapter/productmetrics/metrics_test.go internal/adapter/productmetrics/toolcall_test.go \ + internal/adapter/productmetrics/bounded_test.go +git commit -m "feat(productmetrics): had_tool_call, tool category/outcome via RunAwareToolCallRecorder" +``` + +--- + +### Task 4: `run_duration` histogram + +**Files:** +- Modify: `internal/adapter/productmetrics/metrics.go` +- Test: `internal/adapter/productmetrics/metrics_test.go` + +**Interfaces:** +- Consumes: `perRunState.startedAt` (Task 3). +- Produces: `mecatl.product.run_duration` histogram (seconds). + +- [ ] **Step 1: Write the failing test** + +```go +func TestRecorderRunDurationRecordedFromSessionInitToResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + time.Sleep(5 * time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.run_duration"] + if !ok { + t.Fatal("mecatl.product.run_duration missing") + } + hist, ok := agg.(metricdata.Histogram[float64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[float64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 { + t.Fatalf("expected exactly 1 recorded duration, got %+v", hist.DataPoints) + } + if hist.DataPoints[0].Sum <= 0 { + t.Errorf("recorded duration sum = %v, want > 0", hist.DataPoints[0].Sum) + } +} + +func TestRecorderRunDurationNotRecordedWithoutMatchingSessionInit(t *testing.T) { + // A run whose EvSessionInit this Recorder never observed (e.g. process + // restarted mid-run — an edge case, not a common path) must not record a + // bogus/negative duration. + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "orphan-run", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + if agg, ok := collect(t, reader)["mecatl.product.run_duration"]; ok { + if hist, ok := agg.(metricdata.Histogram[float64]); ok && len(hist.DataPoints) > 0 && hist.DataPoints[0].Count > 0 { + t.Errorf("recorded a duration for a run with no observed EvSessionInit: %+v", hist.DataPoints) + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderRunDuration -v` +Expected: FAIL — instrument doesn't exist yet. + +- [ ] **Step 3: Add the instrument and start/stop bracketing** + +In `NewRecorder`, add: + +```go + if r.runDuration, err = meter.Float64Histogram("mecatl.product.run_duration", + metric.WithDescription("Wall-clock duration of a run, from session init to result, in seconds."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: run_duration histogram: %w", err) + } +``` + +Add `runDuration metric.Float64Histogram` to the `Recorder` struct. + +In `Emit`'s `case session.EvSessionInit:` arm, stamp the start time: + +```go + case session.EvSessionInit: + r.sessionsStarted.Add(ctx, 1) + if ev.RunID != "" { + st := r.perRun.get(ev.RunID) + r.perRun.mu.Lock() + st.startedAt = time.Now() + r.perRun.mu.Unlock() + } +``` + +In `recordResult` (or right after `st := r.perRun.clear(runID)` in `Emit`'s `EvResult` handling), record the duration only when `startedAt` was actually observed: + +```go + if !st.startedAt.IsZero() { + r.runDuration.Record(ctx, time.Since(st.startedAt).Seconds()) + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderRunDuration -v` +Expected: PASS + +- [ ] **Step 5: Run the full package suite with `-race`** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 6: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/metrics_test.go +git commit -m "feat(productmetrics): run_duration histogram" +``` + +--- + +### Task 5: `tool_calls_per_run` histogram + `time_to_first_value` + +**Files:** +- Modify: `internal/adapter/productmetrics/metrics.go` (the `tool_calls_per_run` histogram, recorded at `EvResult` from `perRunState.toolCallCount`) +- Create: `internal/adapter/productmetrics/firstvalue.go` (the one-time marker + histogram) +- Test: `internal/adapter/productmetrics/firstvalue_test.go` +- Modify: `internal/cliconfig/productmetrics.go` (thread the first-value check into `BuildProductMetrics`) + +**Interfaces:** +- Produces: `mecatl.product.tool_calls_per_run` histogram; `mecatl.product.time_to_first_value` histogram (recorded at most once per install); `func LoadOrCreateFirstValueMarker(...) (alreadyRecorded bool, err error)` mirroring `installid.go`'s shape. + +- [ ] **Step 1: `tool_calls_per_run` — write the failing test** + +```go +func TestRecorderToolCallsPerRunRecordedAtResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg := collect(t, reader)["mecatl.product.tool_calls_per_run"] + hist, ok := agg.(metricdata.Histogram[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[int64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Sum != 2 { + t.Fatalf("expected one data point summing to 2, got %+v", hist.DataPoints) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails, then implement** + +Add to `NewRecorder`: + +```go + if r.toolCallsPerRun, err = meter.Int64Histogram("mecatl.product.tool_calls_per_run", + metric.WithDescription("Total tool calls made within a single run.")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls_per_run histogram: %w", err) + } +``` + +In `recordResult`, alongside the existing `st := r.perRun.clear(runID)`: + +```go + r.toolCallsPerRun.Record(ctx, st.toolCallCount) +``` + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestRecorderToolCallsPerRun -v` — expect PASS after the change. + +- [ ] **Step 3: `time_to_first_value` — write the failing test** + +```go +package productmetrics + +import ( + "errors" + "os" + "testing" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + written := map[string][]byte{} + readFile := func(p string) ([]byte, error) { + if d, ok := written[p]; ok { + return d, nil + } + return nil, os.ErrNotExist + } + writeFile := func(p string, d []byte, _ os.FileMode) error { written[p] = d; return nil } + mkdirAll := func(string, os.FileMode) error { return nil } + + already, err := LoadOrCreateFirstValueMarker(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) + } + if already { + t.Error("already = true on first call, want false") + } + + // Second call must report it as already recorded, and not error. + already2, err := LoadOrCreateFirstValueMarker(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateFirstValueMarker: %v", err) + } + if !already2 { + t.Error("already = false on second call, want true") + } +} + +func TestLoadOrCreateFirstValueMarkerFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + if _, err := LoadOrCreateFirstValueMarker(env, nil, nil, nil); err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } +} +``` + +- [ ] **Step 4: Run test to verify it fails** + +Run: `cd internal/adapter/productmetrics && go test ./... -run TestLoadOrCreateFirstValueMarker -v` +Expected: FAIL — `LoadOrCreateFirstValueMarker` undefined. + +- [ ] **Step 5: Write `firstvalue.go`** + +```go +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// firstValueMarkerRelPath is the state-dir-relative path to a bare marker +// file recording whether this install's first meaningful-and-successful run +// has already been observed — so mecatl.product.time_to_first_value is +// recorded at most once per install, ever, mirroring installid.go's +// first-run marker pattern exactly. +const firstValueMarkerRelPath = "mecatl/first-value-recorded" + +// LoadOrCreateFirstValueMarker reports whether this install's first-value +// moment was already recorded (already == true), creating the marker (and +// returning already == false) the first time it is called. Once created, it +// is never removed automatically — deleting it (like the install-id file) +// resets the install and lets time_to_first_value fire once more. +func LoadOrCreateFirstValueMarker( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (already bool, err error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return false, fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + path := filepath.Join(base, firstValueMarkerRelPath) + + if readFile != nil { + if _, rerr := readFile(path); rerr == nil { + return true, nil + } + } + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte("1"), 0o600); werr != nil { + return false, fmt.Errorf("productmetrics: write first-value marker: %w", werr) + } + } + return false, nil +} + +// LoadOrCreateFirstValueMarkerDefault binds LoadOrCreateFirstValueMarker to +// the real process environment and filesystem. +func LoadOrCreateFirstValueMarkerDefault() (already bool, err error) { + return LoadOrCreateFirstValueMarker(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} +``` + +- [ ] **Step 6: Add the `time_to_first_value` instrument and recording logic** + +Add to `NewRecorder`: + +```go + if r.timeToFirstValue, err = meter.Float64Histogram("mecatl.product.time_to_first_value", + metric.WithDescription("One-time-per-install duration from this install's first-seen moment to its first had_tool_call=true, stop=success run."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: time_to_first_value histogram: %w", err) + } +``` + +Add a field to `Recorder`: `firstSeenAt time.Time` (set once, at construction) and `firstValueRecorded *atomic.Bool` (or guard via the marker file check, done ONCE at `BuildProductMetrics` construction time rather than per-event — see Step 7, this is simpler than trying to gate it per-Emit-call). + +Actually, the simplest correct design: do the "has this already been recorded" check ONCE, in `internal/cliconfig.BuildProductMetrics` (Step 7 below), NOT inside `Recorder` itself — pass a plain `bool` (`trackFirstValue`) into `NewRecorder`/`Config`, and have `Recorder.recordResult` check `res.Stop == session.StopEndTurn && st.hadToolCall && !r.firstValueAlreadyRecorded` before recording once and flipping an in-memory flag (`sync.Once` or a guarded bool) — the FILE write (marking it recorded forever) happens in the CALLER once, when `BuildProductMetrics` first observes `already == false` at startup... but that's wrong too, since the marker needs to be written the MOMENT the qualifying run actually happens, not at process startup (a process might never have a qualifying run). Correct shape: `Recorder` itself owns a `sync.Once`-guarded write-through: on the FIRST qualifying `EvResult`, it (a) computes and records the duration, (b) calls a caller-injected `markFirstValueRecorded func() error` closure (wrapping `os.WriteFile` at the real path) exactly once. Thread this closure into `NewRecorder` (or a new `NewRecorderWithFirstValue(mp, alreadyRecorded bool, markRecorded func())` variant) rather than `Config`, to keep `NewRecorder`'s existing signature stable for the (many) existing call sites/tests that don't care about this feature. + +Given the added complexity, the pragmatic shape: + +```go +// Recorder field additions: + firstSeenAt time.Time + firstValueDone bool // true if already recorded (this run OR a prior one) + firstValueRecordFn func() error // writes the local marker file; nil disables recording entirely + firstValueMu sync.Mutex +``` + +`NewRecorder`'s signature stays unchanged (existing callers/tests untouched); add a new setter-style method used only by `BuildProductMetrics`: + +```go +// EnableFirstValueTracking arms mecatl.product.time_to_first_value tracking: +// firstSeenAt is this install's first-seen timestamp (from the SAME local +// install-id file's mtime, or "now" if unavailable — an approximation is +// fine, this metric's whole purpose is a coarse "how long did onboarding +// take" signal, not a billing-grade timer). alreadyRecorded, when true, +// permanently disables further recording for this Recorder's lifetime (this +// install already has its one sample). recordFn persists the marker so a +// LATER process invocation also stays disabled; it is called at most once. +func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecorded bool, recordFn func() error) { + r.firstValueMu.Lock() + defer r.firstValueMu.Unlock() + r.firstSeenAt = firstSeenAt + r.firstValueDone = alreadyRecorded + r.firstValueRecordFn = recordFn +} +``` + +In `recordResult`, after the existing token/had_tool_call recording: + +```go + if res != nil && res.Stop == session.StopEndTurn && st.hadToolCall { + r.firstValueMu.Lock() + if !r.firstValueDone && !r.firstSeenAt.IsZero() { + r.firstValueDone = true + r.timeToFirstValue.Record(ctx, time.Since(r.firstSeenAt).Seconds()) + if r.firstValueRecordFn != nil { + _ = r.firstValueRecordFn() // best-effort; a failed write just risks re-recording once on a later process, not a correctness bug + } + } + r.firstValueMu.Unlock() + } +``` + +- [ ] **Step 7: Wire this into `BuildProductMetrics`** + +In `internal/cliconfig/productmetrics.go`, after constructing `recorder` and before returning: + +```go + firstValueAlready, fvErr := productmetrics.LoadOrCreateFirstValueMarkerDefault() + // A failure here degrades to "track it anyway" (fvErr != nil implies + // firstValueAlready's zero value false) rather than disabling the whole + // pipeline — time_to_first_value is a nice-to-have signal, not + // load-bearing enough to fail product metrics setup entirely over. + firstSeenAt := time.Now() + if info, statErr := os.Stat(installIDFilePath(...)); statErr == nil { // see note below + firstSeenAt = info.ModTime() + } + recorder.EnableFirstValueTracking(firstSeenAt, fvErr == nil && firstValueAlready, func() error { + _, _, err := productmetrics.LoadOrCreateFirstValueMarkerDefault() + return err + }) +``` + +Note to implementer: `installIDFilePath(...)` is illustrative — `installid.go`'s `installIDRelPath` const plus `xdgconfig.UserStateDir` is how the REAL install-id file's path is computed; either export a small helper from `productmetrics` that returns this path (cleanest), or accept the simpler approximation of always using `time.Now()` as `firstSeenAt` when the install-id file's actual mtime isn't easily available at this call site — the metric's own doc comment already says an approximation is acceptable. Use your judgment on which is cleaner; either is acceptable, but document whichever you pick in the instrument's description string if it differs from what Step 6 already says. + +- [ ] **Step 8: Run the full package suite** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` and `cd ../../cliconfig && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 9: Commit** + +```bash +git add internal/adapter/productmetrics/metrics.go internal/adapter/productmetrics/firstvalue.go \ + internal/adapter/productmetrics/firstvalue_test.go internal/cliconfig/productmetrics.go +git commit -m "feat(productmetrics): tool_calls_per_run + time_to_first_value" +``` + +--- + +### Task 6: Update `DryRunRecorder` to match + +**Files:** +- Modify: `internal/adapter/productmetrics/dryrun.go` +- Modify: `internal/adapter/productmetrics/dryrun_test.go` + +**Interfaces:** +- Produces: `DryRunRecorder` now also implements `port.RunAwareToolCallRecorder`, and logs the same new bounded fields (`had_tool_call`, `category`, `outcome`) the real `Recorder` would have recorded — the dry-run's whole purpose is showing EXACTLY what the real pipeline would send, so it must track every new field the real one does. + +- [ ] **Step 1: Write the failing test** + +```go +func TestDryRunRecorderLogsCategoryOutcomeAndHadToolCall(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__someserver__sensitive_tool"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + found := false + for _, args := range diag.allArgs() { // see note: capturingDiag needs a small extension to expose recorded args, not just messages, for this assertion — extend it minimally + for _, a := range args { + if s, ok := a.(string); ok && (s == "someserver" || s == "sensitive_tool") { + t.Fatalf("MCP server/tool name leaked in dry-run output: %q", s) + } + } + } + _ = found +} +``` + +Note to implementer: the existing `capturingDiag` fake (in `dryrun_test.go`) currently only captures `msg string`, not the `args ...any` — extend it minimally to also store `args` per call, since this test (and the privacy discipline this file exists to prove) needs to inspect them. + +- [ ] **Step 2: Run test to verify it fails, then implement** + +In `dryrun.go`, implement `ToolCallForRun` mirroring the real `Recorder`'s logic (category/outcome derivation), and update `ToolCall` to delegate to it with `runID=""`, same shape as Task 3's real `Recorder`: + +```go +func (d *DryRunRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + d.ToolCallForRun("", id, call, result, queued, took) +} + +func (d *DryRunRecorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + outcome := "success" + if result.IsError { + outcome = "error" + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1", + "category", toolCategory(call.Name), "outcome", outcome) +} + +var _ port.RunAwareToolCallRecorder = (*DryRunRecorder)(nil) +``` + +Update `Emit`'s `EvResult` case to also log `had_tool_call` (reuse whatever per-run tracking is simplest for the dry-run path — a lighter-weight version than the real `Recorder`'s is fine here, e.g. its own small `runs *runs`-shaped field, or simply omitting exact had_tool_call tracking in dry-run and logging `"had_tool_call", "unknown (dry-run does not track per-run state)"` if that's meaningfully simpler — use your judgment, document whichever you choose). + +- [ ] **Step 3: Run tests** + +Run: `cd internal/adapter/productmetrics && go test ./... -race -v` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add internal/adapter/productmetrics/dryrun.go internal/adapter/productmetrics/dryrun_test.go +git commit -m "feat(productmetrics): DryRunRecorder mirrors the new category/outcome/had_tool_call fields" +``` + +--- + +### Task 7: mecak8s install-id ConfigMap + +**Files:** +- Create: `deploy/helm/mecak8s/templates/install-id-configmap.yaml` +- Modify: `deploy/helm/mecak8s/templates/deployment.yaml` (mount the id as an env var) +- Modify: `deploy/helm/mecak8s/chart_test.go` (or wherever this chart's existing tests live — extend, don't invent a new test file if one already renders/asserts this chart's templates) +- Modify: `cmd/mecak8s/observability.go` (read the env var instead of calling the local-file mechanism, when set) + +**Interfaces:** +- Produces: a `ConfigMap` named e.g. `{{ include "mecak8s.fullname" . }}-install-id` holding one key (`installId`), generated once via the `lookup`-based idiom and reused across `helm upgrade`; an env var `MECATL_PRODUCT_METRICS_INSTALL_ID` sourced from it, mounted into the `mecak8s` container. + +- [ ] **Step 1: Write the ConfigMap template** + +```yaml +# deploy/helm/mecak8s/templates/install-id-configmap.yaml +# +# Generates ONE stable install-id for this Helm release, reused across every +# replica and every `helm upgrade` — unlike a per-pod local file (which mecak8s +# cannot use at all: it runs storage-free, no PVC, per ADR 0048, and every pod +# restart would otherwise mint a fresh, never-reused id — the worst-case +# cardinality pattern for the product-metrics pipeline this feeds). The +# `lookup` guard is the standard Helm idiom for "generate once, keep stable on +# upgrade": if a ConfigMap of this name already exists in this release's +# namespace, its EXISTING value is reused verbatim; only a genuinely first +# `helm install` (or a deliberately deleted ConfigMap) mints a new one. +{{- $existing := lookup "v1" "ConfigMap" .Release.Namespace (printf "%s-install-id" (include "mecak8s.fullname" .)) }} +{{- $installID := "" }} +{{- if $existing }} +{{- $installID = index $existing.data "installId" }} +{{- else }} +{{- $installID = uuidv4 }} +{{- end }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "mecak8s.fullname" . }}-install-id + labels: + {{- include "mecak8s.labels" . | nindent 4 }} +data: + installId: {{ $installID | quote }} +``` + +(`mecak8s.fullname`/`mecak8s.labels` are illustrative — before writing this, check `deploy/helm/mecak8s/templates/_helpers.tpl` for the chart's REAL helper template names and use those, not invented ones.) + +- [ ] **Step 2: Mount it as an env var in `deployment.yaml`** + +In `deploy/helm/mecak8s/templates/deployment.yaml`, add to the container's `env:` list (find the existing `env:` block — confirmed at line 201 in the prior investigation): + +```yaml + - name: MECATL_PRODUCT_METRICS_INSTALL_ID + valueFrom: + configMapKeyRef: + name: {{ include "mecak8s.fullname" . }}-install-id + key: installId +``` + +- [ ] **Step 3: Read the env var in `cmd/mecak8s/observability.go`** + +Before the existing call to `cliconfig.BuildProductMetrics(...)`, add: + +```go + // mecak8s cannot use the local-file install-id mechanism the other three + // binaries share (storage-free, no PVC, per ADR 0048 — every pod restart + // would mint a fresh id, the worst-case cardinality pattern). Instead, a + // stable per-Helm-release id is provisioned via a ConfigMap (see + // deploy/helm/mecak8s/templates/install-id-configmap.yaml) and threaded + // through this env var. Empty (no chart-provisioned id, e.g. running the + // binary directly outside the chart) falls back to whatever + // BuildProductMetrics's own default local-file mechanism produces — + // which will still work, just without the "one stable id per k8s + // deployment" guarantee the chart provides. + installIDOverride := os.Getenv("MECATL_PRODUCT_METRICS_INSTALL_ID") +``` + +Thread `installIDOverride` into `cliconfig.BuildProductMetrics`'s call — this requires a SMALL signature addition to `BuildProductMetrics` (an optional `installIDOverride string` parameter, empty meaning "use the default local-file mechanism"): when non-empty, `BuildProductMetrics` skips `LoadOrCreateInstallIDDefault()` entirely and uses the override value directly as `Config.InstallID`. Update the OTHER three binaries' call sites to pass `""` (unaffected, unchanged behavior). + +- [ ] **Step 4: Update/extend the chart's existing test** + +Find the existing Helm chart test (`deploy/helm/mecak8s/chart_test.go`, confirmed to exist by the earlier `grep -rln readOnlyRootFilesystem` search) and add a case asserting the new `ConfigMap` template renders with a valid UUID in `data.installId`, and that the `Deployment` template's env var correctly references it via `configMapKeyRef`. + +- [ ] **Step 5: Run the chart tests and the `cmd/mecak8s`/`cliconfig` Go tests** + +Run: `cd deploy/helm/mecak8s && go test ./... -v` (if this is how chart_test.go is invoked — check its actual invocation mechanism, e.g. it may use the `helm` binary via `os/exec` or a Go Helm-templating library; follow whatever the EXISTING tests in this file already do) and `cd /Users/reyniero/work/mecatl/.claude/worktrees/product-metrics-otel/cmd/mecak8s && go test ./... -race -v` and `cd ../../internal/cliconfig && go test ./... -race -v`. +Expected: PASS + +- [ ] **Step 6: Run `task k8s:e2e` or equivalent if this repo has one (check Taskfile.yml for a k8s-specific e2e task) as an extra confidence check, given this touches the real Helm chart** + +If such a task exists, run it; if it requires a live kind cluster and is out of scope for a quick local check, note that in your report and rely on the chart_test.go coverage instead. + +- [ ] **Step 7: Commit** + +```bash +git add deploy/helm/mecak8s/templates/install-id-configmap.yaml deploy/helm/mecak8s/templates/deployment.yaml \ + deploy/helm/mecak8s/chart_test.go cmd/mecak8s/observability.go internal/cliconfig/productmetrics.go +git commit -m "feat(mecak8s): provision a stable per-release install-id via a Helm ConfigMap" +``` + +--- + +### Task 8: ADR + user-docs + PR description updates, final verification + +**Files:** +- Modify: `docs/adr/0319-product-metrics.md` +- Modify: `user-docs/building/what-you-get/observability.md` +- Modify: the open PR's description (via `gh pr edit`) + +**Interfaces:** none — documentation only, plus final verification. + +- [ ] **Step 1: Update the ADR's catalog table** + +Add rows for `mecatl.product.run_duration`, `mecatl.product.tool_calls_per_run`, `mecatl.product.time_to_first_value`; update `runs_completed`'s row to show its new `had_tool_call` attribute; update `tool_calls`'s row to show its new `category`/`outcome` attributes (removing the "no tool/MCP-server name label at all" claim, replacing it with an accurate description of the bounded category scheme). + +- [ ] **Step 2: Add a new ADR section documenting the reinstatement** + +Add a section (e.g. "## Reinstating `mecatl.install.id`, and the mecak8s ConfigMap") recording: why it was reinstated (sized, accepted cost — cite the actual $/month figures from this conversation), the `RunAwareToolCallRecorder` engine addition and why it's additive/non-breaking, and the mecak8s-specific ConfigMap mechanism and why the local-file approach cannot work there (storage-free, ADR 0048, pod churn). + +- [ ] **Step 3: Update `user-docs/building/what-you-get/observability.md`** + +Update the "What's collected" paragraph to include the new fields (tool category/outcome, had_tool_call, run duration, tool-calls-per-run, time-to-first-value) and to accurately state that an anonymous per-install identifier is now collected (reversing the prior "no ... identifier" framing) — be precise and honest here, this is the user-facing disclosure text's source of truth. + +- [ ] **Step 4: Update `internal/cliconfig/productmetrics.go`'s `ProductMetricsDisclosureNotice` string** + +This is the actual STARTUP disclosure text users see — it must also honestly reflect that a per-install identifier is now collected. Update its wording accordingly. + +- [ ] **Step 5: Run every gate** + +```bash +task lint +task test +task build +task docs +task site:build +``` +All must pass clean. + +- [ ] **Step 6: Update the PR description** + +Fetch the current PR body (`gh pr view --json body -q .body`), update the "Metrics catalog" section to reflect the new/changed instruments, add a short "Reinstating install.id" note explaining the reversal and its rationale (cost sizing, mecak8s ConfigMap mechanism), and push via `gh pr edit`. + +- [ ] **Step 7: Commit the doc changes** + +```bash +git add docs/adr/0319-product-metrics.md user-docs/building/what-you-get/observability.md internal/cliconfig/productmetrics.go +git commit -m "docs: document had_tool_call, tool category/outcome, and the install.id reinstatement" +git push +``` diff --git a/docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md b/docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md new file mode 100644 index 0000000000..1904945dca --- /dev/null +++ b/docs/superpowers/specs/2026-09-08-product-metrics-otel-design.md @@ -0,0 +1,236 @@ +# Product (adoption) metrics over OTLP — design + +- Status: Draft +- Date: 2026-09-08 +- Scope: new `internal/adapter/productmetrics`, `internal/cliconfig`, `internal/adapter/permconfig` (new `telemetry:` operator section), `cmd/mecated`, `cmd/mecatui`, `cmd/mecatequi`, `cmd/mecak8s` + +## Context + +mecatl has no visibility into community adoption today: no install counts, no +feature-adoption signal, no aggregate usage depth. Stacklok's infra team has +stood up a dedicated, internet-facing OTLP/HTTP metrics ingest at +`https://metrics.stacklok.com/v1/metrics` specifically for mecatl binaries +running on infrastructure Stacklok does not control (`stacklok/infra#5604`): +API-key-gated at the edge (`x-mecatl-metrics-key` header, stripped before the +collector), and server-side filtered to accept only metric names matching +`^mecatl\..*`. + +This is a **new, separate concern** from mecatl's existing operator-facing +observability. `internal/adapter/telemetry` already ships a full OTel pipeline +(ADR 0018/0045/0098): a `MeterProvider` with an always-on Prometheus reader and +an optional OTLP push reader an *operator* points at *their own* collector, plus +an OTLP trace exporter. That pipeline exists so an operator can observe their +own deployment. It must never become the transport for community-adoption data +— an operator's own `--otlp-endpoint` configuration must have zero effect on +what does or doesn't reach Stacklok, and enabling product metrics must have +zero effect on what an operator's own collector receives. + +`toolhive-core` (already an mecatl dependency, `v0.0.43`) ships +`telemetry/providers`: a small, already-reviewed OTel SDK-wiring layer +(`providers.NewCompositeProvider`) that builds a `metric.MeterProvider` from an +options struct (endpoint, headers, service name/version, custom resource +attributes) without ever installing it as the process-global provider. This is +the natural building block for the new pipeline's OTLP/HTTP exporter — it +already speaks the exact shape the new collector expects (custom headers, +OTLP/HTTP, a resource with service name/version), so mecatl does not need to +hand-roll a third OTLP wiring implementation next to the two it already has +(`internal/adapter/telemetry/otlp.go`'s own inline construction, and the +toolhive-core one). + +## Decision + +### 1. A fully independent adapter, `internal/adapter/productmetrics` + +Zero import relationship with `internal/adapter/telemetry`. It owns: + +- Its own `metric.MeterProvider`, built via `toolhive-core/telemetry/providers` + with a **hardcoded** endpoint (`https://metrics.stacklok.com/v1/metrics`) and + a **hardcoded** header key baked into the binary at build time (matching the + infra PR's `x-mecatl-metrics-key` contract) — neither is operator-configurable. + There is exactly one place this data can go. +- Its own small `Recorder` type implementing `port.EventSink` + + `port.ToolCallRecorder` (the same two seams `internal/adapter/telemetry` + taps), but extracting *only* the bounded counts in the catalog below — the + type has no field or parameter through which a tool name, session id, model + id, or free text could ever flow. +- Its own heartbeat ticker (fires once at start, then every ~24h for + long-running processes; a single fire + flush-before-exit for the short-lived + `mecatequi`, mirroring the existing OTLP-push-with-flush precedent in ADR + 0098). +- Its own install-identity file. + +Composition combines the two independent sinks with a trivial fan-out helper +in `internal/cliconfig` (the existing Rule-of-Three home for cross-binary +telemetry wiring, per ADR 0098) — `internal/app` stays import-free of +`productmetrics`, exactly as it is of `telemetry` today. Each `cmd/*/main.go` +builds its existing operator telemetry pipeline unchanged, and — only when +product metrics are enabled — separately constructs a `productmetrics.Recorder` +and tees it in alongside. + +This means: an operator who disables their own OTLP export still has product +metrics flow (if enabled) to Stacklok, and an operator who fully disables +product metrics has zero effect on their own OTLP/Prometheus pipeline. The two +literally cannot leak into each other because they share no struct, provider, +registry, or destination — only the same two read-only observation points in +the engine (`port.EventSink`, `port.ToolCallRecorder`), which every consumer +of those ports already receives independently per composition's existing +fan-out discipline. + +### 2. Metric catalog + +All instrument names are namespaced under `mecatl.adoption.*` — passes the +collector's `^mecatl\..*` filter, and is visually/query-wise distinct from the +operator-facing `mecatl.tool.*`/`mecatl.runs`/etc. family, so nobody looking at +either series family can mistake one for the other. + +**Resource attributes** (set once per process, not per-metric labels): +- `service.name` = `mecatl`, `service.version` +- `os.type`, `host.arch` (standard OTel semconv, generic platform facts) +- `mecatl.install.id` — a random v4 UUID (see §4) +- `mecatl.binary` — one of `mecated`/`mecatui`/`mecatequi`/`mecak8s` (closed set) + +**Heartbeat** (on start, then every ~24h for long-running processes; single +fire for `mecatequi`): +- `mecatl.adoption.heartbeat` (counter, +1 per fire) — the install/liveness signal. +- `mecatl.adoption.feature_enabled{feature=...}` (counter, +1 per enabled + feature per heartbeat) — `feature` is one of a closed set: `memory`, + `learning`, `guardrails`, `mcp`, `teams`, `subagents`, `scheduling`. +- `mecatl.adoption.provider_configured{family=...}` (counter) — `family` is one + of `anthropic`/`openai`/`openrouter`/`other` (never a model id/alias). +- `mecatl.adoption.deployment_mode{mode=...}` (counter) — `mode` is one of + `interactive`/`headless`/`k8s`. + +**Coarse usage** (derived from the event/tool-call tap, exported on the +provider's normal periodic-reader cadence — no manual batching needed since +these are cumulative counters): +- `mecatl.adoption.sessions_started` (counter) — on `EvSessionInit`. +- `mecatl.adoption.runs_completed{stop=...}` (counter) — on `EvResult`; `stop` + reuses the existing bounded `session.StopReason` enum. +- `mecatl.adoption.tool_calls` (counter, **no tool/MCP-server name label at + all**) — on every `ToolCallRecorder.ToolCall`. +- `mecatl.adoption.tokens{kind=...}` (counter) — `kind` reuses the existing + bounded token-kind enum (`input`/`output`/`cache_read`/`cache_write`/ + `reasoning`). +- `mecatl.adoption.subagent_used` / `mecatl.adoption.team_used` (counter, + bumped at most once per run when that delegation family appears at all — no + def/member/model name ever surfaces). + +Nothing here is free text, a session/run/model identifier, a tool or MCP +server name, a file path, a prompt, or an output. Every label value is drawn +from a closed enum that already exists internally or is defined fresh in this +package as a small closed set. + +### 3. Config & opt-out + +**Enabled by default** (opt-out), but the toggle is **operator-tier only** — +same trust boundary as `guardrails:`/`openrouter:` (AGENTS.md's existing +operator-tier-only precedent): a project-tier `.mecatl/settings.yaml` can +neither enable nor disable it for a user; a project silently overriding a +user's own telemetry choice in either direction would itself be a trust +violation, so it is parsed with the same WARN-and-ignore discipline as the +other operator-only subtrees. + +`~/.config/mecatl/settings.yaml`: +```yaml +telemetry: + productMetrics: + enabled: true # default; set false to opt out +``` + +Additional disable signals, in precedence order (highest first): +1. CLI flag `--product-metrics=false` (or `--no-product-metrics`) on all four binaries. +2. `DO_NOT_TRACK` environment variable (any non-empty value) — the + cross-ecosystem convention (consoledonottrack.com), so a single env var + already used to opt CI fleets and dev machines out of *other* tools' telemetry + also covers mecatl, with no mecatl-specific config needed. +3. `telemetry.productMetrics.enabled: false` in the operator settings file. +4. Default: enabled. + +A dedicated `MECATL_PRODUCT_METRICS=0` env var is deliberately **not** added on +top of `DO_NOT_TRACK` — one standard signal is preferable to two overlapping +ones with subtly different names. + +**First-run disclosure.** The first time a binary is about to actually send +product metrics in a given run (i.e., telemetry is enabled and this is the +first invocation since the install-id file didn't yet exist), it prints one +non-blocking line to stderr: what is collected (link to the user-docs page), +and the exact flag/setting/env var to disable it. This is not a prompt — it +never blocks — but it is a hard requirement for an opt-out default to be +defensible to the community; the same disclosure text is what the mecatui +zero-state and mecated startup banner both use. + +**Dry-run / audit flag.** `--product-metrics-dry-run` prints every metric this +process would have sent to stderr instead of exporting it, so a skeptical +operator can verify the "no PII" claim directly rather than trusting the docs. + +### 4. Install identity + +A random v4 UUID, generated on first use and persisted at +`$XDG_CONFIG_HOME/mecatl/telemetry-id` (or the platform equivalent via the +existing `internal/adapter/xdgconfig` helper). It contains no machine or user +information and is trivially reset by deleting the file (equivalent, from +Stacklok's side, to seeing what looks like a new install). It exists purely so +"ten heartbeats from one install" isn't miscounted as "ten installs" — no +other purpose. + +### 5. Privacy safeguards + +- A reflect-based guard test (mirroring the existing `attrRole`/`attrStop` + bounded-label discipline, and the `engine/port/diagnostics_imports_test.go` + import-tripwire pattern) asserts the `Recorder`'s entire public API accepts + no bare `string`/free-text parameter — only bounded enum types (Go types + with a small closed value set) and counts. This makes "no PII can flow + through this type" a property a future PR's CI run checks, not just a + code-review norm. +- Exporter failures are silent to the app (lazy-dial exporter matching the + existing OTLP exporter pattern; a dead `metrics.stacklok.com` never blocks + or slows down a session). `Shutdown` is bounded (~3s) so a hung network path + can never delay process exit. +- No content, error message text, file path, or identifier of any kind is + ever an attribute value — every label is a value from a fixed Go-level enum + reviewed in this document. + +### 6. Lifecycle across binaries + +All four binaries (`mecated`, `mecatui`, `mecatequi`, `mecak8s`) wire this +identically through the shared `internal/cliconfig` helper (extending the +existing `HeadlessTelemetry`-style Rule-of-Three home). `mecatequi` (short-lived, +often sub-second) fires one heartbeat and flushes before exit, exactly as its +existing OTLP metrics push already does (ADR 0098). The other three run the +24h ticker for the process lifetime and flush on graceful shutdown. + +### 7. Documentation + +- A new ADR (next available number) records this decision, the exact catalog, + and the "operator-tier only, separate pipeline, no PII" invariants — the + same discipline as ADR 0098/0020, and the natural home for the "why opt-out + is acceptable here" rationale (disclosure + easy universal disable + a + published, reviewable catalog). +- A short `user-docs/` page explains what's collected, links the catalog in + this doc, and gives the exact disable instructions (flag, env var, setting). + +## Testing + +- Unit tests for the `Recorder`'s bounded-enum bumps (one per metric, table- + driven against the catalog above). +- The reflect-based "no free-text parameter" guard test (privacy invariant, + §5). +- A conformance-style test that composing the operator sink + product-metrics + sink and disabling one leaves the other's counters unaffected (isolation + invariant). +- `internal/cliconfig` wiring test verifying the opt-out precedence order + (flag > `DO_NOT_TRACK` > settings.yaml > default-enabled). +- Offline only — no live network call to `metrics.stacklok.com` in tests; the + OTLP exporter construction is exercised against a local httptest server, the + same pattern `internal/app/openrouter_route_e2e_test.go` already uses for a + comparable "real adapter against an httptest stand-in" check. + +## Out of scope (deferred) + +- A per-run cost/dollar metric (mirrors the existing deferred #192 cost-metric + gap in the operator pipeline — no `Cost` field exists anywhere yet). +- Any richer per-feature usage counters beyond the catalog above (e.g. per-MCP + transport type, per-model-family latency) — start narrow, revisit only if a + concrete adoption question the catalog can't answer comes up. +- A first-run interactive consent prompt (explicitly declined in favor of + opt-out + disclosure, per the design decision above). diff --git a/engine/CHANGELOG.md b/engine/CHANGELOG.md index 2fe6bc94c3..cc6675d40f 100644 --- a/engine/CHANGELOG.md +++ b/engine/CHANGELOG.md @@ -28,6 +28,11 @@ The covered surface is the eight core packages (`session`, `governance`, `learni - **Session placement authority repair** — removes the orphan exported `session.PlacementSelector` protocol, adds persisted display-only `session.PlacementMetadata`, requires a valid `EnvironmentRef` at aggregate construction, and rejects direct engine runs whose live environment does not match the session identity. Changed (breaking, pre-v1 minor). +- `port.RunAwareToolCallRecorder`: an optional `ToolCallRecorder` extension + that additionally receives the calling run's `RunID`, letting a consumer + correlate a tool call to the run that made it. Purely additive — no + existing `ToolCallRecorder` implementer is affected. Added (minor). + - **Delegation artifact boundary (ADR 0288)** — adds the distinct `agent.ArtifactHandle` type, changes `agent.PreservedForkStore.Preserve` to key retained forks by that opaque handle rather than a physical root, and removes `Workspace`/`WinnerWorkspace` from `session.ParallelPayload`. Parallel results now expose an opaque preserved-artifact handle while physical fork roots remain private orchestration state. Changed (breaking, pre-v1 minor). - **Unified environment and placement identity** — adds `Revision` and `Valid` to `session.EnvironmentRef`, makes that exact `{Kind, ID, Revision}` value the runtime and durable placement identity, and removes the short-lived duplicate `session.PlacementRef`/`PlacementKind` types. Engine-created Subagent, Parallel, and Team child sessions now persist the identity carried by their `tool.Environment`; `port.ScheduleSpec` and `port.SessionDiscoveryMeta` replace workspace paths with the exact private environment identity, with schedules also retaining their trusted placement scope. Changed (breaking, pre-v1 minor). diff --git a/engine/agent/dispatch.go b/engine/agent/dispatch.go index 130defcf8a..79eaa1c8b6 100644 --- a/engine/agent/dispatch.go +++ b/engine/agent/dispatch.go @@ -1290,7 +1290,11 @@ func (e *Engine) execute(ctx context.Context, r *Run, sess *session.Session, env res = session.RepairToolResult(res) if e.deps.ToolCallRecorder != nil { - e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + if aware, ok := e.deps.ToolCallRecorder.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(r.RunID(), sess.ID, c, res, queued, dur) + } else { + e.deps.ToolCallRecorder.ToolCall(sess.ID, c, res, queued, dur) + } } e.emit(r, session.Event{Type: session.EvToolResult, Turn: turnIdx, ToolResult: ptr(res)}) diff --git a/engine/agent/run_aware_tool_call_recorder_test.go b/engine/agent/run_aware_tool_call_recorder_test.go new file mode 100644 index 0000000000..6b0db4dcac --- /dev/null +++ b/engine/agent/run_aware_tool_call_recorder_test.go @@ -0,0 +1,110 @@ +package agent_test + +import ( + "context" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/adapter/memfs" + "github.com/stacklok/mecatl/engine/adapter/mockllm" + "github.com/stacklok/mecatl/engine/agent" + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/engine/tool" +) + +// runAwareFakeRecorder implements BOTH port.ToolCallRecorder and the new +// port.RunAwareToolCallRecorder, recording which method the dispatcher chose. +type runAwareFakeRecorder struct { + plainCalls int + runAwareCalls int + lastRunID string +} + +func (f *runAwareFakeRecorder) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + f.plainCalls++ +} + +func (f *runAwareFakeRecorder) ToolCallForRun(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + f.runAwareCalls++ + f.lastRunID = runID +} + +// newToolCallingEngine builds an engine + session wired with a single Read tool +// call followed by a final text turn, mirroring TestFullCycle's mockllm script, +// with the given ToolCallRecorder injected. +func newToolCallingEngine(t *testing.T, rec interface { + ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) +}) (*agent.Engine, *session.Session) { + t.Helper() + read := &fakeTool{name: "Read", readOnly: true, + exec: func(_ context.Context, in session.ToolCall, _ tool.Workspace) (session.ToolResult, error) { + return session.NewToolResult(in.ID, "file contents"), nil + }} + cat := catalogWith(t, read) + + llm := mockllm.New( + mockllm.ChunksTurn( + mockllm.TextChunk("let me look"), + mockllm.ToolCallChunk(toolCall("c1", "Read", `{"path":"a.go"}`)), + mockllm.UsageChunk(session.Usage{InputTokens: 10, OutputTokens: 2}), + mockllm.DoneChunk(session.StopEndTurn), + ), + mockllm.ChunksTurn( + mockllm.TextChunk("all done"), + mockllm.UsageChunk(session.Usage{InputTokens: 5, OutputTokens: 3}), + mockllm.DoneChunk(session.StopEndTurn), + ), + ) + + clk := &fakeClock{t: time.Unix(0, 0)} + e := newEngine(agent.Deps{LLM: llm, Catalog: cat, Clock: clk, ToolCallRecorder: rec}) + sess := newSession(t, session.Limits{}) + return e, sess +} + +// driveOneToolCallingTurn drives the engine through the single tool-calling +// turn scripted by newToolCallingEngine and returns the enclosing Run's RunID. +func driveOneToolCallingTurn(t *testing.T, e *agent.Engine, sess *session.Session) string { + t.Helper() + ws := memfs.NewWorkspace("/ws") + r := e.Run(context.Background(), sess, agent.EnvForWS(ws, nil), agent.RunRequest{Text: "look at a.go"}) + runID := r.RunID() + drain(r) + return runID +} + +// TestExecutePrefersRunAwareToolCallRecorderWhenImplemented pins that the +// dispatcher, at its one ToolCallRecorder call site, calls ToolCallForRun +// (never both) when the injected recorder implements it, passing the SAME +// RunID the enclosing Run already carries — and falls back to the plain +// ToolCall for a recorder that does not implement the richer interface +// (every existing ToolCallRecorder implementer is unaffected). +func TestExecutePrefersRunAwareToolCallRecorderWhenImplemented(t *testing.T) { + rec := &runAwareFakeRecorder{} + e, sess := newToolCallingEngine(t, rec) + runID := driveOneToolCallingTurn(t, e, sess) + + if rec.plainCalls != 0 { + t.Errorf("plainCalls = %d, want 0 (RunAwareToolCallRecorder must be preferred)", rec.plainCalls) + } + if rec.runAwareCalls == 0 { + t.Fatal("runAwareCalls = 0, want at least 1") + } + if rec.lastRunID != runID { + t.Errorf("lastRunID = %q, want %q (the enclosing Run's own id)", rec.lastRunID, runID) + } +} + +// TestExecuteFallsBackToPlainToolCallRecorder is a regression guard: a +// recorder implementing ONLY port.ToolCallRecorder (not the richer +// RunAwareToolCallRecorder) must keep working exactly as before, using the +// package's existing plain recordingLogger fixture. +func TestExecuteFallsBackToPlainToolCallRecorder(t *testing.T) { + logger := &recordingLogger{} + e, sess := newToolCallingEngine(t, logger) + driveOneToolCallingTurn(t, e, sess) + + if logger.calls != 1 { + t.Fatalf("logger recorded %d tool calls, want 1", logger.calls) + } +} diff --git a/engine/api/port.txt b/engine/api/port.txt index 2e7c81c105..7a77f17637 100644 --- a/engine/api/port.txt +++ b/engine/api/port.txt @@ -31,6 +31,7 @@ ifacemethod func (PrunableStore).Delete(ctx context.Context, id session.SessionID) error ifacemethod func (PrunableStore).List(ctx context.Context) ([]StoredSession, error) ifacemethod func (RetryDispositionError).RetryDisposition() session.RetryDisposition + ifacemethod func (RunAwareToolCallRecorder).ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued time.Duration, took time.Duration) ifacemethod func (ScheduleCreator).Create(ctx context.Context, s Schedule) error ifacemethod func (ScheduleManager).CreateSchedule(ctx context.Context, spec ScheduleSpec) (Schedule, error) ifacemethod func (ScheduleManager).DeleteSchedule(ctx context.Context, name string) error @@ -189,6 +190,7 @@ type PrunableStore interface{Delete(ctx context.Context, id session.SessionID) e type ReadOptions struct{Limit int; Follow bool} type RetryDisposition = session.RetryDisposition type RetryDispositionError interface{RetryDisposition() session.RetryDisposition; error} +type RunAwareToolCallRecorder interface{ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued time.Duration, took time.Duration)} type Schedule struct{Spec ScheduleSpec; State ScheduleState} type ScheduleCreator interface{Create(ctx context.Context, s Schedule) error} type ScheduleFire struct{ID string; ScheduleName string; SessionID session.SessionID; FiredAt time.Time; StartedAt time.Time; ProgressAt time.Time; Deadline time.Time; Stop session.StopReason; Err string} diff --git a/engine/port/log.go b/engine/port/log.go index 881707408a..38784833d0 100644 --- a/engine/port/log.go +++ b/engine/port/log.go @@ -39,3 +39,24 @@ type ToolCallRecorder interface { // Clock is injected. ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) } + +// RunAwareToolCallRecorder is an OPTIONAL capability a ToolCallRecorder may +// ALSO implement to additionally receive the RunID of the run that made the +// call (the same opaque per-run correlation id carried on session.Event.RunID, +// ADR 0249) — the one thing ToolCall's signature cannot express, since a +// SessionID can span many sequential runs over a session's lifetime and +// ToolCall alone gives no way to tell which run a given call belongs to. +// +// The engine TYPE-ASSERTS this interface on Deps.ToolCallRecorder and calls +// ToolCallForRun INSTEAD OF ToolCall (never both) when implemented — so a +// recorder that implements only the base ToolCallRecorder is wholly +// unaffected (no method added to ToolCallRecorder: that would be a breaking +// change, mirroring the HookApprovalLearner precedent in hookrunner.go). +type RunAwareToolCallRecorder interface { + // ToolCallForRun is ToolCall's signature plus the leading runID — the + // same value the enclosing Run stamps onto every session.Event.RunID it + // emits. Consumers that need to correlate a tool call to the run that + // made it (e.g. "did this run have at least one successful tool call") + // use this instead of ToolCall. + ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) +} diff --git a/internal/adapter/permconfig/permconfig.go b/internal/adapter/permconfig/permconfig.go index a04b4d8270..eb83e92fb6 100644 --- a/internal/adapter/permconfig/permconfig.go +++ b/internal/adapter/permconfig/permconfig.go @@ -168,6 +168,7 @@ func (c *Config) UnmarshalYAML(node ast.Node) error { "learning": newPermconfigNodePointer(&c.Learning), "steer": newPermconfigNodePointer(&c.Steer), "openrouter": newPermconfigNodePointer(&c.OpenRouter), + "telemetry": newPermconfigNodePointer(&c.Telemetry), "mcp": newPermconfigNodePointer(&c.MCP), "retention": newPermconfigNodePointer(&c.Retention), "storage_management": newPermconfigNodePointer(&c.StorageManagement), diff --git a/internal/adapter/permconfig/resolve.go b/internal/adapter/permconfig/resolve.go index 3e7abb936f..d379f48266 100644 --- a/internal/adapter/permconfig/resolve.go +++ b/internal/adapter/permconfig/resolve.go @@ -196,6 +196,15 @@ type Resolver struct { // (explicit files) out-ranks user-global (first-non-nil keeps CLI). operatorOpenRouter *OpenRouterSection + // operatorTelemetry is the OPERATOR-TIER telemetry: subtree, read ONCE at + // construction from the user-global + CLI tiers ONLY (the SOLE capture path + // is captureTelemetry from loadUserRules — mirroring captureOpenRouter). A + // project-tier file's telemetry: block is IGNORED with a WARN in + // loadProjectRules. nil when no operator-tier file carried a telemetry: + // section. CLI (explicit files) out-ranks user-global (first-non-nil keeps + // CLI). + operatorTelemetry *TelemetrySection + // operatorMCP is the first complete operator-tier mcp: subtree. Explicit CLI // files are visited before user-global settings, so precedence is whole-block, // first-non-nil; project mcp blocks are warning-only and never captured. @@ -392,6 +401,19 @@ func (r *Resolver) OperatorOpenRouter() *OpenRouterSection { return r.operatorOpenRouter } +// OperatorProductMetricsEnabled returns the operator-tier +// telemetry.productMetrics.enabled: value (user-global + CLI only), or nil +// when none was configured. It is the SOLE accessor composition uses to +// read the product-metrics opt-out from config — by construction it never +// returns a project-tier value (a project telemetry: block is ignored with +// a WARN in loadProjectRules). nil-safe. Mirrors OperatorOpenRouter(). +func (r *Resolver) OperatorProductMetricsEnabled() *bool { + if r == nil || r.operatorTelemetry == nil || r.operatorTelemetry.ProductMetrics == nil { + return nil + } + return r.operatorTelemetry.ProductMetrics.Enabled +} + // OperatorMCP returns the complete operator-tier mcp subtree, or nil when absent. // It is metadata only and can never originate from project settings. func (r *Resolver) OperatorMCP() *MCPSection { @@ -674,6 +696,11 @@ func (r *Resolver) loadProjectRules(ws tool.WorkspaceReader) ([]governance.Rule, "openrouter: IGNORING a project-tier openrouter: block (operator-tier only — a project repo cannot steer the OpenRouter downstream provider; set openrouter in your user-global settings.yaml)", "file", src.path, "root", ws.Root()) } + if cfg.Telemetry != nil { + r.diag.Log(context.Background(), port.LevelWarn, + "telemetry: IGNORING a project-tier telemetry: block (operator-tier only — a project repo cannot change a user's own product-metrics opt-out in either direction; set telemetry in your user-global settings.yaml)", + "file", src.path, "root", ws.Root()) + } if cfg.MCP != nil { r.diag.Log(context.Background(), port.LevelWarn, "mcp: IGNORING a project-tier mcp: block (operator-tier only — a project repo cannot configure global MCP servers)", @@ -905,6 +932,9 @@ func (r *Resolver) loadUserRules(report *Report) []governance.Rule { r.captureModels(cfg.Models) // Operator-tier openrouter: same first-non-nil-keeps-CLI discipline (issue #480). r.captureOpenRouter(cfg.OpenRouter) + // Operator-tier telemetry (opt-out product metrics): same + // first-non-nil-keeps-CLI discipline as openrouter. + r.captureTelemetry(cfg.Telemetry) // Operator-tier MCP profiles: capture the complete first block; never field-merge. r.captureMCP(cfg.MCP) r.captureRetention(cfg.Retention) @@ -944,6 +974,9 @@ func (r *Resolver) loadUserRules(report *Report) []governance.Rule { r.captureModels(cfg.Models) // User-global openrouter: captured only if no higher CLI file already did. r.captureOpenRouter(cfg.OpenRouter) + // Operator-tier telemetry (opt-out product metrics): same + // first-non-nil-keeps-CLI discipline as openrouter. + r.captureTelemetry(cfg.Telemetry) // User-global MCP: captured only if no higher CLI file already did. r.captureMCP(cfg.MCP) r.captureRetention(cfg.Retention) @@ -1086,6 +1119,16 @@ func (r *Resolver) captureOpenRouter(s *OpenRouterSection) { r.operatorOpenRouter = s } +// captureTelemetry records the FIRST operator-tier telemetry: block seen +// during loadUserRules (CLI files out-rank user-global, so first-non-nil +// keeps CLI). Mirrors captureOpenRouter. +func (r *Resolver) captureTelemetry(s *TelemetrySection) { + if s == nil || r.operatorTelemetry != nil { + return + } + r.operatorTelemetry = s +} + // captureMCP records the first complete operator-tier mcp block. It is called // only by loadUserRules, whose explicit-files-before-user order defines precedence. func (r *Resolver) captureMCP(s *MCPSection) { diff --git a/internal/adapter/permconfig/schema.go b/internal/adapter/permconfig/schema.go index 957c34ebf2..33ee540f80 100644 --- a/internal/adapter/permconfig/schema.go +++ b/internal/adapter/permconfig/schema.go @@ -152,6 +152,14 @@ type Config struct { // key was absent. The composition layer reads + validates the maps; permconfig // only carries them. OpenRouter *OpenRouterSection `yaml:"openrouter"` + // Telemetry holds the OPERATOR-TIER `telemetry:` subtree (opt-out product/ + // adoption metrics). Like OpenRouter/Guardrails/Posture it is honoured ONLY + // from the user-global + CLI tiers; a project-tier file's telemetry: block + // is IGNORED with a WARN (a project repo cannot flip a user's own telemetry + // choice in either direction). Parsed STRICTLY (unknown keys error). A nil + // Telemetry means the key was absent — composition then falls through the + // DO_NOT_TRACK env var and finally defaults to enabled. + Telemetry *TelemetrySection `yaml:"telemetry"` // MCP holds named global Streamable HTTP MCP server profiles. It is strict and // OPERATOR-TIER ONLY: project files cannot choose endpoints, authentication, // credential references, or egress policy. Values are metadata only; parsing @@ -1247,6 +1255,46 @@ func (m *OpenRouterModelRoute) strictFields() map[string]any { } } +// TelemetrySection is the `telemetry:` operator-tier YAML subtree: the opt-out +// switch for community/adoption product metrics. Parsed STRICTLY (unknown +// keys error), mirroring OpenRouterSection/GuardrailsSection. +type TelemetrySection struct { + // ProductMetrics is the opt-out product/adoption metrics config. + ProductMetrics *ProductMetricsSection `yaml:"productMetrics"` +} + +func (s *TelemetrySection) strictFields() map[string]any { + return map[string]any{ + "productMetrics": newPermconfigNodePointer(&s.ProductMetrics), + } +} + +// UnmarshalYAML decodes the telemetry: mapping STRICTLY: an unknown key +// (e.g. a typo'd product-metrics:) is a parse error, same discipline as +// openrouter:/guardrails:. +func (s *TelemetrySection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry", s.strictFields()) +} + +// ProductMetricsSection is the `telemetry.productMetrics:` subtree. +type ProductMetricsSection struct { + // Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit + // false: nil = absent (composition falls through to DO_NOT_TRACK then the + // enabled-by-default posture); a non-nil value is honoured exactly. + Enabled *bool `yaml:"enabled"` +} + +func (s *ProductMetricsSection) strictFields() map[string]any { + return map[string]any{ + "enabled": newPermconfigNodePointer(&s.Enabled), + } +} + +// UnmarshalYAML decodes the productMetrics: mapping STRICTLY. +func (s *ProductMetricsSection) UnmarshalYAML(node ast.Node) error { + return decodeStrictMapping(node, "telemetry.productMetrics", s.strictFields()) +} + // UnmarshalYAML decodes an openrouter.models. entry STRICTLY. func (m *OpenRouterModelRoute) UnmarshalYAML(node ast.Node) error { return decodeStrictMapping(node, "openrouter.models[]", m.strictFields()) diff --git a/internal/adapter/permconfig/telemetry_resolve_test.go b/internal/adapter/permconfig/telemetry_resolve_test.go new file mode 100644 index 0000000000..93eac76ad4 --- /dev/null +++ b/internal/adapter/permconfig/telemetry_resolve_test.go @@ -0,0 +1,67 @@ +package permconfig + +import ( + "bytes" + "context" + "strings" + "testing" + + "github.com/stacklok/mecatl/engine/adapter/memfs" + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/slogdiag" +) + +// The telemetry: block (specifically telemetry.productMetrics.enabled) is +// OPERATOR-TIER ONLY: a project repo must never be able to flip a user's own +// product-metrics opt-out in either direction. These tests pin the operator +// capture and the project-tier WARN-ignore (the fail-closed core). Mirrors +// openrouter_test.go. + +const operatorTelemetryYAML = "telemetry:\n productMetrics:\n enabled: false\n" + +// TestOperatorProductMetricsEnabledFromCLIHonoured: an OPERATOR-TIER (CLI +// explicit) telemetry: block is read and returned by +// OperatorProductMetricsEnabled(), parsed faithfully. +func TestOperatorProductMetricsEnabledFromCLIHonoured(t *testing.T) { + env := envWithExplicit("/etc/mecatl/telemetry.yaml", operatorTelemetryYAML) + r := newWithEnv(Options{ExplicitFiles: []string{"/etc/mecatl/telemetry.yaml"}}, env) + if r == nil { + t.Fatal("resolver should be non-nil with an explicit file") + } + got := r.OperatorProductMetricsEnabled() + if got == nil || *got != false { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want explicit false", got) + } +} + +// TestOperatorProductMetricsEnabledAbsentIsNil: no telemetry: config anywhere +// yields a nil accessor result. +func TestOperatorProductMetricsEnabledAbsentIsNil(t *testing.T) { + r := newWithEnv(Options{Conventional: true}, fakeEnv()) + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Fatalf("OperatorProductMetricsEnabled() = %v, want nil (absent)", got) + } +} + +// TestProjectTierTelemetryBlockIsIgnoredWithWarn is the FAIL-CLOSED CORE: a +// PROJECT-TIER telemetry: block must NEVER become the operator config, and the +// resolver WARNs naming why (a project repo cannot change the user's own +// product-metrics opt-out). +func TestProjectTierTelemetryBlockIsIgnoredWithWarn(t *testing.T) { + var buf bytes.Buffer + diag := slogdiag.New(&buf, false, port.LevelDebug) + + ws := &countingWS{Workspace: memfs.NewWorkspace("/repo")} + ws.seed(t, projectFileMecatl, operatorTelemetryYAML) + + r := newWithEnv(Options{Conventional: true, TrustProject: true, Diagnostics: diag}, fakeEnv()) + _ = r.Resolve(context.Background(), ws) + + if got := r.OperatorProductMetricsEnabled(); got != nil { + t.Fatalf("a PROJECT-tier telemetry: block must NOT become the operator config; got %v", got) + } + log := buf.String() + if !strings.Contains(log, "IGNORING a project-tier telemetry") { + t.Fatalf("expected an ignore-WARN naming the project tier; got:\n%s", log) + } +} diff --git a/internal/adapter/permconfig/telemetry_schema_test.go b/internal/adapter/permconfig/telemetry_schema_test.go new file mode 100644 index 0000000000..c3eccbe34c --- /dev/null +++ b/internal/adapter/permconfig/telemetry_schema_test.go @@ -0,0 +1,31 @@ +package permconfig + +import "testing" + +func TestParseYAMLTelemetryProductMetricsEnabled(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enabled: false\n") + cfg, err := parseYAML(data) + if err != nil { + t.Fatalf("parseYAML: %v", err) + } + if cfg.Telemetry == nil || cfg.Telemetry.ProductMetrics == nil { + t.Fatal("Telemetry.ProductMetrics is nil") + } + if cfg.Telemetry.ProductMetrics.Enabled == nil || *cfg.Telemetry.ProductMetrics.Enabled != false { + t.Errorf("Enabled = %v, want explicit false", cfg.Telemetry.ProductMetrics.Enabled) + } +} + +func TestParseYAMLTelemetryUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productmetric:\n enabled: false\n") // typo: productmetric + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown telemetry.productmetric key, got nil") + } +} + +func TestParseYAMLTelemetryProductMetricsUnknownKeyErrors(t *testing.T) { + data := []byte("telemetry:\n productMetrics:\n enable: false\n") // typo: enable + if _, err := parseYAML(data); err == nil { + t.Fatal("expected a strict-parse error for the unknown enable key, got nil") + } +} diff --git a/internal/adapter/productmetrics/bounded_test.go b/internal/adapter/productmetrics/bounded_test.go new file mode 100644 index 0000000000..9c96ea3960 --- /dev/null +++ b/internal/adapter/productmetrics/bounded_test.go @@ -0,0 +1,212 @@ +package productmetrics + +import ( + "context" + "strings" + "testing" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +// allowedAttributeKeys is the COMPLETE set of attribute keys any instrument +// in this package may ever carry. A future change that attaches a new label +// must add it here explicitly — the same "closed set is a reviewed +// decision" discipline as internal/adapter/telemetry's attrRole. +var allowedAttributeKeys = map[string]bool{ + attrStop: true, + attrKind: true, + attrFeature: true, + attrProvider: true, + attrMode: true, + attrHadToolCall: true, + attrCategory: true, + attrOutcome: true, +} + +// sensitiveMarkers are strings injected into every field the Recorder must +// NEVER read, or (for the tool-name markers) must only ever read through a +// closed-set projection. If any of these ever shows up in a collected metric +// name or attribute value, something started emitting a field it shouldn't. +var sensitiveMarkers = []string{ + "sensitive-session-id-marker", + "secret-tool-name-marker", + "secret-tool-content-marker", + "secret-error-text-marker", + // The MCP server + remote tool names inside a namespaced mcp__ tool name: + // operator-chosen free text, which must be bucketed under the single + // literal "mcp" rather than emitted. + "evilserver", + "leak_this_name", +} + +func TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent(t *testing.T) { + r, reader := newTestRecorder(t) + // Arm time_to_first_value so its data points are collected too — an + // unarmed Recorder never records it, which would leave that instrument + // outside the walk below. + r.EnableFirstValueTracking(time.Now(), false, nil) + + // Drive every observation path with deliberately sensitive-looking data. + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopError, + Text: "sensitive-session-id-marker should never be read", + Error: "secret-error-text-marker: connection to 10.0.0.5 failed", + Usage: session.Usage{InputTokens: 1, OutputTokens: 1}, + }, + }) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + r.ToolCall( + session.SessionID("sensitive-session-id-marker"), + session.ToolCall{Name: "secret-tool-name-marker"}, + session.ToolResult{Content: "secret-tool-content-marker", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + // The run-aware path, with an MCP-namespaced name whose server and remote + // tool halves are both operator-chosen free text. EvSessionInit is driven + // FIRST with the SAME RunID so this run is genuinely tracked — exercising + // run_duration and tool_calls_per_run too, not just category/outcome. + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-x"}) + r.ToolCallForRun( + "run-x", + session.SessionID("sensitive-session-id-marker"), + session.ToolCall{Name: "mcp__evilserver__leak_this_name"}, + session.ToolResult{Content: "secret-tool-content-marker"}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-x", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + r.Heartbeat(FeatureSnapshot{ + Memory: true, Guardrails: true, MCP: true, Scheduling: true, + Provider: ProviderOther, Mode: ModeK8s, + }) + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + + totalMetrics := 0 + for _, sm := range rm.ScopeMetrics { + totalMetrics += len(sm.Metrics) + } + // wantInstrumentCount is NewRecorder's exact registered instrument count + // (heartbeat, feature_enabled, provider_configured, deployment_mode, + // sessions_started, runs_completed, tool_calls, tokens, subagent_used, + // team_used, run_duration, tool_calls_per_run, time_to_first_value). + // Asserting the EXACT count, not a floor, means a future instrument this + // test's driving code doesn't happen to exercise fails loudly here rather + // than silently passing the walk below vacuously. + const wantInstrumentCount = 13 + if totalMetrics != wantInstrumentCount { + t.Fatalf("collected %d metrics, want exactly %d (the full mecatl.product.* instrument set) — the walk below would otherwise pass vacuously on a new, unexercised instrument", totalMetrics, wantInstrumentCount) + } + + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + assertNoSensitiveSubstring(t, md.Name) + for _, attrs := range dataPointAttributes(t, md.Name, md.Data) { + iter := attrs.Iter() + for iter.Next() { + kv := iter.Attribute() + key := string(kv.Key) + if !allowedAttributeKeys[key] { + t.Errorf("metric %s carries attribute key %q, not in allowedAttributeKeys", md.Name, key) + } + // Every value must be a STRING: a non-string value would + // slip past the sensitive-substring walk below with an + // empty AsString(), making this guard vacuous for it. + if kv.Value.Type() != attribute.STRING { + t.Errorf("metric %s attribute %q has value type %v, want STRING", md.Name, key, kv.Value.Type()) + } + assertNoSensitiveSubstring(t, kv.Value.AsString()) + } + } + // mecatl.product.tool_calls carries exactly the two bounded + // tool-call keys — the category value can only ever be a built-in + // tool's own name, "mcp", or "other" (see toolCategory), so no + // tool identity beyond mecatl's own fixed catalog can attach. + if md.Name == "mecatl.product.tool_calls" { + for _, attrs := range dataPointAttributes(t, md.Name, md.Data) { + if _, ok := attrs.Value(attrCategory); !ok { + t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrCategory, attrs) + } + if _, ok := attrs.Value(attrOutcome); !ok { + t.Errorf("mecatl.product.tool_calls data point is missing the %s attribute: %v", attrOutcome, attrs) + } + if attrs.Len() != 2 { + t.Errorf("mecatl.product.tool_calls data point carries %d attributes, want exactly 2: %v", + attrs.Len(), attrs) + } + } + } + } + } +} + +// dataPointAttributes returns every collected data point's attribute set, +// whatever aggregation the instrument uses. The walk must cover histograms as +// well as sums: this package publishes run_duration, tool_calls_per_run and +// time_to_first_value as histograms, and an aggregation this helper did not +// know about would silently drop that instrument out of the no-PII guard — +// so an unrecognised shape is a hard failure, never a skip. +func dataPointAttributes(t *testing.T, name string, agg metricdata.Aggregation) []attribute.Set { + t.Helper() + var out []attribute.Set + switch data := agg.(type) { + case metricdata.Sum[int64]: + for _, dp := range data.DataPoints { + out = append(out, dp.Attributes) + } + case metricdata.Histogram[int64]: + for _, dp := range data.DataPoints { + out = append(out, dp.Attributes) + } + case metricdata.Histogram[float64]: + for _, dp := range data.DataPoints { + out = append(out, dp.Attributes) + } + default: + t.Fatalf("metric %s: unhandled aggregation %T — add it to dataPointAttributes so the no-PII walk keeps covering it", name, agg) + } + return out +} + +// TestToolCategoryOnlyEmitsClosedSetValues is the direct unit-level guard on +// the one projection that reads a tool name: whatever it is fed, the output +// must be a member of builtinToolCategories ∪ {"mcp", "other"}. +func TestToolCategoryOnlyEmitsClosedSetValues(t *testing.T) { + inputs := []string{ + "", "Read", "Bash", "mcp__evilserver__leak_this_name", "mcp__", "mcp_", + "secret-tool-name-marker", "MCP__X__Y", "read", "Read ", + "agent-def-derived-name", strings.Repeat("x", 4096), + } + for _, in := range inputs { + got := toolCategory(in) + if got == categoryMCP || got == categoryOther { + continue + } + if !builtinToolCategories[got] { + t.Errorf("toolCategory(%q) = %q, which is outside the closed set (builtins ∪ {%q, %q})", + in, got, categoryMCP, categoryOther) + } + } +} + +func assertNoSensitiveSubstring(t *testing.T, s string) { + t.Helper() + for _, marker := range sensitiveMarkers { + if strings.Contains(s, marker) { + t.Errorf("value %q contains sensitive marker %q", s, marker) + } + } +} diff --git a/internal/adapter/productmetrics/config.go b/internal/adapter/productmetrics/config.go new file mode 100644 index 0000000000..db188ad9ff --- /dev/null +++ b/internal/adapter/productmetrics/config.go @@ -0,0 +1,96 @@ +// Package productmetrics is a fully independent, opt-out-by-default OTel +// metrics adapter reporting bounded adoption/usage counters to Stacklok's +// public metrics collector. It shares no import, struct, MeterProvider, or +// destination with internal/adapter/telemetry (mecatl's operator-facing +// observability pipeline) — the two are combined only at the composition +// edge (internal/cliconfig), by fanning both into the engine's +// port.EventSink/port.ToolCallRecorder seams. +// +// Every exported type in this package that can become a metric attribute is +// a closed Go string-alias enum. Nothing here carries a session id, model +// id/alias, tool or MCP-server name, file path, or free text. +package productmetrics + +// Feature is the closed set of major toggleable features reported at +// heartbeat time. Never a def/model/tool name — only these four values. +type Feature string + +// The four closed Feature values. +const ( + FeatureMemory Feature = "memory" + FeatureGuardrails Feature = "guardrails" + FeatureMCP Feature = "mcp" + FeatureScheduling Feature = "scheduling" +) + +// ProviderFamily is the closed set of configured LLM provider families. +// Never a model id or alias. +type ProviderFamily string + +// The four closed ProviderFamily values. +const ( + ProviderAnthropic ProviderFamily = "anthropic" + ProviderOpenAI ProviderFamily = "openai" + ProviderOpenRouter ProviderFamily = "openrouter" + ProviderOther ProviderFamily = "other" +) + +// DeploymentMode is the closed set of process shapes. +type DeploymentMode string + +// The three closed DeploymentMode values. +const ( + ModeInteractive DeploymentMode = "interactive" + ModeHeadless DeploymentMode = "headless" + ModeK8s DeploymentMode = "k8s" +) + +// Binary is the closed set of the four mecatl entry points. +type Binary string + +// The four closed Binary values. +const ( + BinaryMecated Binary = "mecated" + BinaryMecatui Binary = "mecatui" + BinaryMecatequi Binary = "mecatequi" + BinaryMecak8s Binary = "mecak8s" +) + +// FeatureSnapshot is a closed-shape, read-only snapshot of which major +// features are enabled and which provider family / deployment mode this +// process runs as. It carries no free text and no model id/alias. +type FeatureSnapshot struct { + Memory bool + Guardrails bool + MCP bool + Scheduling bool + Provider ProviderFamily + Mode DeploymentMode +} + +// enabled returns every Feature mapped to whether this snapshot reports it +// enabled. It is the single place Heartbeat iterates, so adding a Feature +// const without adding it here is caught by the exhaustiveness this map +// documents (and by TestFeatureSnapshotEnabledIsClosedAndBounded above). +func (s FeatureSnapshot) enabled() map[Feature]bool { + return map[Feature]bool{ + FeatureMemory: s.Memory, + FeatureGuardrails: s.Guardrails, + FeatureMCP: s.MCP, + FeatureScheduling: s.Scheduling, + } +} + +// Config configures a Provider/Recorder pair for one process. +type Config struct { + // Binary identifies which of the four entry points this process is. + Binary Binary + // Version is the mecatl build version (resource attribute service.version). + Version string + // InstallID is this process's persisted (or externally-provisioned, for + // mecak8s — see Task 7) anonymous install identifier. Reinstated as a + // resource attribute after being sized and accepted: ~$1,930/month at + // 100K installs under worst-case 24/7 uptime on the actual AMP pricing + // model (see the ADR's updated cost-analysis section, Task 8). + InstallID string +} diff --git a/internal/adapter/productmetrics/config_test.go b/internal/adapter/productmetrics/config_test.go new file mode 100644 index 0000000000..1107c03a75 --- /dev/null +++ b/internal/adapter/productmetrics/config_test.go @@ -0,0 +1,23 @@ +package productmetrics + +import "testing" + +func TestFeatureSnapshotEnabledIsClosedAndBounded(t *testing.T) { + snap := FeatureSnapshot{Memory: true, MCP: true, Provider: ProviderAnthropic, Mode: ModeInteractive} + got := snap.enabled() + + want := map[Feature]bool{ + FeatureMemory: true, + FeatureGuardrails: false, + FeatureMCP: true, + FeatureScheduling: false, + } + if len(got) != len(want) { + t.Fatalf("enabled() returned %d entries, want %d (%v)", len(got), len(want), got) + } + for f, v := range want { + if got[f] != v { + t.Errorf("enabled()[%q] = %v, want %v", f, got[f], v) + } + } +} diff --git a/internal/adapter/productmetrics/dryrun.go b/internal/adapter/productmetrics/dryrun.go new file mode 100644 index 0000000000..a2a010e47d --- /dev/null +++ b/internal/adapter/productmetrics/dryrun.go @@ -0,0 +1,157 @@ +package productmetrics + +import ( + "context" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// DryRunRecorder implements the same ports as Recorder (port.EventSink + +// port.ToolCallRecorder + port.RunAwareToolCallRecorder) but logs every +// would-be observation via an injected port.Diagnostics instead of exporting +// it over OTLP — the --product-metrics-dry-run audit path, so a skeptical +// operator can see exactly what this pipeline would have sent without +// trusting the docs. It logs ONLY the same bounded fields Recorder ever reads +// (event type, stop reason, had_tool_call, run_duration, tool_calls_per_run, +// the closed-set tool category and outcome, token counts by kind, +// feature/provider/mode enum values) — never a session id, a raw tool name, +// or free-text content, mirroring Recorder's own privacy discipline exactly. +// +// The audit path must stay in LOCKSTEP with Recorder: an attribute Recorder +// attaches but DryRunRecorder omits makes this surface understate what is +// sent, which is the one thing it exists to rule out. +// +// time_to_first_value is the ONE deliberate divergence, by design: Recorder's +// version is a once-ever, install-scoped, persisted-marker sample (armed via +// EnableFirstValueTracking, composition-only). DryRunRecorder has — and must +// have — NO persistent state (no XDG_STATE_HOME reads/writes; it stays a +// stateless, one-shot audit tool per the ADR), so it cannot reproduce "at +// most once, ever" across process restarts. Instead it logs a would-be +// time_to_first_value observation on EVERY run that qualifies (StopEndTurn + +// at least one successful tool call), not just the first one this process +// happens to see. That is MORE verbose than what Recorder would actually +// send, but it is the honest, simpler choice for a diagnostic surface whose +// job is "show what COULD be sent" — an operator sees every candidate moment +// rather than only whichever one a real install's persisted marker allowed. +type DryRunRecorder struct { + diag port.Diagnostics + perRun *perRunTracker +} + +// Compile-time interface checks. +var ( + _ port.EventSink = (*DryRunRecorder)(nil) + _ port.ToolCallRecorder = (*DryRunRecorder)(nil) + _ port.RunAwareToolCallRecorder = (*DryRunRecorder)(nil) +) + +// NewDryRunRecorder builds a DryRunRecorder over the given Diagnostics sink. +func NewDryRunRecorder(diag port.Diagnostics) *DryRunRecorder { + return &DryRunRecorder{diag: diag, perRun: newPerRunTracker()} +} + +// Emit logs the bounded event type (and, for EvResult, the stop reason, +// had_tool_call, run_duration_seconds, tool_calls_per_run, token counts by +// kind, and a separate would-be time_to_first_value line when the run +// qualifies) — the exact same underlying facts Recorder.Emit reads. +func (d *DryRunRecorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record sessions_started+1") + d.perRun.markStarted(ev.RunID) + case session.EvResult: + st, tracked := d.perRun.finish(ev.RunID) + d.emitResult(ctx, ev.Result, st, tracked) + case session.EvSubagentStart: + if d.perRun.markFamilyUsed(ev.RunID, familySubagent) { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record subagent_used+1") + } + case session.EvTeamStart: + if d.perRun.markFamilyUsed(ev.RunID, familyTeam) { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record team_used+1") + } + } +} + +func (d *DryRunRecorder) emitResult(ctx context.Context, res *session.ResultPayload, st perRunState, tracked bool) { + stop := session.StopNone + if res != nil { + stop = res.Stop + } + + fields := []any{ + "stop", string(stop), + attrHadToolCall, st.hadToolCall, + } + // tool_calls_per_run mirrors Recorder.recordResult's tracked guard: an + // untracked run (no RunID, or a RunID this recorder never saw an + // EvSessionInit/tool call for) would otherwise report a fabricated 0. + if tracked { + fields = append(fields, "tool_calls_per_run", st.toolCallCount) + } + // run_duration is only meaningful when this recorder actually observed the + // run's EvSessionInit (mirroring Recorder.recordResult's zero-startedAt + // guard) — a run whose start this process missed reports no duration + // rather than a nonsense one measured from time.Time{}. + if !st.startedAt.IsZero() { + fields = append(fields, "run_duration_seconds", time.Since(st.startedAt).Seconds()) + } + + msg := "product metrics (dry-run): would record runs_completed" + if res != nil { + u := res.Usage + msg = "product metrics (dry-run): would record runs_completed + tokens" + fields = append(fields, + "input_tokens", u.InputTokens, + "output_tokens", u.OutputTokens, + "cache_read_tokens", u.CacheReadTokens, + "cache_write_tokens", u.CacheWriteTokens, + "reasoning_tokens", u.ReasoningTokens) + } + d.diag.Log(ctx, port.LevelInfo, msg, fields...) + + // See the DryRunRecorder doc comment: unlike Recorder's once-ever, + // persisted-marker time_to_first_value, dry-run logs this on EVERY + // qualifying run (no state to track "first" against) — the honest + // simplification for a stateless audit surface. + if stop == session.StopEndTurn && st.hadToolCall { + d.diag.Log(ctx, port.LevelInfo, "product metrics (dry-run): would record time_to_first_value") + } +} + +// ToolCall logs the run-less form, matching Recorder.ToolCall's delegation. +func (d *DryRunRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + d.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun logs the two bounded tool-call attributes Recorder attaches — +// the closed-set category (never the raw name for anything outside mecatl's +// own catalog, never an MCP server/tool name) and the outcome — and tallies +// the same per-run state, matching Recorder.ToolCallForRun exactly. +func (d *DryRunRecorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + outcome := outcomeSuccess + if result.IsError { + outcome = outcomeError + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record tool_calls+1", + attrCategory, toolCategory(call.Name), + attrOutcome, outcome) + d.perRun.markToolCall(runID, result.IsError) +} + +// Heartbeat logs the closed-enum feature/provider/mode signal, matching +// Recorder.Heartbeat's fields exactly. +func (d *DryRunRecorder) Heartbeat(snap FeatureSnapshot) { + enabled := make([]string, 0, 4) + for f, on := range snap.enabled() { + if on { + enabled = append(enabled, string(f)) + } + } + d.diag.Log(context.Background(), port.LevelInfo, "product metrics (dry-run): would record heartbeat", + "features_enabled", enabled, + "provider_family", string(snap.Provider), + "deployment_mode", string(snap.Mode)) +} diff --git a/internal/adapter/productmetrics/dryrun_test.go b/internal/adapter/productmetrics/dryrun_test.go new file mode 100644 index 0000000000..36fce7fd2a --- /dev/null +++ b/internal/adapter/productmetrics/dryrun_test.go @@ -0,0 +1,272 @@ +package productmetrics + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +type capturingDiag struct { + lines []string + args [][]any +} + +func (c *capturingDiag) Log(_ context.Context, _ port.Level, msg string, args ...any) { + c.lines = append(c.lines, msg) + c.args = append(c.args, args) +} +func (c *capturingDiag) With(...any) port.Diagnostics { return c } + +func TestDryRunRecorderLogsInsteadOfExporting(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "sensitive-name"}, session.ToolResult{Content: "sensitive-content"}, 0, time.Millisecond) + + if len(diag.lines) != 2 { + t.Fatalf("got %d logged lines, want 2: %v", len(diag.lines), diag.lines) + } + for i, line := range diag.lines { + if strings.Contains(line, "sensitive") { + t.Errorf("dry-run log line leaked sensitive content: %q", line) + } + for _, a := range diag.args[i] { + if s, ok := a.(string); ok && strings.Contains(s, "sensitive") { + t.Errorf("dry-run log args leaked sensitive content: %v", diag.args[i]) + } + } + } +} + +// TestDryRunRecorderImplementsPorts pins the compile-time interface guards +// (var _ port.EventSink = ...) via an explicit assignment, so a signature +// drift on either port fails this test with a clear message rather than only +// the package-level var block. +func TestDryRunRecorderImplementsPorts(_ *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + var _ port.EventSink = r + var _ port.ToolCallRecorder = r + var _ port.RunAwareToolCallRecorder = r +} + +// TestDryRunRecorderMirrorsRecorderToolCallAttributes pins the lockstep +// contract: the audit path must log the SAME bounded attributes Recorder +// attaches (an audit surface that understates what is sent defeats its own +// purpose), and the category must still be the closed-set projection — never +// the raw name, never an MCP server/tool name. +func TestDryRunRecorderMirrorsRecorderToolCallAttributes(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.ToolCallForRun("run-1", session.SessionID("s"), + session.ToolCall{Name: "mcp__evilserver__leak_this_name"}, + session.ToolResult{IsError: true}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + if len(diag.args) != 2 { + t.Fatalf("got %d logged lines, want 2: %v", len(diag.lines), diag.lines) + } + if !hasArg(diag.args[0], attrCategory, categoryMCP) { + t.Errorf("tool_calls dry-run log args = %v, want %s=%s", diag.args[0], attrCategory, categoryMCP) + } + if !hasArg(diag.args[0], attrOutcome, outcomeError) { + t.Errorf("tool_calls dry-run log args = %v, want %s=%s", diag.args[0], attrOutcome, outcomeError) + } + if !hasArg(diag.args[1], attrHadToolCall, false) { + t.Errorf("runs_completed dry-run log args = %v, want %s=false (the only tool call errored)", diag.args[1], attrHadToolCall) + } + for _, args := range diag.args { + for _, a := range args { + if s, ok := a.(string); ok && (strings.Contains(s, "evilserver") || strings.Contains(s, "leak_this_name")) { + t.Errorf("dry-run log leaked an MCP server/tool name: %v", args) + } + } + } +} + +// hasArg reports whether a Diagnostics key/value arg slice carries key=want. +func hasArg(args []any, key string, want any) bool { + for i := 0; i+1 < len(args); i += 2 { + if k, ok := args[i].(string); ok && k == key && args[i+1] == want { + return true + } + } + return false +} + +// TestDryRunRecorderResultNeverLeaksFreeText covers the EvResult branch (not +// exercised by the brief's original test) with a non-nil ResultPayload, +// asserting the log carries only the bounded stop/token fields and never the +// free-text Text/Error fields on ResultPayload. +func TestDryRunRecorderResultNeverLeaksFreeText(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopEndTurn, + Text: "sensitive final answer text", + Usage: session.Usage{ + InputTokens: 10, + OutputTokens: 20, + }, + }, + }) + + if len(diag.lines) != 1 { + t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) + } + for _, a := range diag.args[0] { + if s, ok := a.(string); ok && strings.Contains(s, "sensitive") { + t.Errorf("dry-run log args leaked ResultPayload.Text: %v", diag.args[0]) + } + } +} + +func TestDryRunRecorderHeartbeatLogsOnlyEnums(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Heartbeat(FeatureSnapshot{ + Memory: true, + Guardrails: false, + MCP: true, + Scheduling: false, + Provider: ProviderAnthropic, + Mode: ModeInteractive, + }) + + if len(diag.lines) != 1 { + t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) + } +} + +// TestDryRunRecorderLogsRunDurationAndToolCallsPerRun pins the two +// runs_completed fields this task adds: run_duration_seconds (present only +// when this recorder observed the run's EvSessionInit) and +// tool_calls_per_run (the run's total tool-call count, successful or not). +func TestDryRunRecorderLogsRunDurationAndToolCallsPerRun(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + // diag.args: [0]=sessions_started, [1]=Read tool_calls, [2]=Bash tool_calls, [3]=runs_completed. + if len(diag.args) != 4 { + t.Fatalf("got %d logged lines, want 4: %v", len(diag.lines), diag.lines) + } + runsCompleted := diag.args[3] + if !hasArg(runsCompleted, "tool_calls_per_run", int64(2)) { + t.Errorf("runs_completed dry-run log args = %v, want tool_calls_per_run=2", runsCompleted) + } + if !hasArg(runsCompleted, attrHadToolCall, true) { + t.Errorf("runs_completed dry-run log args = %v, want %s=true (one successful call)", runsCompleted, attrHadToolCall) + } + found := false + for i := 0; i+1 < len(runsCompleted); i += 2 { + if k, ok := runsCompleted[i].(string); ok && k == "run_duration_seconds" { + found = true + if _, ok := runsCompleted[i+1].(float64); !ok { + t.Errorf("run_duration_seconds arg = %v (%T), want float64", runsCompleted[i+1], runsCompleted[i+1]) + } + } + } + if !found { + t.Errorf("runs_completed dry-run log args = %v, want a run_duration_seconds field (EvSessionInit was observed)", runsCompleted) + } +} + +// TestDryRunRecorderOmitsRunDurationWhenSessionInitUnseen covers a run whose +// EvSessionInit this recorder never observed (e.g. it started before this +// process attached) — run_duration_seconds must be omitted rather than +// reporting a bogus duration measured from a zero time. +func TestDryRunRecorderOmitsRunDurationWhenSessionInitUnseen(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-never-started", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + if len(diag.args) != 1 { + t.Fatalf("got %d logged lines, want 1: %v", len(diag.lines), diag.lines) + } + for i := 0; i+1 < len(diag.args[0]); i += 2 { + if k, ok := diag.args[0][i].(string); ok && k == "run_duration_seconds" { + t.Errorf("run_duration_seconds present for a run whose EvSessionInit was never observed: %v", diag.args[0]) + } + } +} + +// TestDryRunRecorderLogsTimeToFirstValueEveryQualifyingRun pins this task's +// documented design choice: unlike Recorder's once-ever, persisted-marker +// time_to_first_value, the stateless dry-run path logs a would-be +// time_to_first_value observation on EVERY run that qualifies (StopEndTurn + +// at least one successful tool call) — proven here across TWO separate +// qualifying runs, both logging it. +func TestDryRunRecorderLogsTimeToFirstValueEveryQualifyingRun(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + qualify := func(runID string) { + r.ToolCallForRun(runID, session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: runID, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + } + qualify("run-a") + qualify("run-b") + + count := 0 + for _, line := range diag.lines { + if line == "product metrics (dry-run): would record time_to_first_value" { + count++ + } + } + if count != 2 { + t.Errorf("got %d time_to_first_value lines across two qualifying runs, want 2 (dry-run logs every qualifying run, not once-ever): %v", count, diag.lines) + } +} + +// TestDryRunRecorderOmitsTimeToFirstValueWhenNotQualifying covers the two +// non-qualifying shapes: no successful tool call, and a non-StopEndTurn stop. +func TestDryRunRecorderOmitsTimeToFirstValueWhenNotQualifying(t *testing.T) { + diag := &capturingDiag{} + r := NewDryRunRecorder(diag) + + // No tool call at all: StopEndTurn but hadToolCall stays false. + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-no-tools", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + // A successful tool call but a non-EndTurn stop. + r.ToolCallForRun("run-error-stop", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-error-stop", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + for _, line := range diag.lines { + if line == "product metrics (dry-run): would record time_to_first_value" { + t.Errorf("time_to_first_value logged for a non-qualifying run: %v", diag.lines) + } + } +} diff --git a/internal/adapter/productmetrics/firstvalue.go b/internal/adapter/productmetrics/firstvalue.go new file mode 100644 index 0000000000..2eabef07d6 --- /dev/null +++ b/internal/adapter/productmetrics/firstvalue.go @@ -0,0 +1,130 @@ +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// firstValueMarkerRelPath is the state-dir-relative path to a bare marker file +// recording whether this install's first meaningful-and-successful run has +// already been observed — so mecatl.product.time_to_first_value is recorded at +// most once per install, ever. It lives beside the install-id file under +// XDG_STATE_HOME (machine-written runtime state, not human config) and mirrors +// installid.go's injected-filesystem shape exactly. +const firstValueMarkerRelPath = "mecatl/first-value-recorded" + +// firstValueMarkerPath resolves the marker's absolute path, failing closed when +// no state directory can be resolved at all. +func firstValueMarkerPath(env xdgconfig.ResolveEnv) (string, error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return "", fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + return filepath.Join(base, firstValueMarkerRelPath), nil +} + +// FirstValueRecorded reports whether this install's time_to_first_value sample +// was already recorded by some EARLIER process. It is a pure READ: it never +// creates the marker, because the marker means "the qualifying run happened", +// and process startup is not that moment (a process may exit without ever +// having one). The marker is written later, by LoadOrCreateFirstValueMarker, +// at the instant the qualifying run is observed. +// +// readFile is injected for testing; FirstValueRecordedDefault binds the real +// filesystem. +func FirstValueRecorded(env xdgconfig.ResolveEnv, readFile func(string) ([]byte, error)) (bool, error) { + path, err := firstValueMarkerPath(env) + if err != nil { + return false, err + } + if readFile == nil { + return false, nil + } + if _, rerr := readFile(path); rerr != nil { + return false, nil + } + return true, nil +} + +// FirstValueRecordedDefault binds FirstValueRecorded to the real process +// environment and filesystem. +func FirstValueRecordedDefault() (bool, error) { + return FirstValueRecorded(xdgconfig.OSEnv, os.ReadFile) +} + +// LoadOrCreateFirstValueMarker reports whether this install's first-value +// moment was already marked (already == true), creating the marker (and +// returning already == false) the first time it is called. It is the WRITE +// half, called at the moment a qualifying run is observed — not at startup. +// Once created, the marker is never removed automatically; deleting it (like +// the install-id file) resets the install and lets time_to_first_value fire +// once more. +// +// The create is an ATOMIC cross-process claim, not a read-then-write check: +// createExclusive must fail when the marker already exists (os.IsExist), +// e.g. via O_CREATE|O_EXCL. Two processes racing this call therefore never +// both observe already==false — exactly one create wins, and the loser +// reliably reports already==true, even when the two calls are strictly +// sequential rather than concurrent (the marker created by an earlier +// process's call is still there when a later process's call runs). +// firstValueTracker.claim() depends on this: it treats a false "won" from +// its recordFn as "another process already has this install's one sample" +// and skips recording, so a non-atomic check-then-act here would silently +// let two processes each record a sample. +// +// mkdirAll/createExclusive are injected for testing; +// LoadOrCreateFirstValueMarkerDefault binds the real filesystem. +func LoadOrCreateFirstValueMarker( + env xdgconfig.ResolveEnv, + mkdirAll func(string, os.FileMode) error, + createExclusive func(string, []byte, os.FileMode) error, +) (already bool, err error) { + path, err := firstValueMarkerPath(env) + if err != nil { + return false, err + } + + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if createExclusive == nil { + return false, nil + } + switch cerr := createExclusive(path, []byte("1"), 0o600); { + case cerr == nil: + return false, nil + case os.IsExist(cerr): + return true, nil + default: + return false, fmt.Errorf("productmetrics: write first-value marker: %w", cerr) + } +} + +// createFileExclusive creates path only if it does not already exist, +// returning an os.IsExist-satisfying error otherwise (O_CREATE|O_EXCL) — the +// atomic cross-process claim LoadOrCreateFirstValueMarker's contract +// depends on; a plain os.WriteFile (create-or-truncate) would let two +// concurrent callers both "win". +func createFileExclusive(path string, data []byte, perm os.FileMode) error { + f, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, perm) + if err != nil { + return err + } + _, werr := f.Write(data) + cerr := f.Close() + if werr != nil { + return werr + } + return cerr +} + +// LoadOrCreateFirstValueMarkerDefault binds LoadOrCreateFirstValueMarker to the +// real process environment and filesystem. +func LoadOrCreateFirstValueMarkerDefault() (already bool, err error) { + return LoadOrCreateFirstValueMarker(xdgconfig.OSEnv, os.MkdirAll, createFileExclusive) +} diff --git a/internal/adapter/productmetrics/firstvalue_test.go b/internal/adapter/productmetrics/firstvalue_test.go new file mode 100644 index 0000000000..785e3b2c7e --- /dev/null +++ b/internal/adapter/productmetrics/firstvalue_test.go @@ -0,0 +1,272 @@ +package productmetrics + +import ( + "context" + "errors" + "os" + "testing" + "time" + + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func testResolveEnv() xdgconfig.ResolveEnv { + return xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } +} + +// fakeMarkerFS is the injected filesystem the marker helpers are driven over. +type fakeMarkerFS struct { + written map[string][]byte +} + +func newFakeMarkerFS() *fakeMarkerFS { return &fakeMarkerFS{written: map[string][]byte{}} } + +func (f *fakeMarkerFS) readFile(p string) ([]byte, error) { + if d, ok := f.written[p]; ok { + return d, nil + } + return nil, os.ErrNotExist +} + +// createExclusive is the fake's atomic-claim primitive, mirroring the real +// createFileExclusive's O_CREATE|O_EXCL contract: it fails with an +// os.IsExist-satisfying error when the path already exists, so tests can +// exercise the SAME "second caller loses" semantics the real filesystem +// enforces via one flag. +func (f *fakeMarkerFS) createExclusive(p string, d []byte, _ os.FileMode) error { + if _, ok := f.written[p]; ok { + return os.ErrExist + } + f.written[p] = d + return nil +} + +func (*fakeMarkerFS) mkdirAll(string, os.FileMode) error { return nil } + +func TestLoadOrCreateFirstValueMarkerFirstTimeReportsNotYetRecorded(t *testing.T) { + env, fs := testResolveEnv(), newFakeMarkerFS() + + already, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive) + if err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) + } + if already { + t.Error("already = true on first call, want false") + } + + already2, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive) + if err != nil { + t.Fatalf("second LoadOrCreateFirstValueMarker: %v", err) + } + if !already2 { + t.Error("already = false on second call, want true") + } +} + +// TestLoadOrCreateFirstValueMarkerClaimIsAtomicAcrossCallers pins the fix for +// the discarded-ownership-result finding: TWO callers racing (or even just +// calling sequentially before either has observed the other) the SAME +// createExclusive-backed marker must have EXACTLY ONE winner (already == +// false) and every other caller must observe already == true — never both +// reporting false, which would let two processes each record the "at most +// once per install, ever" sample. +func TestLoadOrCreateFirstValueMarkerClaimIsAtomicAcrossCallers(t *testing.T) { + env, fs := testResolveEnv(), newFakeMarkerFS() + + var wins int + const callers = 8 + for i := 0; i < callers; i++ { + already, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive) + if err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker (call %d): %v", i, err) + } + if !already { + wins++ + } + } + if wins != 1 { + t.Errorf("wins = %d across %d calls, want exactly 1", wins, callers) + } +} + +func TestLoadOrCreateFirstValueMarkerFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + if _, err := LoadOrCreateFirstValueMarker(env, nil, nil); err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } + if _, err := FirstValueRecorded(env, nil); err == nil { + t.Fatal("expected FirstValueRecorded to error when no state dir can be resolved, got nil") + } +} + +// FirstValueRecorded is a pure read: a startup check must NOT create the +// marker, or an install whose first process never has a qualifying run would +// lose its one sample forever. +func TestFirstValueRecordedNeverCreatesTheMarker(t *testing.T) { + env, fs := testResolveEnv(), newFakeMarkerFS() + + already, err := FirstValueRecorded(env, fs.readFile) + if err != nil { + t.Fatalf("FirstValueRecorded: %v", err) + } + if already { + t.Error("already = true with no marker present, want false") + } + if len(fs.written) != 0 { + t.Fatalf("FirstValueRecorded wrote %v, want no writes", fs.written) + } + + if _, err := LoadOrCreateFirstValueMarker(env, fs.mkdirAll, fs.createExclusive); err != nil { + t.Fatalf("LoadOrCreateFirstValueMarker: %v", err) + } + already, err = FirstValueRecorded(env, fs.readFile) + if err != nil { + t.Fatalf("FirstValueRecorded after marking: %v", err) + } + if !already { + t.Error("already = false after the marker was created, want true") + } +} + +// firstValueHistogram returns the collected time_to_first_value data points, +// or nil when the instrument recorded nothing at all. +func firstValueHistogram(t *testing.T, agg metricdata.Aggregation, present bool) []metricdata.HistogramDataPoint[float64] { + t.Helper() + if !present { + return nil + } + hist, ok := agg.(metricdata.Histogram[float64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[float64]", agg) + } + return hist.DataPoints +} + +func qualifyingRun(t *testing.T, r *Recorder, runID string) { + t.Helper() + r.ToolCallForRun(runID, session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: runID, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) +} + +func TestRecorderRecordsTimeToFirstValueOnceOnly(t *testing.T) { + r, reader := newTestRecorder(t) + var marks int + r.EnableFirstValueTracking(time.Now().Add(-90*time.Second), false, func() (bool, error) { marks++; return true, nil }) + + qualifyingRun(t, r, "run-1") + qualifyingRun(t, r, "run-2") + + agg, present := collect(t, reader)["mecatl.product.time_to_first_value"] + dps := firstValueHistogram(t, agg, present) + if len(dps) != 1 || dps[0].Count != 1 { + t.Fatalf("expected exactly one sample, got %+v", dps) + } + if dps[0].Sum < 90 { + t.Errorf("recorded duration %v s, want at least the 90 s since firstSeenAt", dps[0].Sum) + } + if marks != 1 { + t.Errorf("marker persisted %d times, want exactly 1", marks) + } +} + +// TestRecorderSkipsTimeToFirstValueWhenRecordFnLosesTheClaim pins the fix for +// the discarded-ownership-result finding at the Recorder level: even though +// THIS process's in-memory tracker had not yet observed the metric as +// recorded (alreadyRecorded=false at arm time — the cross-process race +// window), recordFn's own atomic claim can still report won=false (another +// process's LoadOrCreateFirstValueMarker call got there first). The +// Recorder must honor that and emit NOTHING, not record a duplicate sample +// just because ITS in-memory state hadn't caught up. +func TestRecorderSkipsTimeToFirstValueWhenRecordFnLosesTheClaim(t *testing.T) { + r, reader := newTestRecorder(t) + var calls int + r.EnableFirstValueTracking(time.Now().Add(-time.Second), false, func() (bool, error) { + calls++ + return false, nil // another process already won the cross-process claim. + }) + + qualifyingRun(t, r, "run-1") + + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded despite recordFn reporting won=false: %+v", agg) + } + if calls != 1 { + t.Errorf("recordFn called %d times, want exactly 1", calls) + } + + // A second qualifying run must not retry the claim: this process already + // made its one attempt. + qualifyingRun(t, r, "run-2") + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded on a second run after losing the claim: %+v", agg) + } + if calls != 1 { + t.Errorf("recordFn called %d times after a second run, want still exactly 1 (no retry)", calls) + } +} + +func TestRecorderSkipsTimeToFirstValueWhenAlreadyRecorded(t *testing.T) { + r, reader := newTestRecorder(t) + var marks int + r.EnableFirstValueTracking(time.Now().Add(-time.Second), true, func() (bool, error) { marks++; return true, nil }) + + qualifyingRun(t, r, "run-1") + + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded despite alreadyRecorded=true: %+v", agg) + } + if marks != 0 { + t.Errorf("marker persisted %d times, want 0", marks) + } +} + +func TestRecorderSkipsTimeToFirstValueWhenNotArmed(t *testing.T) { + r, reader := newTestRecorder(t) + qualifyingRun(t, r, "run-1") + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded without EnableFirstValueTracking: %+v", agg) + } +} + +// A run that ended cleanly but took no action, and a run that acted but did not +// end cleanly, are both non-qualifying: the metric measures time to the first +// run that BOTH acted and succeeded. +func TestRecorderTimeToFirstValueRequiresToolCallAndCleanStop(t *testing.T) { + r, reader := newTestRecorder(t) + r.EnableFirstValueTracking(time.Now().Add(-time.Second), false, nil) + + // Clean stop, no tool call. + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + // Tool call, non-clean stop. + r.ToolCallForRun("run-2", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-2", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + if agg, present := collect(t, reader)["mecatl.product.time_to_first_value"]; present { + t.Fatalf("time_to_first_value recorded for a non-qualifying run: %+v", agg) + } + + // The genuinely qualifying run does record. + qualifyingRun(t, r, "run-3") + agg, present := collect(t, reader)["mecatl.product.time_to_first_value"] + if dps := firstValueHistogram(t, agg, present); len(dps) != 1 || dps[0].Count != 1 { + t.Fatalf("expected one sample after the qualifying run, got %+v", dps) + } +} diff --git a/internal/adapter/productmetrics/heartbeat.go b/internal/adapter/productmetrics/heartbeat.go new file mode 100644 index 0000000000..40809d82d8 --- /dev/null +++ b/internal/adapter/productmetrics/heartbeat.go @@ -0,0 +1,50 @@ +package productmetrics + +import ( + "context" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// DefaultHeartbeatInterval is the steady-state heartbeat cadence for +// long-running processes (mecated, mecatui, mecak8s). mecatequi (short-lived) +// passes 0 — a single immediate fire only, no ticker. +const DefaultHeartbeatInterval = 24 * time.Hour + +// Heartbeat records the periodic liveness + feature/provider/mode signal. +// Every attribute value comes from the closed Feature/ProviderFamily/ +// DeploymentMode enums — never a def/model/tool name. +func (r *Recorder) Heartbeat(snap FeatureSnapshot) { + ctx := context.Background() + r.heartbeat.Add(ctx, 1) + for feature, on := range snap.enabled() { + if on { + r.featureEnabled.Add(ctx, 1, metric.WithAttributes(attribute.String(attrFeature, string(feature)))) + } + } + r.providerConfig.Add(ctx, 1, metric.WithAttributes(attribute.String(attrProvider, string(snap.Provider)))) + r.deploymentMode.Add(ctx, 1, metric.WithAttributes(attribute.String(attrMode, string(snap.Mode)))) +} + +// RunHeartbeat fires one heartbeat immediately, then one every interval, +// until ctx is done. interval<=0 disables the ticker (a single fire only — +// mecatequi's shape). Meant to run in its own goroutine, owned by the +// caller (composition), which cancels ctx on shutdown. +func RunHeartbeat(ctx context.Context, r *Recorder, interval time.Duration, snap FeatureSnapshot) { + r.Heartbeat(snap) + if interval <= 0 { + return + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + r.Heartbeat(snap) + } + } +} diff --git a/internal/adapter/productmetrics/heartbeat_test.go b/internal/adapter/productmetrics/heartbeat_test.go new file mode 100644 index 0000000000..56880b2e5c --- /dev/null +++ b/internal/adapter/productmetrics/heartbeat_test.go @@ -0,0 +1,48 @@ +package productmetrics + +import ( + "context" + "testing" + "time" +) + +func TestRecorderHeartbeatRecordsClosedLabelsOnly(t *testing.T) { + r, reader := newTestRecorder(t) + r.Heartbeat(FeatureSnapshot{ + Memory: true, MCP: true, + Provider: ProviderAnthropic, Mode: ModeInteractive, + }) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.product.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want 1", got) + } + featureAgg := collected["mecatl.product.feature_enabled"] + if got := sumPoint(t, featureAgg, "feature", "memory"); got != 1 { + t.Errorf("feature_enabled{feature=memory} = %d, want 1", got) + } + if got := sumPoint(t, featureAgg, "feature", "mcp"); got != 1 { + t.Errorf("feature_enabled{feature=mcp} = %d, want 1", got) + } + // guardrails/scheduling were false in the snapshot: TestRecorderNeverAttachesUnboundedAttributesOrSensitiveContent + // (Task 6) is the exhaustive "no other data point" check; this test + // only asserts the enabled ones are present with the right value. + if got := sumPoint(t, collected["mecatl.product.provider_configured"], "family", "anthropic"); got != 1 { + t.Errorf("provider_configured{family=anthropic} = %d, want 1", got) + } + if got := sumPoint(t, collected["mecatl.product.deployment_mode"], "mode", "interactive"); got != 1 { + t.Errorf("deployment_mode{mode=interactive} = %d, want 1", got) + } +} + +func TestRunHeartbeatFiresImmediatelyThenStopsOnCtxDone(t *testing.T) { + r, reader := newTestRecorder(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancelled BEFORE RunHeartbeat: only the immediate fire happens. + + RunHeartbeat(ctx, r, time.Hour, FeatureSnapshot{Mode: ModeHeadless}) + + if got := sumValue(t, collect(t, reader)["mecatl.product.heartbeat"]); got != 1 { + t.Errorf("heartbeat = %d, want exactly 1 (immediate fire only)", got) + } +} diff --git a/internal/adapter/productmetrics/installid.go b/internal/adapter/productmetrics/installid.go new file mode 100644 index 0000000000..e98c8ed2ec --- /dev/null +++ b/internal/adapter/productmetrics/installid.go @@ -0,0 +1,73 @@ +package productmetrics + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +// installIDRelPath is the state-dir-relative path to the persisted anonymous +// install identifier — machine-written runtime state, not human config, so +// it lives under XDG_STATE_HOME (mirroring mecatui's +// $XDG_STATE_HOME/mecatl/mecatui.log precedent), not XDG_CONFIG_HOME. +const installIDRelPath = "mecatl/telemetry-id" + +// LoadOrCreateInstallID reads the persisted install UUID, creating one if +// absent or unparseable. The id is a bare random v4 UUID: it carries no +// machine or user information, and is trivially reset by deleting the file +// (the next opt-in mints a new one). firstRun is true whenever a new id was +// just minted — the caller uses it ONLY to decide whether to print the +// one-time disclosure notice. readFile/writeFile/mkdirAll are injected for +// testing; LoadOrCreateInstallIDDefault binds the real filesystem. +// +// The returned id IS threaded into an exported resource attribute (see +// provider.go) — this package makes no attempt to keep the id local-only; +// that was a prior, now-reverted design (see git history / the ADR's +// cost-analysis section for why it was reinstated). +func LoadOrCreateInstallID( + env xdgconfig.ResolveEnv, + readFile func(string) ([]byte, error), + writeFile func(string, []byte, os.FileMode) error, + mkdirAll func(string, os.FileMode) error, +) (id string, firstRun bool, err error) { + base := xdgconfig.UserStateDir(env) + if base == "" { + return "", false, fmt.Errorf("productmetrics: cannot resolve a state directory (no XDG_STATE_HOME and no home dir)") + } + path := filepath.Join(base, installIDRelPath) + + if readFile != nil { + if data, rerr := readFile(path); rerr == nil { + if existing := strings.TrimSpace(string(data)); existing != "" { + if _, perr := uuid.Parse(existing); perr == nil { + return existing, false, nil + } + // Corrupt file: fall through and regenerate. + } + } + } + + fresh := uuid.NewString() + if mkdirAll != nil { + if merr := mkdirAll(filepath.Dir(path), 0o700); merr != nil { + return "", false, fmt.Errorf("productmetrics: create state dir: %w", merr) + } + } + if writeFile != nil { + if werr := writeFile(path, []byte(fresh), 0o600); werr != nil { + return "", false, fmt.Errorf("productmetrics: write install id: %w", werr) + } + } + return fresh, true, nil +} + +// LoadOrCreateInstallIDDefault binds LoadOrCreateInstallID to the real +// process environment and filesystem. +func LoadOrCreateInstallIDDefault() (id string, firstRun bool, err error) { + return LoadOrCreateInstallID(xdgconfig.OSEnv, os.ReadFile, os.WriteFile, os.MkdirAll) +} diff --git a/internal/adapter/productmetrics/installid_test.go b/internal/adapter/productmetrics/installid_test.go new file mode 100644 index 0000000000..2adc0c94b6 --- /dev/null +++ b/internal/adapter/productmetrics/installid_test.go @@ -0,0 +1,90 @@ +package productmetrics + +import ( + "errors" + "os" + "testing" + + "github.com/google/uuid" + + "github.com/stacklok/mecatl/internal/adapter/xdgconfig" +) + +func TestLoadOrCreateInstallIDCreatesOnFirstRun(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + written := map[string][]byte{} + readFile := func(path string) ([]byte, error) { + data, ok := written[path] + if !ok { + return nil, os.ErrNotExist + } + return data, nil + } + writeFile := func(path string, data []byte, _ os.FileMode) error { + written[path] = data + return nil + } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on an empty store, want true") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + + // Second call reads back the SAME id and reports firstRun=false. + id2, firstRun2, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("second LoadOrCreateInstallID: %v", err) + } + if firstRun2 { + t.Error("firstRun = true on second call, want false") + } + if id2 != id { + t.Errorf("second call returned id %q, want %q (unchanged)", id2, id) + } +} + +func TestLoadOrCreateInstallIDRegeneratesOnCorruptFile(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "/home/tester", nil }, + } + readFile := func(string) ([]byte, error) { return []byte("not-a-uuid"), nil } + var gotWrite []byte + writeFile := func(_ string, data []byte, _ os.FileMode) error { gotWrite = data; return nil } + mkdirAll := func(string, os.FileMode) error { return nil } + + id, firstRun, err := LoadOrCreateInstallID(env, readFile, writeFile, mkdirAll) + if err != nil { + t.Fatalf("LoadOrCreateInstallID: %v", err) + } + if !firstRun { + t.Error("firstRun = false on a corrupt file, want true (treated as absent)") + } + if _, perr := uuid.Parse(id); perr != nil { + t.Errorf("id %q is not a valid UUID: %v", id, perr) + } + if string(gotWrite) != id { + t.Errorf("written content %q != returned id %q", gotWrite, id) + } +} + +func TestLoadOrCreateInstallIDFailsClosedWithNoStateDir(t *testing.T) { + env := xdgconfig.ResolveEnv{ + Getenv: func(string) string { return "" }, + UserHomeDir: func() (string, error) { return "", errors.New("no home") }, + } + _, _, err := LoadOrCreateInstallID(env, nil, nil, nil) + if err == nil { + t.Fatal("expected an error when no state dir can be resolved, got nil") + } +} diff --git a/internal/adapter/productmetrics/metrics.go b/internal/adapter/productmetrics/metrics.go new file mode 100644 index 0000000000..b4f8ca97eb --- /dev/null +++ b/internal/adapter/productmetrics/metrics.go @@ -0,0 +1,445 @@ +package productmetrics + +import ( + "context" + "fmt" + "strconv" + "sync" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// meterName is the instrumentation scope name for this package's meter. +const meterName = "github.com/stacklok/mecatl/internal/adapter/productmetrics" + +// Attribute keys. Every value ever attached under these keys is drawn from a +// bounded closed set (session.StopReason, the fixed token-kind strings, this +// package's own Feature/ProviderFamily/DeploymentMode enums, or "true"/"false") +// — never a session id, model id, or free text. The tool_calls instrument's +// two keys live with their closed-set projection in toolcall.go. +const ( + attrStop = "stop" + attrKind = "kind" + attrFeature = "feature" + attrProvider = "family" + attrMode = "mode" + attrHadToolCall = "had_tool_call" +) + +// Recorder is the product-metrics adapter: it implements port.EventSink +// (this file) and port.ToolCallRecorder + port.RunAwareToolCallRecorder +// (toolcall.go), deriving ONLY the bounded counts in the design's catalog. It +// never reads a session id, model id, result content, or any free-text field; +// the one thing it reads from a tool call is its NAME, and only to map it +// through the closed-set projection in toolcall.go (toolCategory), which can +// emit nothing but a built-in tool's own name, "mcp", or "other". +type Recorder struct { + heartbeat metric.Int64Counter + featureEnabled metric.Int64Counter + providerConfig metric.Int64Counter + deploymentMode metric.Int64Counter + sessionsStarted metric.Int64Counter + runsCompleted metric.Int64Counter + toolCalls metric.Int64Counter + tokens metric.Int64Counter + subagentUsed metric.Int64Counter + teamUsed metric.Int64Counter + runDuration metric.Float64Histogram + toolCallsPerRun metric.Int64Histogram + timeToFirstValue metric.Float64Histogram + + // perRun holds the bounded per-live-run facts this package derives across + // the Emit/ToolCallForRun boundary. See perRunTracker. + perRun *perRunTracker + + // firstValue holds the once-per-install time_to_first_value state. It is + // deliberately SEPARATE from perRun (which is per-live-run, keyed by run id + // and dropped at EvResult): this is install-scoped, single-slot state whose + // whole lifetime is the process. + firstValue firstValueTracker +} + +// firstValueTracker guards the once-ever time_to_first_value state. Armed by +// EnableFirstValueTracking (composition), read and flipped at most once by +// recordResult. +// +// CONCURRENCY: as with perRunTracker, the "is it done" test and the "mark it +// done" write are ONE critical section — concurrent EvResult observations on a +// fan-out deployment would otherwise both pass the test and record two samples +// for a metric whose entire contract is "at most one, ever". +type firstValueTracker struct { + mu sync.Mutex + // armed is false until EnableFirstValueTracking is called; an unarmed + // Recorder never records the metric at all (the default, byte-identical to + // the pre-feature posture for every existing caller of NewRecorder). + armed bool + // firstSeenAt is this install's first-seen moment; the recorded duration is + // measured from it. + firstSeenAt time.Time + // done is true once THIS process has attempted its one claim — either + // because the sample was already recorded (per the persisted marker) by + // an earlier process, or because this process itself just attempted the + // cross-process claim below. It does NOT by itself mean the sample was + // actually emitted — see claim()'s use of recordFn's won return. + done bool + // recordFn persists the marker as an ATOMIC cross-process claim and + // reports whether THIS call actually won it (won=true: this call + // created the marker first) versus lost it (won=false: another process + // already owns it — do not record, even though this process's own + // in-memory tracker had not yet observed that). Called at most once. + // Its error is best-effort: a failed write risks re-recording once on a + // later process (a fidelity wobble in a coarse onboarding signal, not a + // correctness bug worth failing anything over), so claim() still treats + // an ERRORING call as a win rather than silently dropping the sample. + recordFn func() (won bool, err error) +} + +// EnableFirstValueTracking arms mecatl.product.time_to_first_value recording. +// firstSeenAt is this install's first-seen timestamp; alreadyRecorded, when +// true, permanently disables recording for this Recorder's lifetime (this +// install already has its one sample). recordFn persists the local marker as +// an ATOMIC cross-process claim so a later process invocation also stays +// disabled; it is called at most once, and may be nil (in-memory-only +// tracking). Its won return MUST reflect whether THIS call actually created +// the marker — discarding it (always treating the call as a win) breaks the +// "at most once per install, ever" contract across two processes that both +// pass the alreadyRecorded=false startup check before either has recorded. +// +// It is a separate arming step rather than a NewRecorder parameter so that +// NewRecorder's signature — and every existing caller and test of it — stays +// unchanged; an unarmed Recorder simply never records this instrument. +func (r *Recorder) EnableFirstValueTracking(firstSeenAt time.Time, alreadyRecorded bool, recordFn func() (bool, error)) { + r.firstValue.mu.Lock() + defer r.firstValue.mu.Unlock() + r.firstValue.armed = true + r.firstValue.firstSeenAt = firstSeenAt + r.firstValue.done = alreadyRecorded + r.firstValue.recordFn = recordFn +} + +// claim reports whether THIS observation is the install's first-value +// moment, persisting the marker as one atomic cross-process step and +// honoring its outcome. It returns the firstSeenAt to measure from; a false +// claim means the metric must not be recorded (unarmed, already recorded, +// no usable firstSeenAt, OR — the cross-process case — this process's own +// atomic marker-claim call reports that another process already won it). +func (t *firstValueTracker) claim() (time.Time, bool) { + t.mu.Lock() + defer t.mu.Unlock() + if !t.armed || t.done || t.firstSeenAt.IsZero() { + return time.Time{}, false + } + t.done = true + if t.recordFn != nil { + won, err := t.recordFn() + if err != nil { + // Best-effort: a failed write risks re-recording once on a later + // process, which is a fidelity wobble in a coarse onboarding + // signal — not a correctness bug worth failing anything over. + // Fall through and treat this call as the winner, matching the + // pre-existing behavior on a write failure. + return t.firstSeenAt, true + } + if !won { + // Another process's marker-claim call already won this + // install's one-ever sample — honor that result instead of + // discarding it, or two processes that both started before + // either recorded would each independently emit a sample. + return time.Time{}, false + } + } + return t.firstSeenAt, true +} + +// perRunState is the bounded set of facts tracked for ONE live run, keyed by +// the loop-stamped session.Event.RunID (an opaque per-run correlation id, ADR +// 0249 — never a session id, tool name, or free-text field, so this package's +// no-PII invariant holds). Every field is a count or a boolean derived from a +// closed vocabulary; nothing here is ever attached as an attribute VALUE. +type perRunState struct { + // subagentSeen/teamSeen dedup subagentUsed/teamUsed to their documented + // "at least once per run" semantics: a run's first EvSubagentStart or + // EvTeamStart increments the counter, later ones in the SAME run (e.g. a + // fan-out of concurrent Subagent calls) do not. + subagentSeen bool + teamSeen bool + + // hadToolCall records whether the run made at least one SUCCESSFUL tool + // call — the product definition of "this run took an action", read at + // EvResult time as the runs_completed had_tool_call attribute. + hadToolCall bool + + // toolCallCount is the run's total tool calls (successful or not), + // published at EvResult as the tool_calls_per_run distribution. + toolCallCount int64 + + // startedAt is the wall-clock time this run's EvSessionInit was observed, + // used at EvResult time to compute run_duration. It stays the zero Time + // for a run whose EvSessionInit this Recorder never saw (e.g. a process + // restart mid-run) — recordResult must never record a duration from a + // zero startedAt. + startedAt time.Time +} + +// perRunTracker guards the live-run state map. Bounded to concurrently-live +// runs: a run's entry is created lazily on its first observed fact and dropped +// on its EvResult, so the map never grows across a process's lifetime. +// +// CONCURRENCY: every method below performs its map lookup AND its field +// mutation as ONE critical section under mu, and no *perRunState pointer ever +// escapes a locked region. Callers therefore cannot race on a state's fields: +// Emit (per event) and ToolCallForRun (per tool call) run concurrently on a +// fan-out run, and the only shape that is provably safe is "the lock covers +// both halves". +type perRunTracker struct { + mu sync.Mutex + states map[string]*perRunState +} + +func newPerRunTracker() *perRunTracker { + return &perRunTracker{states: make(map[string]*perRunState)} +} + +// stateLocked returns runID's live state, creating it if absent. The caller +// MUST hold t.mu, and must not retain the pointer past the critical section. +func (t *perRunTracker) stateLocked(runID string) *perRunState { + st, ok := t.states[runID] + if !ok { + st = &perRunState{} + t.states[runID] = st + } + return st +} + +// markFamilyUsed reports whether this is the first time, within the run +// identified by runID, that family has been observed — marking it seen as a +// side effect. An empty runID (no run context to dedup against) always counts, +// matching the pre-dedup behavior. +func (t *perRunTracker) markFamilyUsed(runID string, family delegationFamily) bool { + if runID == "" { + return true + } + t.mu.Lock() + defer t.mu.Unlock() + st := t.stateLocked(runID) + seen := &st.subagentSeen + if family == familyTeam { + seen = &st.teamSeen + } + if *seen { + return false + } + *seen = true + return true +} + +// markStarted stamps runID's start time as now. Called once, from Emit's +// EvSessionInit case; a run with no observed EvSessionInit never has this +// called and its startedAt stays the zero Time. +func (t *perRunTracker) markStarted(runID string) { + if runID == "" { + return + } + t.mu.Lock() + defer t.mu.Unlock() + st := t.stateLocked(runID) + st.startedAt = time.Now() +} + +// markToolCall tallies one tool call against runID. An empty runID (a caller +// on the base port.ToolCallRecorder path, with no run to correlate against) is +// tracked nowhere — its call is still counted on the tool_calls instrument, +// but it can contribute to no run's had_tool_call. +func (t *perRunTracker) markToolCall(runID string, errored bool) { + if runID == "" { + return + } + t.mu.Lock() + defer t.mu.Unlock() + st := t.stateLocked(runID) + st.toolCallCount++ + if !errored { + st.hadToolCall = true + } +} + +// finish drops runID's live state at EvResult and returns a COPY of what was +// there (the zero value if the run recorded nothing — e.g. a run with no tool +// calls and no delegation-family use), plus tracked reporting whether this +// run ever had an entry in the map at all. tracked is false both for an empty +// runID (no run correlation available — the base ToolCallRecorder.ToolCall +// path, or a caller like RetryFailedStep's RunRequest{} that mints no RunID) +// and for a nonempty runID this Recorder never observed an EvSessionInit or +// tool call for (e.g. a process restart mid-run). Distinguishing "genuinely +// tracked, made zero tool calls" from "never tracked at all" matters: without +// it, tool_calls_per_run would record a FABRICATED 0 for an untracked run +// that may have made many real tool calls this Recorder simply never +// correlated to it — recordResult uses tracked to skip that histogram +// sample entirely rather than report a number that isn't true. Returning a +// copy, not the pointer, keeps every field read outside the lock race-free. +func (t *perRunTracker) finish(runID string) (st perRunState, tracked bool) { + if runID == "" { + return perRunState{}, false + } + t.mu.Lock() + defer t.mu.Unlock() + existing, ok := t.states[runID] + if !ok { + return perRunState{}, false + } + delete(t.states, runID) + return *existing, true +} + +// Compile-time interface checks. +var ( + _ port.EventSink = (*Recorder)(nil) + _ port.ToolCallRecorder = (*Recorder)(nil) + _ port.RunAwareToolCallRecorder = (*Recorder)(nil) +) + +// NewRecorder constructs every instrument from the given MeterProvider. It +// returns an error if any instrument fails to construct — the OTel meter API +// is fallible. +func NewRecorder(mp metric.MeterProvider) (*Recorder, error) { + meter := mp.Meter(meterName) + r := &Recorder{perRun: newPerRunTracker()} + var err error + + if r.heartbeat, err = meter.Int64Counter("mecatl.product.heartbeat", + metric.WithDescription("Process liveness heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: heartbeat counter: %w", err) + } + if r.featureEnabled, err = meter.Int64Counter("mecatl.product.feature_enabled", + metric.WithDescription("Major feature enabled, by closed feature name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: feature_enabled counter: %w", err) + } + if r.providerConfig, err = meter.Int64Counter("mecatl.product.provider_configured", + metric.WithDescription("Configured LLM provider family, by closed family name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: provider_configured counter: %w", err) + } + if r.deploymentMode, err = meter.Int64Counter("mecatl.product.deployment_mode", + metric.WithDescription("Process deployment mode, by closed mode name, per heartbeat.")); err != nil { + return nil, fmt.Errorf("productmetrics: deployment_mode counter: %w", err) + } + if r.sessionsStarted, err = meter.Int64Counter("mecatl.product.sessions_started", + metric.WithDescription("Total sessions started.")); err != nil { + return nil, fmt.Errorf("productmetrics: sessions_started counter: %w", err) + } + if r.runsCompleted, err = meter.Int64Counter("mecatl.product.runs_completed", + metric.WithDescription("Total runs completed, by bounded stop reason and whether the run made at least one successful tool call.")); err != nil { + return nil, fmt.Errorf("productmetrics: runs_completed counter: %w", err) + } + if r.toolCalls, err = meter.Int64Counter("mecatl.product.tool_calls", + metric.WithDescription(`Total tool calls executed, by bounded category (a built-in tool's own name, the single value "mcp" for any MCP-server tool, or "other") and outcome.`)); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls counter: %w", err) + } + if r.tokens, err = meter.Int64Counter("mecatl.product.tokens", + metric.WithDescription("Total tokens accounted, by bounded kind."), + metric.WithUnit("{token}")); err != nil { + return nil, fmt.Errorf("productmetrics: tokens counter: %w", err) + } + if r.subagentUsed, err = meter.Int64Counter("mecatl.product.subagent_used", + metric.WithDescription("Runs that used the Subagent delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: subagent_used counter: %w", err) + } + if r.teamUsed, err = meter.Int64Counter("mecatl.product.team_used", + metric.WithDescription("Runs that used the Team delegation family at least once.")); err != nil { + return nil, fmt.Errorf("productmetrics: team_used counter: %w", err) + } + if r.runDuration, err = meter.Float64Histogram("mecatl.product.run_duration", + metric.WithDescription("Wall-clock duration of a run, from session init to result, in seconds."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: run_duration histogram: %w", err) + } + if r.toolCallsPerRun, err = meter.Int64Histogram("mecatl.product.tool_calls_per_run", + metric.WithDescription("Total tool calls made within a single run."), + metric.WithUnit("{tool_call}")); err != nil { + return nil, fmt.Errorf("productmetrics: tool_calls_per_run histogram: %w", err) + } + if r.timeToFirstValue, err = meter.Float64Histogram("mecatl.product.time_to_first_value", + metric.WithDescription("One-time-per-install duration, in seconds, from the start of the process that observed this install's first run that both made a successful tool call and ended cleanly (an approximation of onboarding time, not install age: a process started days after install and reaching that run in minutes reports minutes, not days)."), + metric.WithUnit("s")); err != nil { + return nil, fmt.Errorf("productmetrics: time_to_first_value histogram: %w", err) + } + return r, nil +} + +// Emit derives coarse, bounded counts from a single domain Event. It reads +// ONLY ev.Type, ev.RunID, ev.Result.Stop, and ev.Result.Usage — never a +// session id, model id/alias, tool name, or any free-text field +// (ev.Result.Text/Error are never touched). ev.RunID is an opaque per-run +// correlation id (ADR 0249), not a session id, and is used ONLY as the key of +// the per-run tracker (dedup'ing subagentUsed/teamUsed and resolving +// had_tool_call at EvResult); it never becomes an attribute value. +func (r *Recorder) Emit(ctx context.Context, ev session.Event) { + switch ev.Type { + case session.EvSessionInit: + r.sessionsStarted.Add(ctx, 1) + r.perRun.markStarted(ev.RunID) + case session.EvResult: + // finish both reads and clears the run's state, so the had_tool_call + // resolution and the bounded-map cleanup are one step. + st, tracked := r.perRun.finish(ev.RunID) + r.recordResult(ctx, ev.Result, st, tracked) + case session.EvSubagentStart: + if r.perRun.markFamilyUsed(ev.RunID, familySubagent) { + r.subagentUsed.Add(ctx, 1) + } + case session.EvTeamStart: + if r.perRun.markFamilyUsed(ev.RunID, familyTeam) { + r.teamUsed.Add(ctx, 1) + } + } +} + +// delegationFamily is the closed set of families dedup'd per run. +type delegationFamily int + +const ( + familySubagent delegationFamily = iota + familyTeam +) + +// recordResult counts the completed run against its bounded stop reason and +// the had_tool_call fact carried by the run's just-finished state. tracked +// (from perRunTracker.finish) gates tool_calls_per_run: an untracked run +// (empty RunID, or a RunID this Recorder never observed an EvSessionInit or +// tool call for) must not report a fabricated 0 — see finish's doc comment. +func (r *Recorder) recordResult(ctx context.Context, res *session.ResultPayload, st perRunState, tracked bool) { + stop := session.StopNone + if res != nil { + stop = res.Stop + } + r.runsCompleted.Add(ctx, 1, metric.WithAttributes( + attribute.String(attrStop, string(stop)), + attribute.String(attrHadToolCall, strconv.FormatBool(st.hadToolCall)))) + if tracked { + r.toolCallsPerRun.Record(ctx, st.toolCallCount) + } + if !st.startedAt.IsZero() { + r.runDuration.Record(ctx, time.Since(st.startedAt).Seconds()) + } + // The install's first-value moment: the first run that BOTH took an action + // (a successful tool call) and ended cleanly. Recorded at most once ever, + // across process restarts — see firstValueTracker. + if stop == session.StopEndTurn && st.hadToolCall { + if firstSeenAt, claimed := r.firstValue.claim(); claimed { + r.timeToFirstValue.Record(ctx, time.Since(firstSeenAt).Seconds()) + } + } + if res == nil { + return + } + u := res.Usage + r.tokens.Add(ctx, int64(u.InputTokens), metric.WithAttributes(attribute.String(attrKind, "input"))) + r.tokens.Add(ctx, int64(u.OutputTokens), metric.WithAttributes(attribute.String(attrKind, "output"))) + r.tokens.Add(ctx, int64(u.CacheReadTokens), metric.WithAttributes(attribute.String(attrKind, "cache_read"))) + r.tokens.Add(ctx, int64(u.CacheWriteTokens), metric.WithAttributes(attribute.String(attrKind, "cache_write"))) + r.tokens.Add(ctx, int64(u.ReasoningTokens), metric.WithAttributes(attribute.String(attrKind, "reasoning"))) +} diff --git a/internal/adapter/productmetrics/metrics_test.go b/internal/adapter/productmetrics/metrics_test.go new file mode 100644 index 0000000000..e877ad7d57 --- /dev/null +++ b/internal/adapter/productmetrics/metrics_test.go @@ -0,0 +1,361 @@ +package productmetrics + +import ( + "context" + "testing" + "time" + + "go.opentelemetry.io/otel/attribute" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +func newTestRecorder(t *testing.T) (*Recorder, *sdkmetric.ManualReader) { + t.Helper() + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + r, err := NewRecorder(mp) + if err != nil { + t.Fatalf("NewRecorder: %v", err) + } + return r, reader +} + +func collect(t *testing.T, reader *sdkmetric.ManualReader) map[string]metricdata.Aggregation { + t.Helper() + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + out := make(map[string]metricdata.Aggregation) + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + out[md.Name] = md.Data + } + } + return out +} + +func sumValue(t *testing.T, agg metricdata.Aggregation) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + var total int64 + for _, dp := range sum.DataPoints { + total += dp.Value + } + return total +} + +func sumPoint(t *testing.T, agg metricdata.Aggregation, key, value string) int64 { + t.Helper() + sum, ok := agg.(metricdata.Sum[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Sum[int64]", agg) + } + for _, dp := range sum.DataPoints { + if v, present := dp.Attributes.Value(attribute.Key(key)); present && v.AsString() == value { + return dp.Value + } + } + t.Fatalf("no data point with %s=%q", key, value) + return 0 +} + +func TestRecorderEmitSessionsStarted(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit}) + + agg, ok := collect(t, reader)["mecatl.product.sessions_started"] + if !ok { + t.Fatal("mecatl.product.sessions_started missing") + } + if got := sumValue(t, agg); got != 2 { + t.Errorf("sessions_started = %d, want 2", got) + } +} + +func TestRecorderEmitRunsCompletedByStopReason(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, "stop", "end_turn"); got != 1 { + t.Errorf("runs_completed{stop=end_turn} = %d, want 1", got) + } + if got := sumPoint(t, agg, "stop", "error"); got != 1 { + t.Errorf("runs_completed{stop=error} = %d, want 1", got) + } +} + +func TestRecorderEmitTokensByKind(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, + Result: &session.ResultPayload{ + Stop: session.StopEndTurn, + Usage: session.Usage{ + InputTokens: 100, + OutputTokens: 50, + CacheReadTokens: 20, + CacheWriteTokens: 5, + ReasoningTokens: 10, + }, + }, + }) + + agg := collect(t, reader)["mecatl.product.tokens"] + cases := map[string]int64{"input": 100, "output": 50, "cache_read": 20, "cache_write": 5, "reasoning": 10} + for kind, want := range cases { + if got := sumPoint(t, agg, "kind", kind); got != want { + t.Errorf("tokens{kind=%s} = %d, want %d", kind, got, want) + } + } +} + +func TestRecorderEmitSubagentAndTeamUsed(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart}) + r.Emit(context.Background(), session.Event{Type: session.EvTeamStart}) + + collected := collect(t, reader) + if got := sumValue(t, collected["mecatl.product.subagent_used"]); got != 1 { + t.Errorf("subagent_used = %d, want 1", got) + } + if got := sumValue(t, collected["mecatl.product.team_used"]); got != 1 { + t.Errorf("team_used = %d, want 1", got) + } +} + +// TestRecorderSubagentUsedCountsOncePerRun pins the documented "at least +// once per run" semantics: a run fanning out several concurrent Subagent +// calls (explicitly supported, e.g. the child concurrency gate) must count +// once, not once per EvSubagentStart. +func TestRecorderSubagentUsedCountsOncePerRun(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + + if got := sumValue(t, collect(t, reader)["mecatl.product.subagent_used"]); got != 1 { + t.Errorf("subagent_used = %d, want 1 (deduped within one run)", got) + } +} + +// TestRecorderSubagentUsedCountsEachDistinctRun proves the dedup is scoped +// to a run, not global: two separate runs each using Subagent count twice, +// and a run's dedup state is dropped on EvResult so a later run with the +// same RunID (unlikely in practice, but bounds correctness) still counts. +func TestRecorderSubagentUsedCountsEachDistinctRun(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-2"}) + + if got := sumValue(t, collect(t, reader)["mecatl.product.subagent_used"]); got != 2 { + t.Errorf("subagent_used = %d, want 2 (two distinct runs)", got) + } + + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: "run-1"}) + + if got := sumValue(t, collect(t, reader)["mecatl.product.subagent_used"]); got != 3 { + t.Errorf("subagent_used = %d, want 3 (run-1's dedup entry cleared on its EvResult)", got) + } +} + +func TestRecorderRunsCompletedHadToolCallTrueWhenASuccessfulToolCallOccurred(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1", got) + } +} + +func TestRecorderRunsCompletedHadToolCallFalseWithNoToolCall(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-2", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1", got) + } +} + +// TestRecorderRunsCompletedHadToolCallFalseWhenOnlyToolCallErrored pins the +// product definition: had_tool_call means the run took at least one +// SUCCESSFUL action, so a run whose only tool call errored is false. +func TestRecorderRunsCompletedHadToolCallFalseWhenOnlyToolCallErrored(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-3", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-3", + Result: &session.ResultPayload{Stop: session.StopError}, + }) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1 (the only tool call errored)", got) + } +} + +// TestRecorderRunsCompletedHadToolCallOnNilResult pins that a result-less +// EvResult still carries the attribute (the stop-reason arm already defaults +// to StopNone) rather than emitting an attribute-shape that differs from +// every other data point on the instrument. +func TestRecorderRunsCompletedHadToolCallOnNilResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-4", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-4"}) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1", got) + } + if got := sumPoint(t, agg, attrStop, string(session.StopNone)); got != 1 { + t.Errorf("runs_completed{stop=none} = %d, want 1", got) + } +} + +// TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns proves the per-run +// state is scoped to its run: two interleaved runs (a real possibility — +// Team/Parallel fan-out, or two concurrent client sessions on one process) +// must not leak their tool-call facts into each other. +func TestRecorderPerRunStateIsIsolatedAcrossConcurrentRuns(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-a", session.SessionID("s1"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-b", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-a", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg := collect(t, reader)["mecatl.product.runs_completed"] + if got := sumPoint(t, agg, attrHadToolCall, "false"); got != 1 { + t.Errorf("runs_completed{had_tool_call=false} = %d, want 1 (run-a's tool call must not leak into run-b)", got) + } + if got := sumPoint(t, agg, attrHadToolCall, "true"); got != 1 { + t.Errorf("runs_completed{had_tool_call=true} = %d, want 1 (run-a's own tool call)", got) + } +} + +func TestRecorderRunDurationRecordedFromSessionInitToResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + time.Sleep(5 * time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.run_duration"] + if !ok { + t.Fatal("mecatl.product.run_duration missing") + } + hist, ok := agg.(metricdata.Histogram[float64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[float64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 { + t.Fatalf("expected exactly 1 recorded duration, got %+v", hist.DataPoints) + } + if hist.DataPoints[0].Sum <= 0 { + t.Errorf("recorded duration sum = %v, want > 0", hist.DataPoints[0].Sum) + } +} + +func TestRecorderRunDurationNotRecordedWithoutMatchingSessionInit(t *testing.T) { + // A run whose EvSessionInit this Recorder never observed (e.g. process + // restarted mid-run — an edge case, not a common path) must not record a + // bogus/negative duration. + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "orphan-run", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + if agg, ok := collect(t, reader)["mecatl.product.run_duration"]; ok { + if hist, ok := agg.(metricdata.Histogram[float64]); ok && len(hist.DataPoints) > 0 && hist.DataPoints[0].Count > 0 { + t.Errorf("recorded a duration for a run with no observed EvSessionInit: %+v", hist.DataPoints) + } + } +} + +func TestRecorderToolCallsPerRunRecordedAtResult(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{}, 0, time.Millisecond) + // An ERRORED call still counts toward the run's tool-call total (only + // had_tool_call is success-gated). + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Grep"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"] + if !ok { + t.Fatal("mecatl.product.tool_calls_per_run missing") + } + hist, ok := agg.(metricdata.Histogram[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[int64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 || hist.DataPoints[0].Sum != 3 { + t.Fatalf("expected one data point with count 1 summing to 3, got %+v", hist.DataPoints) + } +} + +func TestRecorderToolCallsPerRunRecordsZeroForAToollessRun(t *testing.T) { + // A TRACKED run (its EvSessionInit was observed) that called no tool still + // contributes a 0 sample — otherwise the distribution silently + // over-reports by omitting its whole left tail. This is distinct from an + // UNTRACKED run (no EvSessionInit ever observed, e.g. a retry path that + // mints no RunID, or a process restart mid-run): see + // TestRecorderToolCallsPerRunSkippedForAnUntrackedRun below — recording a + // 0 there would be a FABRICATED sample, not an honest one. + r, reader := newTestRecorder(t) + r.Emit(context.Background(), session.Event{Type: session.EvSessionInit, RunID: "run-1"}) + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"] + if !ok { + t.Fatal("mecatl.product.tool_calls_per_run missing") + } + hist, ok := agg.(metricdata.Histogram[int64]) + if !ok { + t.Fatalf("aggregation is %T, want Histogram[int64]", agg) + } + if len(hist.DataPoints) != 1 || hist.DataPoints[0].Count != 1 || hist.DataPoints[0].Sum != 0 { + t.Fatalf("expected one data point with count 1 summing to 0, got %+v", hist.DataPoints) + } +} + +// TestRecorderToolCallsPerRunSkippedForAnUntrackedRun pins the fix for the +// finding in the final whole-branch review: a run whose EvSessionInit this +// Recorder never observed (e.g. RetryFailedStep's RunRequest{}, which mints +// no RunID, or a process restart mid-run) must not report a 0 on +// tool_calls_per_run — that 0 would be indistinguishable from a genuine +// zero-tool-call run, silently biasing the distribution downward for a run +// that may have made many real tool calls this Recorder simply never +// correlated. An untracked run must record NOTHING on this instrument. +func TestRecorderToolCallsPerRunSkippedForAnUntrackedRun(t *testing.T) { + r, reader := newTestRecorder(t) + // No EvSessionInit for "run-1" — this run is genuinely untracked, even + // though it carries a real, nonempty RunID. + r.Emit(context.Background(), session.Event{Type: session.EvResult, RunID: "run-1", Result: &session.ResultPayload{Stop: session.StopEndTurn}}) + + if agg, ok := collect(t, reader)["mecatl.product.tool_calls_per_run"]; ok { + if hist, ok := agg.(metricdata.Histogram[int64]); ok && len(hist.DataPoints) > 0 { + t.Fatalf("tool_calls_per_run recorded %+v for an untracked run, want no data point at all", hist.DataPoints) + } + } +} diff --git a/internal/adapter/productmetrics/provider.go b/internal/adapter/productmetrics/provider.go new file mode 100644 index 0000000000..36a1247786 --- /dev/null +++ b/internal/adapter/productmetrics/provider.go @@ -0,0 +1,216 @@ +package productmetrics + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.opentelemetry.io/otel/sdk/resource" + semconv "go.opentelemetry.io/otel/semconv/v1.41.0" +) + +// endpoint and headerKeyName are the ONE destination this pipeline can ever +// send to (stacklok/infra#5604): a dedicated, internet-facing OTLP/HTTP +// ingest at mecatl.metrics.stacklok.com, gated by a single shared key baked into +// the binary. Neither is operator-configurable — an operator's own +// --otlp-endpoint has zero effect on this path, and this path has zero +// effect on the operator's own OTLP/Prometheus pipeline (a completely +// separate MeterProvider, never installed as global). endpoint is a var +// (not a const) so tests can point it at an httptest server. +// +// endpoint is passed VERBATIM to otlpmetrichttp.WithEndpointURL, which sets +// the exporter's URL path to exactly endpoint's own path — an empty path is +// treated as the literal root "/", NOT as "use the exporter's documented +// /v1/metrics default" (see WithEndpointURL's own doc comment in +// go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp's +// internal/oconf package). endpoint therefore MUST spell out "/v1/metrics" +// itself. This is deliberately NOT toolhive-core's otlp.NewMetricReader, +// whose createMetricExporter instead SPLITS a supplied endpoint into +// host+basePath and, when a basePath is present, APPENDS its own +// "/v1/metrics" suffix onto it — so a base URL that already ends in +// "/v1/metrics" would there produce "/v1/metrics/v1/metrics" against the +// real collector. Building the exporter directly over otlpmetrichttp (as +// here) sidesteps that concatenation entirely: the path below is sent +// exactly as written, once. +var ( + endpoint = "https://mecatl.metrics.stacklok.com/v1/metrics" + headerKeyName = "x-mecatl-metrics-key" +) + +// bakedKey is the shared ingest key baked into the binary at build time via +// `-X github.com/stacklok/mecatl/internal/adapter/productmetrics.bakedKey=…` +// (see Taskfile.yml's BUILD_LDFLAGS). An empty key — every local/dev/CI-test +// build that does not set the ldflag — disables the pipeline entirely: +// NewProvider refuses to construct, so a non-release build can never +// accidentally phone home with an invalid or absent key. +var bakedKey = "" + +// Available reports whether this build has an ingest key baked in — i.e. +// whether NewProvider can ever construct a real pipeline. Composition MUST +// check this BEFORE minting/persisting any local install-id state (issue: +// disclosure/first-run ordering): a keyless build (every local/dev/CI-test +// build) that instead created the install-id file first, then failed here, +// would leave that file behind — so a LATER release build's genuine first +// export would read it back and report firstRun=false, silently skipping +// the disclosure notice ADR 0338 requires before that first export. +func Available() bool { return bakedKey != "" } + +// SetBakedKeyForTest overrides bakedKey for the duration of a test and +// returns a restore func the caller must defer. bakedKey is normally set +// exactly once, at build time, via the `-X …productmetrics.bakedKey=…` +// ldflag (see BUILD_LDFLAGS in Taskfile.yml); every other package's tests +// need a way to exercise the Available()==true path (e.g. cliconfig's +// disclosure-ordering tests) without actually shipping a real key, hence +// this seam — mirroring the ForTest helpers elsewhere in internal/adapter +// (e.g. scheduler.RunOnceForTest). +func SetBakedKeyForTest(key string) (restore func()) { + orig := bakedKey + bakedKey = key + return func() { bakedKey = orig } +} + +// SetEndpointForTest overrides endpoint for the duration of a test and +// returns a restore func the caller must defer — the cross-package sibling +// of SetBakedKeyForTest, so a caller in another package (e.g. cliconfig's +// disclosure-ordering tests) that also needs Available()==true can point +// the exporter at an httptest server instead of the real production +// ingest. +func SetEndpointForTest(url string) (restore func()) { + orig := endpoint + endpoint = url + return func() { endpoint = orig } +} + +// allowedResourceAttrs is the CLOSED set of resource attribute keys this +// pipeline may ever export, enforced by allowlistExporter below regardless +// of what the OTel SDK itself adds to a MeterProvider's resource. +var allowedResourceAttrs = map[string]bool{ + "service.name": true, + "service.version": true, + "mecatl.install.id": true, + "mecatl.binary": true, +} + +// allowlistExporter wraps a sdkmetric.Exporter and rewrites every exported +// ResourceMetrics.Resource down to allowedResourceAttrs immediately before +// serialization/transmission. This is the LAST line of defense against +// undeclared resource attributes reaching this vendor pipeline: the OTel SDK +// itself is not a clean pass-through here — metric.WithResource +// unconditionally merges the resource it is given with +// resource.Environment() (i.e. the ambient OTEL_RESOURCE_ATTRIBUTES env var) +// inside go.opentelemetry.io/otel/sdk/metric's own WithResource option, with +// no application-facing way to opt out. An operator's own +// OTEL_RESOURCE_ATTRIBUTES (meant for their OTLP/Prometheus pipeline) must +// never reach the Stacklok product-metrics destination, so filtering at +// export time — after the SDK's merge has already happened — is the only +// point that can enforce the closed catalog. +type allowlistExporter struct { + next sdkmetric.Exporter +} + +func (e *allowlistExporter) Temporality(k sdkmetric.InstrumentKind) metricdata.Temporality { + return e.next.Temporality(k) +} + +func (e *allowlistExporter) Aggregation(k sdkmetric.InstrumentKind) sdkmetric.Aggregation { + return e.next.Aggregation(k) +} + +func (e *allowlistExporter) Export(ctx context.Context, rm *metricdata.ResourceMetrics) error { + rm.Resource = allowlistResource(rm.Resource) + return e.next.Export(ctx, rm) +} + +func (e *allowlistExporter) ForceFlush(ctx context.Context) error { return e.next.ForceFlush(ctx) } + +func (e *allowlistExporter) Shutdown(ctx context.Context) error { return e.next.Shutdown(ctx) } + +// allowlistResource returns a fresh Resource carrying ONLY the attributes in +// allowedResourceAttrs from res — dropping anything else the SDK, an +// ambient env var, or a future dependency change might have added. +func allowlistResource(res *resource.Resource) *resource.Resource { + if res == nil { + return res + } + var kept []attribute.KeyValue + for _, kv := range res.Attributes() { + if allowedResourceAttrs[string(kv.Key)] { + kept = append(kept, kv) + } + } + return resource.NewSchemaless(kept...) +} + +// Provider wraps an OTLP metrics MeterProvider built directly over the OTel +// SDK's own otlpmetrichttp exporter (NOT toolhive-core's +// providers.NewCompositeProvider, which unconditionally adds +// resource.WithFromEnv() and resource.WithHost() to the exported resource on +// top of the SDK's own unconditional env merge — see allowlistExporter). +// Its MeterProvider is NEVER installed as the process-global provider +// (mirrors internal/adapter/telemetry's own discipline in otlp.go), so it +// cannot collide with an operator's own OTel setup. +type Provider struct { + meterProvider *sdkmetric.MeterProvider +} + +// NewProvider builds the product-metrics MeterProvider for one process. A +// network-unreachable endpoint is NOT an error here — the OTLP/HTTP +// exporter dials lazily on first export, matching the existing exporters in +// internal/adapter/telemetry/otlp.go. +func NewProvider(ctx context.Context, cfg Config) (*Provider, error) { + if bakedKey == "" { + return nil, fmt.Errorf("productmetrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") + } + // mecatl.install.id is a per-install random UUID, deliberately attached + // as a resource attribute (so it flattens onto every instrument this + // provider exports). This was removed once (see git history) over + // unbounded-cardinality concerns on the Prometheus-remote-write + // destination (stacklok/infra#5604), then reinstated after the actual + // cost was sized against real AMP pricing and accepted — see the ADR's + // cost-analysis section for the numbers. mecak8s provisions this value + // differently (a stable per-Helm-release ConfigMap, not this package's + // local install-id file — see internal/cliconfig's mecak8s wiring and + // deploy/helm/mecak8s/templates/install-id-configmap.yaml), since a + // pod-local file would mint a new id on every pod restart. + res, err := resource.New(ctx, + resource.WithAttributes( + semconv.ServiceName("mecatl"), + semconv.ServiceVersion(cfg.Version), + attribute.String("mecatl.install.id", cfg.InstallID), + attribute.String("mecatl.binary", string(cfg.Binary)), + ), + ) + if err != nil { + return nil, fmt.Errorf("productmetrics: build resource: %w", err) + } + + // otlpmetrichttp.WithEndpointURL parses endpoint itself and derives the + // host, the URL path (verbatim — see the endpoint var doc), and + // TLS-vs-insecure transport (https:// scheme => secure, everything else + // => insecure) — so the plain http:// endpoint this package's own tests + // point at an httptest.Server against needs no separate WithInsecure() + // call. + exp, err := otlpmetrichttp.New(ctx, + otlpmetrichttp.WithEndpointURL(endpoint), + otlpmetrichttp.WithHeaders(map[string]string{headerKeyName: bakedKey}), + ) + if err != nil { + return nil, fmt.Errorf("productmetrics: build exporter: %w", err) + } + + mp := sdkmetric.NewMeterProvider( + sdkmetric.WithResource(res), + sdkmetric.WithReader(sdkmetric.NewPeriodicReader(&allowlistExporter{next: exp})), + ) + return &Provider{meterProvider: mp}, nil +} + +// Meter returns the underlying metric.MeterProvider for instrument construction. +func (p *Provider) Meter() metric.MeterProvider { return p.meterProvider } + +// Shutdown flushes and stops the provider, bounded by the caller's ctx. +func (p *Provider) Shutdown(ctx context.Context) error { return p.meterProvider.Shutdown(ctx) } diff --git a/internal/adapter/productmetrics/provider_test.go b/internal/adapter/productmetrics/provider_test.go new file mode 100644 index 0000000000..662e59853a --- /dev/null +++ b/internal/adapter/productmetrics/provider_test.go @@ -0,0 +1,144 @@ +package productmetrics + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "os" + "testing" + + otlpmetrics "go.opentelemetry.io/proto/otlp/collector/metrics/v1" + "google.golang.org/protobuf/proto" +) + +func TestNewProviderFailsClosedWithNoBakedKey(t *testing.T) { + orig := bakedKey + bakedKey = "" + defer func() { bakedKey = orig }() + + _, err := NewProvider(context.Background(), Config{Binary: BinaryMecated, Version: "test"}) + if err == nil { + t.Fatal("expected an error when no ingest key is baked into the build, got nil") + } +} + +// fakeMetricsCollector is a minimal httptest OTLP/HTTP metrics ingest that +// records the exact request path and the raw ExportMetricsServiceRequest +// body of every export, so tests can assert on both — not merely that a +// request without a path arrived (which cannot detect a doubled signal path +// or a leaked resource attribute). +type fakeMetricsCollector struct { + srv *httptest.Server + gotPath string + gotBody *otlpmetrics.ExportMetricsServiceRequest + gotAuth string +} + +func newFakeMetricsCollector(t *testing.T) *fakeMetricsCollector { + t.Helper() + c := &fakeMetricsCollector{} + c.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c.gotPath = r.URL.Path + c.gotAuth = r.Header.Get(headerKeyName) + body, err := io.ReadAll(r.Body) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + req := new(otlpmetrics.ExportMetricsServiceRequest) + if perr := proto.Unmarshal(body, req); perr != nil { + http.Error(w, perr.Error(), http.StatusBadRequest) + return + } + c.gotBody = req + w.WriteHeader(http.StatusOK) + })) + t.Cleanup(c.srv.Close) + return c +} + +// TestNewProviderExportsToConfiguredEndpoint pins the endpoint against a +// PRODUCTION-SHAPED base URL — the real `endpoint` var, not a bare +// pathless httptest.Server URL — so a regression that re-introduces the +// doubled "/v1/metrics/v1/metrics" signal path (toolhive-core's +// otlp.NewMetricReader appends its own "/v1/metrics" to any non-empty base +// path) fails this test instead of passing silently. It also seeds an +// ambient OTEL_RESOURCE_ATTRIBUTES value and asserts the exported +// ResourceMetrics.Resource carries ONLY the four declared attributes, so a +// regression back to providers.NewCompositeProvider's unconditional +// resource.WithFromEnv()/resource.WithHost() also fails here. +func TestNewProviderExportsToConfiguredEndpoint(t *testing.T) { + origKey := bakedKey + bakedKey = "test-key" + defer func() { bakedKey = origKey }() + + collector := newFakeMetricsCollector(t) + + origEndpoint := endpoint + // Mirror the real production endpoint shape exactly: a base URL with NO + // signal path, matching `endpoint`'s documented contract. + endpoint = collector.srv.URL + "/v1/metrics" + defer func() { endpoint = origEndpoint }() + + origAttrs, hadAttrs := os.LookupEnv("OTEL_RESOURCE_ATTRIBUTES") + if err := os.Setenv("OTEL_RESOURCE_ATTRIBUTES", "tenant.id=customer-a"); err != nil { + t.Fatalf("Setenv: %v", err) + } + defer func() { + if hadAttrs { + os.Setenv("OTEL_RESOURCE_ATTRIBUTES", origAttrs) //nolint:errcheck // test cleanup + } else { + os.Unsetenv("OTEL_RESOURCE_ATTRIBUTES") //nolint:errcheck // test cleanup + } + }() + + p, err := NewProvider(context.Background(), Config{ + Binary: BinaryMecated, + Version: "test", + InstallID: "11111111-1111-1111-1111-111111111111", + }) + if err != nil { + t.Fatalf("NewProvider: %v", err) + } + defer p.Shutdown(context.Background()) + + meter := p.Meter().Meter("test") + counter, cerr := meter.Int64Counter("mecatl.product.test") + if cerr != nil { + t.Fatalf("Int64Counter: %v", cerr) + } + counter.Add(context.Background(), 1) + if err := p.Shutdown(context.Background()); err != nil { + t.Fatalf("Shutdown: %v", err) + } + + if collector.gotAuth != "test-key" { + t.Errorf("collector received %s=%q, want %q", headerKeyName, collector.gotAuth, "test-key") + } + if collector.gotPath != "/v1/metrics" { + t.Errorf("collector received request path %q, want %q (a doubled signal path regression)", collector.gotPath, "/v1/metrics") + } + if collector.gotBody == nil { + t.Fatal("collector received no decodable ExportMetricsServiceRequest") + } + + // Assert on the RESOURCE, not merely the instrument/data-point + // attributes: NewCompositeProvider's resource.WithFromEnv() + + // resource.WithHost() would surface both the seeded ambient + // OTEL_RESOURCE_ATTRIBUTES ("tenant.id") and the local hostname here. + allowed := map[string]bool{ + "service.name": true, + "service.version": true, + "mecatl.install.id": true, + "mecatl.binary": true, + } + for _, rm := range collector.gotBody.GetResourceMetrics() { + for _, attr := range rm.GetResource().GetAttributes() { + if !allowed[attr.GetKey()] { + t.Errorf("resource carries undeclared attribute %q=%q (privacy contract violation)", + attr.GetKey(), attr.GetValue().GetStringValue()) + } + } + } +} diff --git a/internal/adapter/productmetrics/toolcall.go b/internal/adapter/productmetrics/toolcall.go new file mode 100644 index 0000000000..89e8ed8f55 --- /dev/null +++ b/internal/adapter/productmetrics/toolcall.go @@ -0,0 +1,126 @@ +package productmetrics + +import ( + "context" + "strings" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/stacklok/mecatl/engine/session" +) + +// Attribute keys for the tool_calls instrument. Both draw from a closed set: +// attrCategory from builtinToolCategories ∪ {"mcp", "other"} (see +// toolCategory), attrOutcome from {"success", "error"}. +const ( + attrCategory = "category" + attrOutcome = "outcome" +) + +// The two bounded category values that are not a built-in tool's own name. +const ( + categoryMCP = "mcp" + categoryOther = "other" +) + +// mcpToolPrefix is the STRUCTURAL naming convention every MCP-server tool is +// registered under (internal/adapter/mcp/clientmcp.go: `"mcp__" + server + +// "__" + toolName`). Bucketing the whole prefix under one literal means an +// MCP server or remote tool name — operator-chosen free text, and the one +// genuinely unbounded slice of the catalog — can never reach an attribute +// value, structurally, whatever a future MCP integration is named. +const mcpToolPrefix = "mcp__" + +// builtinToolCategories is the CLOSED set of tool names this package may emit +// verbatim as a category: mecatl's own fixed catalog, where the name carries +// no operator or user content and the cardinality is fixed at compile time. +// +// It is deliberately an allowlist rather than a "not mcp__-prefixed ⇒ safe" +// inference: a catalog is not only built-ins plus MCP tools (an agent def, +// a learned skill, or a future extension seam can register a name derived +// from operator or model input), so the safe default for an unrecognised +// name is the single literal categoryOther — never the name itself. A new +// built-in showing up as "other" in the metric is the visible, harmless +// prompt to add a line here. +var builtinToolCategories = map[string]bool{ + // Filesystem + shell (engine/adapter/fstools, engine/agent). + "Read": true, "ListDir": true, "Edit": true, "Write": true, + "Copy": true, "Move": true, "Remove": true, "Grep": true, "Glob": true, + "Bash": true, "BashStatus": true, + // Outbound reads (engine/adapter/webfetch, engine/adapter/search). + "WebFetch": true, "WebSearch": true, + // MCP meta-tools — mecatl's OWN fixed names, distinct from the + // mcp__-prefixed server tools they operate over. + "ListMcpResources": true, "ReadMcpResource": true, + "CallMcpWithQuery": true, "FetchMcpResource": true, + // Delegation (engine/agent). + "Subagent": true, "SubagentStatus": true, "InspectSubagent": true, + "Parallel": true, "Team": true, "InspectMember": true, "SubmitResult": true, + // Team coordination (engine/agent/teamtools.go). + "SendMessage": true, "AddTask": true, "ClaimTask": true, + "CompleteTask": true, "ListTasks": true, "RecordFinding": true, + // Session debugging (internal/adapter/sessiondebug). + "InspectSession": true, + // Skills. + "Skill": true, "SkillDraft": true, + // Project memory (internal/adapter/memory). + "Remember": true, "Recall": true, "SearchMemory": true, + "InspectMemory": true, "ForgetMemory": true, "UndoMemory": true, + // User model (internal/adapter/memory). + "RememberUser": true, "RecallUser": true, "SearchUserModel": true, + "InspectUserMemory": true, "ForgetUserMemory": true, "UndoUserMemory": true, + // Composition-owned + miscellaneous built-ins. + "PresentPlan": true, "Schedule": true, "ScheduleQuery": true, + "DiscoverModels": true, "ToolSearch": true, +} + +// The two bounded outcome values. +const ( + outcomeSuccess = "success" + outcomeError = "error" +) + +// toolCategory projects a tool name onto the bounded category attribute: the +// tool's own name for a recognised built-in, categoryMCP for anything +// MCP-server-provided, categoryOther for everything else. +func toolCategory(name string) string { + if strings.HasPrefix(name, mcpToolPrefix) { + return categoryMCP + } + if builtinToolCategories[name] { + return name + } + return categoryOther +} + +// ToolCall satisfies port.ToolCallRecorder. It is the fallback path for a +// caller that drives the base port without the run correlation — it records +// with no run id, so the call is counted on tool_calls but can contribute to +// no run's had_tool_call. The engine itself always prefers ToolCallForRun +// (it type-asserts port.RunAwareToolCallRecorder), so in production this arm +// serves only a non-loop caller. +func (r *Recorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + r.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder. It records the +// bounded category/outcome attributes and tallies the run's per-run state +// (had_tool_call, and the tool-call count published as tool_calls_per_run). +// +// It reads exactly two things off its arguments: call.Name, only through the +// closed-set toolCategory projection, and result.IsError, a boolean. The +// session id, the queued/took durations, and every free-text field +// (result.Content, call arguments) are ignored — the ignored parameters are +// accepted only because the port's signature requires them. +func (r *Recorder) ToolCallForRun(runID string, _ session.SessionID, call session.ToolCall, result session.ToolResult, _, _ time.Duration) { + outcome := outcomeSuccess + if result.IsError { + outcome = outcomeError + } + r.toolCalls.Add(context.Background(), 1, metric.WithAttributes( + attribute.String(attrCategory, toolCategory(call.Name)), + attribute.String(attrOutcome, outcome))) + r.perRun.markToolCall(runID, result.IsError) +} diff --git a/internal/adapter/productmetrics/toolcall_test.go b/internal/adapter/productmetrics/toolcall_test.go new file mode 100644 index 0000000000..181a2cb616 --- /dev/null +++ b/internal/adapter/productmetrics/toolcall_test.go @@ -0,0 +1,197 @@ +package productmetrics + +import ( + "context" + "sync" + "testing" + "time" + + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/session" +) + +func TestRecorderToolCallCountsWithoutIdentity(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCall( + session.SessionID("sensitive-session-id"), + session.ToolCall{Name: "read_secret_file"}, + session.ToolResult{Content: "super secret content", IsError: true}, + 10*time.Millisecond, 20*time.Millisecond, + ) + r.ToolCall(session.SessionID("other"), session.ToolCall{Name: "another_tool"}, session.ToolResult{}, 0, 0) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumValue(t, agg); got != 2 { + t.Errorf("tool_calls = %d, want 2", got) + } +} + +func TestRecorderToolCallForRunCategorizesBuiltinsByName(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrCategory, "Bash"); got != 1 { + t.Errorf("tool_calls{category=Bash} = %d, want 1", got) + } + if got := sumPoint(t, agg, attrCategory, "Read"); got != 1 { + t.Errorf("tool_calls{category=Read} = %d, want 1", got) + } +} + +// TestRecorderToolCallForRunBucketsUnrecognisedNamesUnderOther pins the +// closed-set discipline: a name that is NOT in mecatl's own fixed catalog is +// not assumed safe to emit verbatim (an agent def, a learned skill, or a +// future extension seam could derive one from operator or model input), so it +// lands on the single literal "other". +func TestRecorderToolCallForRunBucketsUnrecognisedNamesUnderOther(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "some-operator-named-tool"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "another_unknown"}, session.ToolResult{}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrCategory, categoryOther); got != 2 { + t.Errorf("tool_calls{category=other} = %d, want 2 (unrecognised names must not be emitted verbatim)", got) + } +} + +func TestRecorderToolCallForRunBucketsMCPToolsUnderOneCategory(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__github__list_issues"}, session.ToolResult{}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "mcp__slack__post_message"}, session.ToolResult{}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrCategory, categoryMCP); got != 2 { + t.Errorf("tool_calls{category=mcp} = %d, want 2 (both MCP-server tools bucketed together)", got) + } + + // The real server/tool names must never appear as an attribute value. + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + for _, sm := range rm.ScopeMetrics { + for _, md := range sm.Metrics { + sum, ok := md.Data.(metricdata.Sum[int64]) + if !ok { + continue + } + for _, dp := range sum.DataPoints { + iter := dp.Attributes.Iter() + for iter.Next() { + kv := iter.Attribute() + for _, leak := range []string{"github", "list_issues", "slack", "post_message"} { + if kv.Value.AsString() == leak { + t.Fatalf("MCP server/tool name leaked as an attribute value: %s=%s", kv.Key, kv.Value.AsString()) + } + } + } + } + } + } +} + +func TestRecorderToolCallForRunRecordsOutcome(t *testing.T) { + r, reader := newTestRecorder(t) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: false}, 0, time.Millisecond) + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Bash"}, session.ToolResult{IsError: true}, 0, time.Millisecond) + + agg := collect(t, reader)["mecatl.product.tool_calls"] + if got := sumPoint(t, agg, attrOutcome, outcomeSuccess); got != 1 { + t.Errorf("tool_calls{outcome=success} = %d, want 1", got) + } + if got := sumPoint(t, agg, attrOutcome, outcomeError); got != 1 { + t.Errorf("tool_calls{outcome=error} = %d, want 1", got) + } +} + +// TestRecorderPerRunTrackerIsRaceFreeUnderConcurrentUse drives the two +// concurrent entry points into the shared per-run map — ToolCallForRun (per +// tool call, from the dispatcher's read-parallel batches) and Emit (per +// event) — against overlapping run ids, so `-race` exercises the one +// lock-discipline invariant this task introduces. +// +// The live runs are finished SEQUENTIALLY afterwards, deliberately: a +// concurrent EvResult racing its own run's tool calls has no defined +// ordering, so the bounded-map assertion below would be a coin flip rather +// than an invariant. Runs finished concurrently use disjoint ids. +func TestRecorderPerRunTrackerIsRaceFreeUnderConcurrentUse(t *testing.T) { + r, _ := newTestRecorder(t) + live := []string{"run-a", "run-b", "run-c"} + finishing := []string{"run-d", "run-e", "run-f"} + + var wg sync.WaitGroup + for _, runID := range live { + for i := 0; i < 8; i++ { + wg.Add(2) + go func(runID string) { + defer wg.Done() + r.ToolCallForRun(runID, session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + }(runID) + go func(runID string) { + defer wg.Done() + r.Emit(context.Background(), session.Event{Type: session.EvSubagentStart, RunID: runID}) + }(runID) + } + } + for _, runID := range finishing { + wg.Add(1) + go func(runID string) { + defer wg.Done() + r.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: runID, + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + }(runID) + } + wg.Wait() + + for _, runID := range live { + st, tracked := r.perRun.finish(runID) + if !tracked { + t.Errorf("%s: finish reported untracked, want tracked (tool calls were made)", runID) + } + if st.toolCallCount != 8 { + t.Errorf("%s toolCallCount = %d, want 8", runID, st.toolCallCount) + } + if !st.hadToolCall || !st.subagentSeen { + t.Errorf("%s state = %+v, want hadToolCall and subagentSeen both true", runID, st) + } + } + + // Every run has now been finished, so no state may be left behind. + r.perRun.mu.Lock() + left := len(r.perRun.states) + r.perRun.mu.Unlock() + if left != 0 { + t.Errorf("perRun.states holds %d entries after every run's EvResult, want 0 (the map must stay bounded to live runs)", left) + } +} + +// TestRecorderToolCallForRunTalliesPerRunCount pins the per-run tool-call +// count the later tool-calls-per-run task reads. It is tallied but not yet +// published as an instrument, so it is asserted on the state directly. +func TestRecorderToolCallForRunTalliesPerRunCount(t *testing.T) { + r, _ := newTestRecorder(t) + for i := 0; i < 3; i++ { + r.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{IsError: i == 0}, 0, 0) + } + // A call with no run correlation must not land on any run. + r.ToolCall(session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + + st, tracked := r.perRun.finish("run-1") + if !tracked { + t.Error("finish(\"run-1\") reported untracked, want tracked") + } + if st.toolCallCount != 3 { + t.Errorf("toolCallCount = %d, want 3", st.toolCallCount) + } + if !st.hadToolCall { + t.Error("hadToolCall = false, want true (two of the three calls succeeded)") + } + if got, gotTracked := r.perRun.finish(""); got != (perRunState{}) || gotTracked { + t.Errorf("finish(\"\") = (%+v, tracked=%v), want the zero state and tracked=false", got, gotTracked) + } +} diff --git a/internal/adapter/server/mutation_capability.go b/internal/adapter/server/mutation_capability.go index 3df558bbcf..922b9cd6b4 100644 --- a/internal/adapter/server/mutation_capability.go +++ b/internal/adapter/server/mutation_capability.go @@ -117,3 +117,27 @@ func (r capabilityToolCallRecorder) ToolCall(id session.SessionID, call session. r.next.ToolCall(id, call, result, queued, took) } } + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder, forwarding to +// r.next's richer form under the SAME capability.allows(id) gate ToolCall +// uses. Without this method, wrapping a RunAwareToolCallRecorder-capable +// recorder (e.g. productmetrics.Recorder) behind this capability guard would +// ERASE the optional capability: the engine's dispatch.go type-assertion is +// on Deps.ToolCallRecorder itself, so a wrapper implementing only the base +// interface fails that assertion regardless of what it wraps, silently +// disabling had_tool_call/tool_calls_per_run/time_to_first_value. +func (r capabilityToolCallRecorder) ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + if !r.capability.allows(id) { + return + } + if aware, ok := r.next.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(runID, id, call, result, queued, took) + return + } + r.next.ToolCall(id, call, result, queued, took) +} + +// Compile-time interface check: capabilityToolCallRecorder must keep +// forwarding port.RunAwareToolCallRecorder, or the same capability silently +// goes inert whenever a productmetrics.Recorder is guarded by it. +var _ port.RunAwareToolCallRecorder = capabilityToolCallRecorder{} diff --git a/internal/adapter/server/mutation_capability_toolcall_test.go b/internal/adapter/server/mutation_capability_toolcall_test.go new file mode 100644 index 0000000000..61397725c7 --- /dev/null +++ b/internal/adapter/server/mutation_capability_toolcall_test.go @@ -0,0 +1,82 @@ +package server + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" +) + +// runAwareRecorderFunc implements BOTH port.ToolCallRecorder and +// port.RunAwareToolCallRecorder, recording which method was actually called. +type runAwareRecorderFunc struct { + plainCalls *int + runAwareCalls *[]string +} + +func (f runAwareRecorderFunc) ToolCall(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + *f.plainCalls++ +} + +func (f runAwareRecorderFunc) ToolCallForRun(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + *f.runAwareCalls = append(*f.runAwareCalls, runID) +} + +// TestCapabilityToolCallRecorderForwardsRunAwareCapability pins the bug found +// in the final whole-branch review of the had_tool_call/tool_calls_per_run/ +// time_to_first_value work: GuardToolCallRecorder's returned value is what +// lands in Deps.ToolCallRecorder, so if it implemented only the base +// port.ToolCallRecorder, the engine's dispatch.go type-assertion for +// port.RunAwareToolCallRecorder would ALWAYS fail — silently making +// had_tool_call/tool_calls_per_run/time_to_first_value inert in every real +// binary, despite the wrapped recorder correctly implementing the richer +// interface. This test drives the guarded value exactly the way dispatch.go +// does: type-assert, then call ToolCallForRun if it succeeds. +func TestCapabilityToolCallRecorderForwardsRunAwareCapability(t *testing.T) { + plainCalls := 0 + var runAwareCalls []string + next := runAwareRecorderFunc{plainCalls: &plainCalls, runAwareCalls: &runAwareCalls} + + mc := NewSessionMutationCapability(false) // disabled gate: allows(id) always true + guarded := mc.GuardToolCallRecorder(next) + + aware, ok := guarded.(port.RunAwareToolCallRecorder) + if !ok { + t.Fatal("GuardToolCallRecorder's result does not implement port.RunAwareToolCallRecorder — had_tool_call/tool_calls_per_run/time_to_first_value would be inert in production") + } + aware.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if plainCalls != 0 { + t.Errorf("plainCalls = %d, want 0 (the run-aware form must be preferred)", plainCalls) + } + if len(runAwareCalls) != 1 || runAwareCalls[0] != "run-1" { + t.Errorf("runAwareCalls = %v, want [run-1]", runAwareCalls) + } +} + +// TestCapabilityToolCallRecorderGatesRunAwareOnAllows confirms +// ToolCallForRun respects the SAME capability.allows(id) gate ToolCall uses: +// once a session's mutation capability is invalidated, neither method should +// reach the wrapped recorder. +func TestCapabilityToolCallRecorderGatesRunAwareOnAllows(t *testing.T) { + plainCalls := 0 + var runAwareCalls []string + next := runAwareRecorderFunc{plainCalls: &plainCalls, runAwareCalls: &runAwareCalls} + + mc := NewSessionMutationCapability(true) + id := session.SessionID("s") + mc.Grant(id) + mc.Invalidate(id) + + guarded := mc.GuardToolCallRecorder(next) + aware, ok := guarded.(port.RunAwareToolCallRecorder) + if !ok { + t.Fatal("GuardToolCallRecorder's result does not implement port.RunAwareToolCallRecorder") + } + aware.ToolCallForRun("run-1", id, session.ToolCall{}, session.ToolResult{}, 0, 0) + + if plainCalls != 0 || len(runAwareCalls) != 0 { + t.Errorf("plainCalls=%d runAwareCalls=%v, want both empty (invalidated capability must block ToolCallForRun same as ToolCall)", plainCalls, runAwareCalls) + } +} diff --git a/internal/cliconfig/productmetrics.go b/internal/cliconfig/productmetrics.go new file mode 100644 index 0000000000..d5895ddcb9 --- /dev/null +++ b/internal/cliconfig/productmetrics.go @@ -0,0 +1,224 @@ +// Package cliconfig assembles the BuildProductMetrics composition helper — +// the full opt-out product-metrics pipeline (install id → provider → +// recorder → heartbeat goroutine) that a cmd main threads into its +// EventSink/ToolCallRecorder fan-out. Kept separate from +// productmetrics_config.go (the precedence/fan-out helpers Task 9 added): +// this file is the thing that actually constructs the pipeline, not the +// pure-function policy that decides whether to. +package cliconfig + +import ( + "context" + "fmt" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// ProductMetricsDisclosureNotice is printed ONCE — the first run after +// product metrics were enabled and this install's telemetry id did not yet +// exist — to stderr, non-blockingly, before any pipeline is built. Opt-out +// telemetry without a visible disclosure is the pattern that burns +// community trust; this is the whole of that disclosure. +const ProductMetricsDisclosureNotice = `mecatl reports anonymous product-adoption metrics (version, OS/arch, which +major features you have enabled, an anonymous per-install identifier, and +coarse session/run/tool-call counts by bounded category — never a prompt, +file path, raw tool name, or model id) to help Stacklok understand community +adoption. This is on by default. To opt out: pass +--product-metrics=false, set MECATL_PRODUCT_METRICS=false, set DO_NOT_TRACK=1, +or set telemetry.productMetrics.enabled: false in your settings.yaml. Details: +see docs/adr/0338-product-metrics.md. +` + +// ProductMetricsHandles bundles the handles a cmd main threads into its +// EventSink/ToolCallRecorder fan-out (via TeeToolCallRecorder / +// internal/adapter/telemetry.NewSink alongside the operator sink) and its +// shutdown defer. Every field is zero-valued when telemetry is disabled. +type ProductMetricsHandles struct { + Sink port.EventSink + ToolCallRecorder port.ToolCallRecorder + // Shutdown flushes + stops the provider. Always non-nil (a no-op when + // disabled), so a caller can defer it unconditionally. + Shutdown func(context.Context) error + // FirstRun is true the first time this install's telemetry id was just + // minted — the caller prints ProductMetricsDisclosureNotice when true. + FirstRun bool +} + +// BuildProductMetrics constructs the full opt-out product-metrics pipeline +// when enabled is true; when false it returns zero handles (the +// byte-identical disabled posture) and no error. heartbeatInterval is +// productmetrics.DefaultHeartbeatInterval for long-running processes, or 0 +// for a single-fire-only short-lived process (mecatequi). heartbeatCtx is +// cancelled by the caller on shutdown to stop the periodic ticker goroutine +// this starts. +// +// When dryRun is true (and enabled is also true), this builds a +// productmetrics.DryRunRecorder over diag instead of the real OTLP +// pipeline — the --product-metrics-dry-run audit path: no install-id +// read/write, no real provider, no real heartbeat ticker. It fires exactly +// ONE representative Heartbeat call SYNCHRONOUSLY, before returning (a dry +// run only needs to show one sample, not simulate the full cadence, and a +// short-lived process like mecatequi can exit before an unawaited goroutine +// ever runs) and returns handles wrapping the DryRunRecorder as both Sink +// and ToolCallRecorder. +// +// installIDOverride, when non-empty, is used verbatim as the install id and +// the local-file mechanism (LoadOrCreateInstallIDDefault) is skipped entirely. +// It exists for mecak8s, which runs storage-free with no PVC (ADR 0048): its +// Helm chart provisions ONE stable id per release in a ConfigMap and threads +// it in via MECATL_PRODUCT_METRICS_INSTALL_ID, because a local file would mint +// a fresh, never-reused id on every pod restart. An override never reports +// FirstRun (nothing was minted here, and the chart — not this process — owns +// the id's lifecycle). Every other binary passes "" and keeps the local-file +// behaviour unchanged. +// +// notify, when firstRun is true, is called EXACTLY ONCE, SYNCHRONOUSLY, +// BEFORE this function starts the heartbeat goroutine (whose first +// Heartbeat call fires immediately — see RunHeartbeat) and before it +// returns. ADR 0338 makes visible advance disclosure load-bearing for +// opt-out collection: printing the notice only after the caller later +// notices ProductMetricsHandles.FirstRun — e.g. after its own startup work, +// or worse, only at shutdown/flush time — leaves a window where the +// pipeline can record and export data before a human ever sees the notice. +// Calling notify here, before ANY export-capable state exists, closes that +// window regardless of what the caller does afterward (including an error +// path that flushes and discards the handles without ever consulting +// FirstRun). notify is nil-safe: a nil notify simply skips the call (kept +// for the disabled/dry-run/override paths and existing test callers that +// don't exercise disclosure). FirstRun is still returned on the handles for +// callers/tests that want to observe it, but it must never be the sole +// trigger for actually showing the notice. +func BuildProductMetrics( + ctx, heartbeatCtx context.Context, + enabled, dryRun bool, + binary productmetrics.Binary, + version string, + heartbeatInterval time.Duration, + snap productmetrics.FeatureSnapshot, + installIDOverride string, + diag port.Diagnostics, + notify func(string), +) (ProductMetricsHandles, error) { + noop := func(context.Context) error { return nil } + if !enabled { + return ProductMetricsHandles{Shutdown: noop}, nil + } + if dryRun { + rec := productmetrics.NewDryRunRecorder(diag) + rec.Heartbeat(snap) + return ProductMetricsHandles{Sink: rec, ToolCallRecorder: rec, Shutdown: noop}, nil + } + + // LoadOrCreateInstallIDDefault persists (or reads back) this process's + // local install-id file and reports firstRun for the disclosure notice + // below. Reinstated as a real, exported resource attribute (see + // provider.go's doc comment) after its cardinality cost was sized and + // accepted. An externally provisioned id (see installIDOverride) bypasses + // it entirely: there is no file to read, write, or report a first run + // from, so Available() gates ONLY this local-file branch — a keyless + // build (every local/dev/CI-test build) must not mint and persist an id + // file that a later release build's LoadOrCreateInstallIDDefault would + // then read back as "already exists", silently reporting firstRun=false + // for that build's genuine first export (see Available's doc comment). + // The override path still proceeds and fails later, at provider + // construction, exactly as before. + installID, firstRun := installIDOverride, false + if installID == "" { + if !productmetrics.Available() { + return ProductMetricsHandles{Shutdown: noop}, + fmt.Errorf("product metrics: no ingest key baked into this build (see BUILD_LDFLAGS in Taskfile.yml)") + } + var err error + installID, firstRun, err = productmetrics.LoadOrCreateInstallIDDefault() + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: install id: %w", err) + } + } + + provider, err := productmetrics.NewProvider(ctx, productmetrics.Config{ + Binary: binary, + Version: version, + InstallID: installID, + }) + if err != nil { + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: provider: %w", err) + } + + recorder, err := productmetrics.NewRecorder(provider.Meter()) + if err != nil { + _ = provider.Shutdown(ctx) + return ProductMetricsHandles{Shutdown: noop}, fmt.Errorf("product metrics: recorder: %w", err) + } + + // Disclosure BEFORE the pipeline goes live: notify runs synchronously, + // here, before the heartbeat goroutine below is even started — so no + // export-capable state exists yet when a human is expected to have seen + // the notice. + if firstRun && notify != nil { + notify(ProductMetricsDisclosureNotice) + } + + go productmetrics.RunHeartbeat(heartbeatCtx, recorder, heartbeatInterval, snap) + + armFirstValueTracking(ctx, recorder, installIDOverride, diag) + + return ProductMetricsHandles{ + Sink: recorder, + ToolCallRecorder: recorder, + Shutdown: provider.Shutdown, + FirstRun: firstRun, + }, nil +} + +// armFirstValueTracking enables mecatl.product.time_to_first_value on +// recorder — EXCEPT when installIDOverride is non-empty (mecak8s), where it +// deliberately does nothing. +// +// mecak8s runs storage-free with no PVC (ADR 0048) — the SAME reason its +// install-id comes from a Helm ConfigMap rather than a local file (see +// provider.go's doc comment). The once-ever contract of time_to_first_value +// depends on the SAME kind of durable local marker +// (LoadOrCreateFirstValueMarkerDefault, under $XDG_STATE_HOME) the install-id +// mechanism does, and mecak8s's Helm chart provisions no equivalent for it. +// Arming anyway would make every pod restart/replica rearm with +// alreadyRecorded=false, so a continuously-rolling deployment would emit a +// steady stream of "time to first value" samples that are really +// "time from this pod's start to its first qualifying run" — indistinguishable +// in the backend, under one stable mecatl.install.id, from a stream of +// brand-new installs onboarding continuously. Silently shipping that under a +// "once per install, ever" label would be worse than not shipping the metric +// at all for this one binary; a future durable marker (the ConfigMap, or +// Redis, since mecak8s already depends on it — ADR 0048) can lift this +// restriction later. +// +// firstSeenAt is time.Now(): this process's start, not the install-id file's +// mtime. The approximation is deliberate and sound for the signal's purpose (a +// coarse "how long did onboarding take", not a billing-grade timer) — the +// marker read below means the metric can only ever fire on an install that has +// not yet had a qualifying run, and for a genuinely new install this process IS +// the first one, so "now" is that install's first-seen moment to within the +// process's own startup. It also keeps the install-id file's path private to +// the productmetrics package. +// +// A marker-read failure degrades to "track it anyway" rather than disabling +// anything: time_to_first_value is a nice-to-have signal, not load-bearing +// enough to fail the whole product-metrics pipeline over. The worst case is one +// duplicate sample from a later process. +func armFirstValueTracking(ctx context.Context, recorder *productmetrics.Recorder, installIDOverride string, diag port.Diagnostics) { + if installIDOverride != "" { + return + } + already, err := productmetrics.FirstValueRecordedDefault() + if err != nil && diag != nil { + diag.Log(ctx, port.LevelDebug, + "product metrics: could not read the first-value marker; time_to_first_value may be re-recorded once", + "error", err) + already = false + } + recorder.EnableFirstValueTracking(time.Now(), already, func() (bool, error) { + alreadyExisted, werr := productmetrics.LoadOrCreateFirstValueMarkerDefault() + return !alreadyExisted, werr + }) +} diff --git a/internal/cliconfig/productmetrics_config.go b/internal/cliconfig/productmetrics_config.go new file mode 100644 index 0000000000..5e5e33c738 --- /dev/null +++ b/internal/cliconfig/productmetrics_config.go @@ -0,0 +1,166 @@ +// Package cliconfig provides product-metrics opt-out precedence and a +// ToolCallRecorder fan-out helper. +package cliconfig + +import ( + "os" + "strconv" + "strings" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +// doNotTrackOptOut reports whether a DO_NOT_TRACK env value means "opt out", +// per the donottrack.sh convention: unset/empty and the conventional +// "off" spellings ("0", "false", case-insensitive) are NOT an opt-out; any +// other value is. +func doNotTrackOptOut(v string) bool { + switch strings.ToLower(v) { + case "", "0", "false": + return false + default: + return true + } +} + +// mecatlProductMetricsOverride parses the MECATL_PRODUCT_METRICS env var: a +// mecatl-specific boolean override, distinct from the generic DO_NOT_TRACK +// convention. Named to match --product-metrics/telemetry.productMetrics.enabled +// exactly (one vocabulary word across all three surfaces) rather than a +// "track"/"telemetry"-flavored name — this codebase's OWN "telemetry" already +// means the unrelated, opt-in operator OTLP/Prometheus pipeline +// (internal/adapter/telemetry), so a same-flavored name here would misleadingly +// suggest it also touches that pipeline; it does not and never should. +// set is false when the var is empty/unset/unparseable, letting the caller +// fall through to the next precedence tier. +func mecatlProductMetricsOverride(v string) (value, set bool) { + if v == "" { + return false, false + } + b, err := strconv.ParseBool(v) + if err != nil { + return false, false + } + return b, true +} + +// ProductMetricsPrecedence carries the opt-out inputs +// ResolveProductMetricsEnabled folds, highest precedence first: an explicit +// CLI flag, then the mecatl-specific MECATL_PRODUCT_METRICS env var, then the +// DO_NOT_TRACK env var convention (donottrack.sh), then the operator +// settings.yaml value, then default-enabled. +type ProductMetricsPrecedence struct { + // FlagSet/FlagValue report whether --product-metrics was explicitly + // passed on the command line and its value. + FlagSet bool + FlagValue bool + // Getenv abstracts os.Getenv for MECATL_PRODUCT_METRICS / DO_NOT_TRACK / + // testing. Defaults to os.Getenv when nil. + Getenv func(string) string + // SettingsEnabled is permconfig.Resolver.OperatorProductMetricsEnabled() + // — nil when the operator set no telemetry.productMetrics.enabled value. + SettingsEnabled *bool +} + +// ResolveProductMetricsEnabled applies the opt-out precedence documented on +// ProductMetricsPrecedence. Default (nothing set anywhere) is true — product +// metrics are OPT-OUT, not opt-in. +func ResolveProductMetricsEnabled(p ProductMetricsPrecedence) bool { + if p.FlagSet { + return p.FlagValue + } + getenv := p.Getenv + if getenv == nil { + getenv = os.Getenv + } + if v, set := mecatlProductMetricsOverride(getenv("MECATL_PRODUCT_METRICS")); set { + return v + } + if doNotTrackOptOut(getenv("DO_NOT_TRACK")) { + return false + } + if p.SettingsEnabled != nil { + return *p.SettingsEnabled + } + return true +} + +// ResolveProviderFamily derives the closed-set productmetrics.ProviderFamily +// from the same two CLI-level signals every one of the four mecatl binaries +// resolves at flag-parse time (useOpenAI is a dedicated --openai bool that +// exists on mecated/mecatequi/mecak8s; defaultProvider is --default-provider +// on all four). It NEVER returns the type's zero value — the enum has no +// zero-value member, only ProviderAnthropic/OpenAI/OpenRouter/Other — so a +// heartbeat can never emit the invalid provider_configured{family=""} that a +// hand-rolled, only-partly-populated FeatureSnapshot produced before. Kept +// here (not duplicated per binary) as the SINGLE shared oracle every +// productMetricsSnapshot in cmd/mecated, cmd/mecatui, cmd/mecatequi, and +// cmd/mecak8s calls, mirroring mecated's original Task 11 switch verbatim. +func ResolveProviderFamily(useOpenAI bool, defaultProvider string) productmetrics.ProviderFamily { + lower := strings.ToLower(defaultProvider) + switch { + case useOpenAI: + return productmetrics.ProviderOpenAI + case strings.Contains(lower, "openrouter"): + return productmetrics.ProviderOpenRouter + case strings.Contains(lower, "openai"): + return productmetrics.ProviderOpenAI + case defaultProvider == "" || strings.Contains(lower, "anthropic"): + return productmetrics.ProviderAnthropic + default: + return productmetrics.ProviderOther + } +} + +// TeeToolCallRecorder combines multiple ToolCallRecorders into one — the +// ToolCallRecorder twin of internal/adapter/telemetry.NewSink's EventSink +// fan-out (no such helper existed before product metrics, because until now +// only one ToolCallRecorder ever observed a given engine). nil entries are +// skipped, so a caller can pass an always-present operator recorder +// alongside an optional product-metrics one without a conditional slice +// build. +func TeeToolCallRecorder(recorders ...port.ToolCallRecorder) port.ToolCallRecorder { + var non []port.ToolCallRecorder + for _, r := range recorders { + if r != nil { + non = append(non, r) + } + } + return multiToolCallRecorder(non) +} + +type multiToolCallRecorder []port.ToolCallRecorder + +func (m multiToolCallRecorder) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + m.ToolCallForRun("", id, call, result, queued, took) +} + +// ToolCallForRun satisfies port.RunAwareToolCallRecorder: it forwards runID to +// any element that implements the richer interface, and falls back to that +// element's plain ToolCall otherwise. Without this method, wrapping a +// RunAwareToolCallRecorder-capable recorder (e.g. productmetrics.Recorder) in +// a multiToolCallRecorder would ERASE the optional capability — the engine's +// dispatch.go type-assertion is on Deps.ToolCallRecorder itself, and a +// composed value that only implements the base interface fails that +// assertion regardless of what it wraps. This is the general hazard with +// decorating an optional-capability interface: every decorator in the chain +// must forward it, or the capability silently stops reaching the type that +// actually needs it. +func (m multiToolCallRecorder) ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + for _, r := range m { + if aware, ok := r.(port.RunAwareToolCallRecorder); ok { + aware.ToolCallForRun(runID, id, call, result, queued, took) + continue + } + r.ToolCall(id, call, result, queued, took) + } +} + +// Compile-time interface check: multiToolCallRecorder must keep forwarding +// port.RunAwareToolCallRecorder, or had_tool_call/tool_calls_per_run/ +// time_to_first_value silently go inert in every binary that tees a +// productmetrics.Recorder through this helper. +var _ port.RunAwareToolCallRecorder = multiToolCallRecorder(nil) diff --git a/internal/cliconfig/productmetrics_config_test.go b/internal/cliconfig/productmetrics_config_test.go new file mode 100644 index 0000000000..0788a28818 --- /dev/null +++ b/internal/cliconfig/productmetrics_config_test.go @@ -0,0 +1,182 @@ +package cliconfig + +import ( + "testing" + "time" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +func boolPtr(b bool) *bool { return &b } + +// envMap builds a Getenv func from a name->value map; an unlisted name +// returns "" (unset), matching os.Getenv's own behavior. +func envMap(m map[string]string) func(string) string { + return func(name string) string { return m[name] } +} + +// TestResolveProviderFamily pins the closed-set mapping the product-metrics +// heartbeat relies on across all four mecatl binaries: it must never return +// the type's zero value (empty string) — the enum has no such member — so +// every case below lands on a real ProviderFamily. +func TestResolveProviderFamily(t *testing.T) { + cases := []struct { + name string + useOpenAI bool + defaultProvider string + want productmetrics.ProviderFamily + }{ + {"empty defaults to anthropic", false, "", productmetrics.ProviderAnthropic}, + {"useOpenAI wins regardless of defaultProvider", true, "openrouter/some-model", productmetrics.ProviderOpenAI}, + {"defaultProvider names openrouter", false, "openrouter/anthropic/claude", productmetrics.ProviderOpenRouter}, + {"defaultProvider names openai", false, "openai/gpt-5", productmetrics.ProviderOpenAI}, + {"defaultProvider names anthropic", false, "anthropic/claude-opus", productmetrics.ProviderAnthropic}, + {"defaultProvider is case-insensitive", false, "OpenRouter/x", productmetrics.ProviderOpenRouter}, + {"unrecognized defaultProvider falls to other", false, "some-custom-gateway", productmetrics.ProviderOther}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := ResolveProviderFamily(tc.useOpenAI, tc.defaultProvider); got != tc.want { + t.Errorf("ResolveProviderFamily(%v, %q) = %q, want %q", tc.useOpenAI, tc.defaultProvider, got, tc.want) + } + }) + } +} + +func TestResolveProductMetricsEnabledPrecedence(t *testing.T) { + cases := []struct { + name string + p ProductMetricsPrecedence + want bool + }{ + {"flag true wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: true, Getenv: envMap(map[string]string{"DO_NOT_TRACK": "1", "MECATL_PRODUCT_METRICS": "0"}), SettingsEnabled: boolPtr(false)}, true}, + {"flag false wins over everything", ProductMetricsPrecedence{FlagSet: true, FlagValue: false, Getenv: envMap(nil), SettingsEnabled: boolPtr(true)}, false}, + {"MECATL_PRODUCT_METRICS wins over DO_NOT_TRACK", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"MECATL_PRODUCT_METRICS": "true", "DO_NOT_TRACK": "1"}), SettingsEnabled: boolPtr(false)}, true}, + {"MECATL_PRODUCT_METRICS=false wins over an unset DO_NOT_TRACK and settings", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"MECATL_PRODUCT_METRICS": "false"}), SettingsEnabled: boolPtr(true)}, false}, + {"MECATL_PRODUCT_METRICS unparseable falls through to DO_NOT_TRACK", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"MECATL_PRODUCT_METRICS": "yes", "DO_NOT_TRACK": "1"}), SettingsEnabled: boolPtr(true)}, false}, + {"DO_NOT_TRACK disables when no flag or MECATL_PRODUCT_METRICS", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "1"}), SettingsEnabled: boolPtr(true)}, false}, + {"settings.yaml honoured when no flag/env", ProductMetricsPrecedence{Getenv: envMap(nil), SettingsEnabled: boolPtr(false)}, false}, + {"default enabled when nothing set", ProductMetricsPrecedence{Getenv: envMap(nil), SettingsEnabled: nil}, true}, + {"DO_NOT_TRACK=0 is not an opt-out", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "0"}), SettingsEnabled: boolPtr(true)}, true}, + {"DO_NOT_TRACK=false is not an opt-out", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "false"}), SettingsEnabled: boolPtr(true)}, true}, + {"DO_NOT_TRACK=true disables", ProductMetricsPrecedence{Getenv: envMap(map[string]string{"DO_NOT_TRACK": "true"}), SettingsEnabled: boolPtr(true)}, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := ResolveProductMetricsEnabled(tc.p); got != tc.want { + t.Errorf("ResolveProductMetricsEnabled(%+v) = %v, want %v", tc.p, got, tc.want) + } + }) + } +} + +// TestMecatlProductMetricsOverride pins the parsing rules for the +// MECATL_PRODUCT_METRICS env var directly: any strconv.ParseBool-recognized +// spelling is honored: unset/empty/unparseable means "not set" (falls through +// to the next precedence tier), never a false positive. +func TestMecatlProductMetricsOverride(t *testing.T) { + cases := []struct { + in string + wantValue bool + wantSet bool + }{ + {"", false, false}, + {"1", true, true}, + {"true", true, true}, + {"TRUE", true, true}, + {"0", false, true}, + {"false", false, true}, + {"FALSE", false, true}, + {"yes", false, false}, + {"no", false, false}, + } + for _, tc := range cases { + t.Run(tc.in, func(t *testing.T) { + value, set := mecatlProductMetricsOverride(tc.in) + if value != tc.wantValue || set != tc.wantSet { + t.Errorf("mecatlProductMetricsOverride(%q) = (%v, %v), want (%v, %v)", tc.in, value, set, tc.wantValue, tc.wantSet) + } + }) + } +} + +func TestTeeToolCallRecorderCallsEveryNonNilRecorder(t *testing.T) { + var calls []string + rec := func(name string) port.ToolCallRecorder { + return recorderFunc(func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + calls = append(calls, name) + }) + } + tee := TeeToolCallRecorder(rec("a"), nil, rec("b")) + tee.ToolCall(session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if len(calls) != 2 || calls[0] != "a" || calls[1] != "b" { + t.Errorf("calls = %v, want [a b] (nil skipped, order preserved)", calls) + } +} + +// recorderFunc adapts a plain func to port.ToolCallRecorder for this test. +type recorderFunc func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) + +func (f recorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f(id, call, result, queued, took) +} + +// runAwareRecorderFunc additionally implements port.RunAwareToolCallRecorder, +// so tests can distinguish which method a caller actually invoked. +type runAwareRecorderFunc struct { + plain func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) + runAware func(string, session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) +} + +func (f runAwareRecorderFunc) ToolCall(id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f.plain(id, call, result, queued, took) +} + +func (f runAwareRecorderFunc) ToolCallForRun(runID string, id session.SessionID, call session.ToolCall, result session.ToolResult, queued, took time.Duration) { + f.runAware(runID, id, call, result, queued, took) +} + +// TestTeeToolCallRecorderForwardsRunAwareCapability pins the bug found in the +// final whole-branch review of the had_tool_call/tool_calls_per_run/ +// time_to_first_value work: TeeToolCallRecorder's returned value is what +// lands in Deps.ToolCallRecorder, so if it implemented only the base +// port.ToolCallRecorder, the engine's dispatch.go type-assertion for +// port.RunAwareToolCallRecorder would ALWAYS fail — silently making +// had_tool_call/tool_calls_per_run/time_to_first_value inert in every real +// binary, despite productmetrics.Recorder itself correctly implementing the +// richer interface. This test drives the composed value exactly the way +// dispatch.go does: type-assert, then call ToolCallForRun if it succeeds. +func TestTeeToolCallRecorderForwardsRunAwareCapability(t *testing.T) { + var runAwareCalls []string + var plainCalls []string + + runAware := runAwareRecorderFunc{ + plain: func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + plainCalls = append(plainCalls, "runaware-recorder-plain") + }, + runAware: func(runID string, _ session.SessionID, _ session.ToolCall, _ session.ToolResult, _, _ time.Duration) { + runAwareCalls = append(runAwareCalls, runID) + }, + } + baseOnly := recorderFunc(func(session.SessionID, session.ToolCall, session.ToolResult, time.Duration, time.Duration) { + plainCalls = append(plainCalls, "base-only-recorder") + }) + + tee := TeeToolCallRecorder(baseOnly, runAware) + + aware, ok := tee.(port.RunAwareToolCallRecorder) + if !ok { + t.Fatal("TeeToolCallRecorder's result does not implement port.RunAwareToolCallRecorder — had_tool_call/tool_calls_per_run/time_to_first_value would be inert in production") + } + aware.ToolCallForRun("run-1", session.SessionID(""), session.ToolCall{}, session.ToolResult{}, 0, 0) + + if len(runAwareCalls) != 1 || runAwareCalls[0] != "run-1" { + t.Errorf("runAwareCalls = %v, want the run-aware element to receive ToolCallForRun with runID %q", runAwareCalls, "run-1") + } + if len(plainCalls) != 1 || plainCalls[0] != "base-only-recorder" { + t.Errorf("plainCalls = %v, want the base-only element to fall back to ToolCall exactly once", plainCalls) + } +} diff --git a/internal/cliconfig/productmetrics_test.go b/internal/cliconfig/productmetrics_test.go new file mode 100644 index 0000000000..5b304602e2 --- /dev/null +++ b/internal/cliconfig/productmetrics_test.go @@ -0,0 +1,273 @@ +package cliconfig + +import ( + "context" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + + "github.com/stacklok/mecatl/engine/port" + "github.com/stacklok/mecatl/engine/session" + "github.com/stacklok/mecatl/internal/adapter/productmetrics" +) + +func TestBuildProductMetricsDisabledReturnsZeroHandles(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), false, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=false): %v", err) + } + if h.Sink != nil || h.ToolCallRecorder != nil { + t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder: %+v", h) + } + if h.Shutdown == nil { + t.Fatal("Shutdown must be non-nil even when disabled (a no-op)") + } + if err := h.Shutdown(context.Background()); err != nil { + t.Errorf("no-op Shutdown returned an error: %v", err) + } +} + +func TestBuildProductMetricsEnabledFailsClosedWithNoBakedKey(t *testing.T) { + // bakedKey is empty in every non-release build/test — enabling must + // surface the error rather than silently disabling, so a caller notices + // its release build is missing the ldflag. + _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) + if err == nil { + t.Fatal("expected an error when enabled=true with no baked ingest key, got nil") + } +} + +// TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider proves the +// dry-run branch takes the DryRunRecorder short-circuit BEFORE the +// install-id/provider construction that requires a baked ingest key — so +// dryRun=true must succeed with no error even though the enabled-for-real +// case (above) fails closed with no baked key. +func TestBuildProductMetricsDryRunNeverTouchesInstallIDOrRealProvider(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), true, true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=true, dryRun=true): %v", err) + } + if h.Sink == nil || h.ToolCallRecorder == nil { + t.Errorf("dry-run handles must carry a non-nil Sink/ToolCallRecorder: %+v", h) + } + if h.Shutdown == nil { + t.Fatal("Shutdown must be non-nil in dry-run mode (a no-op)") + } + if err := h.Shutdown(context.Background()); err != nil { + t.Errorf("dry-run no-op Shutdown returned an error: %v", err) + } + if h.FirstRun { + t.Error("dry-run must never mint/read an install id, so FirstRun must stay false") + } +} + +// TestBuildProductMetricsDisabledDryRunStillNoop proves dryRun is inert when +// enabled is false — the disabled posture must stay byte-identical +// regardless of the dry-run flag's value. +func TestBuildProductMetricsDisabledDryRunStillNoop(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), false, true, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) + if err != nil { + t.Fatalf("BuildProductMetrics(enabled=false, dryRun=true): %v", err) + } + if h.Sink != nil || h.ToolCallRecorder != nil { + t.Errorf("disabled handles carry a non-nil Sink/ToolCallRecorder even with dryRun=true: %+v", h) + } +} + +// TestBuildProductMetricsInstallIDOverrideSkipsTheLocalFile is the mecak8s +// contract (storage-free, no PVC, ADR 0048): a chart-provisioned install id +// must bypass LoadOrCreateInstallIDDefault ENTIRELY, not merely take +// precedence over whatever it returns. +// +// The oracle is which failure surfaces, and at which layer. bakedKey is +// empty in every non-release build/test, so BuildProductMetrics.Available() +// gates the no-override path BEFORE it ever calls +// LoadOrCreateInstallIDDefault — its error therefore has NO "provider:" +// wrapping. An override skips that Available() gate (and the file read) +// entirely and instead fails later, inside productmetrics.NewProvider +// itself, whose error IS "provider:"-wrapped. If the override were applied +// after the Available()/install-id gate, both calls would report the exact +// same unwrapped error. +func TestBuildProductMetricsInstallIDOverrideSkipsTheLocalFile(t *testing.T) { + _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, + "", port.NopDiagnostics{}, nil) + if err == nil || strings.Contains(err.Error(), "provider:") { + t.Fatalf("no override: err = %v, want the unwrapped Available() gate error, not a provider-construction failure", err) + } + + _, err = BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, + "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}, nil) + if err == nil || !strings.Contains(err.Error(), "provider:") { + t.Fatalf("override: err = %v, want a provider-construction failure (the Available()/install-id gate skipped)", err) + } +} + +// TestBuildProductMetricsInstallIDOverrideNeverReportsFirstRun pins the +// disclosure-notice half: an override mints nothing locally, so this process +// has no first run to announce — the chart owns the id's lifecycle. Asserted +// on the disabled and dry-run paths, the only two that return handles without +// a baked ingest key. +func TestBuildProductMetricsInstallIDOverrideNeverReportsFirstRun(t *testing.T) { + for _, tc := range []struct { + name string + enabled, dryRun bool + }{ + {name: "disabled", enabled: false}, + {name: "dry run", enabled: true, dryRun: true}, + } { + t.Run(tc.name, func(t *testing.T) { + h, err := BuildProductMetrics(context.Background(), context.Background(), tc.enabled, tc.dryRun, + productmetrics.BinaryMecak8s, "test-version", 0, productmetrics.FeatureSnapshot{}, + "11111111-2222-3333-4444-555555555555", port.NopDiagnostics{}, nil) + if err != nil { + t.Fatalf("BuildProductMetrics with an install-id override: %v", err) + } + if h.FirstRun { + t.Error("an externally provisioned install id must never report FirstRun") + } + }) + } +} + +// TestArmFirstValueTrackingSkipsWhenInstallIDIsOverridden pins the fix for the +// finding in the final whole-branch review: mecak8s (which passes a non-empty +// installIDOverride) has no durable local marker for time_to_first_value's +// once-ever contract, the same storage-free problem (ADR 0048) install-id +// solves via a Helm ConfigMap. Arming anyway would make every pod +// restart/replica rearm with alreadyRecorded=false, turning "once per +// install, ever" into "once per pod start" — a silent correctness bug in the +// metric's own contract. armFirstValueTracking must therefore no-op entirely +// when installIDOverride is non-empty: EnableFirstValueTracking must never be +// called, so a subsequent qualifying EvResult records nothing. +func TestArmFirstValueTrackingSkipsWhenInstallIDIsOverridden(t *testing.T) { + reader := sdkmetric.NewManualReader() + mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader)) + recorder, err := productmetrics.NewRecorder(mp) + if err != nil { + t.Fatalf("NewRecorder: %v", err) + } + + armFirstValueTracking(context.Background(), recorder, "11111111-2222-3333-[REDACTED]", port.NopDiagnostics{}) + + // Drive a qualifying run: a successful tool call followed by a clean-ended + // result. If tracking were (incorrectly) armed, this would record a + // time_to_first_value sample. + recorder.ToolCallForRun("run-1", session.SessionID("s"), session.ToolCall{Name: "Read"}, session.ToolResult{}, 0, 0) + recorder.Emit(context.Background(), session.Event{ + Type: session.EvResult, RunID: "run-1", + Result: &session.ResultPayload{Stop: session.StopEndTurn}, + }) + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("Collect: %v", err) + } + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + if m.Name == "mecatl.product.time_to_first_value" { + if hist, ok := m.Data.(metricdata.Histogram[float64]); ok && len(hist.DataPoints) > 0 { + t.Fatalf("time_to_first_value recorded %d data point(s) despite an install-id override — armFirstValueTracking must skip arming entirely for mecak8s", len(hist.DataPoints)) + } + } + } + } +} + +// TestBuildProductMetricsNeverMintsInstallIDWithoutABakedKey pins the +// keyless-build-to-release transition finding: without a baked ingest key +// (every local/dev/CI-test build), BuildProductMetrics must return an error +// WITHOUT ever creating the local install-id file. If it created that file +// anyway (the prior ordering), a LATER release build with a real baked key +// would read the file back as "already exists" and silently report +// FirstRun=false on its genuine first export — skipping the ADR +// 0338-mandated disclosure notice for that install's actual first +// transmission. +func TestBuildProductMetricsNeverMintsInstallIDWithoutABakedKey(t *testing.T) { + stateDir := t.TempDir() + t.Setenv("XDG_STATE_HOME", stateDir) + + _, err := BuildProductMetrics(context.Background(), context.Background(), true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, nil) + if err == nil { + t.Fatal("expected an error with no baked ingest key, got nil") + } + + if _, statErr := os.Stat(filepath.Join(stateDir, "mecatl", "telemetry-id")); statErr == nil { + t.Fatal("BuildProductMetrics must not create the install-id file when no ingest key is baked into this build") + } else if !os.IsNotExist(statErr) { + t.Fatalf("unexpected error checking for the install-id file: %v", statErr) + } +} + +// TestBuildProductMetricsNotifiesSynchronouslyBeforeReturning pins the +// disclosure-ordering fix: notify must be called EXACTLY ONCE, synchronously +// — before BuildProductMetrics returns, before the heartbeat goroutine's +// first (immediate) export-eligible recording — precisely when firstRun is +// true, and never on a second call against the same install-id state (where +// firstRun is false). This uses SetBakedKeyForTest/SetEndpointForTest +// (pointed at a local httptest server, never the real production endpoint) +// so the Available()-gated real-provider path actually runs. +func TestBuildProductMetricsNotifiesSynchronouslyBeforeReturning(t *testing.T) { + restoreKey := productmetrics.SetBakedKeyForTest("test-key") + defer restoreKey() + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + restoreEndpoint := productmetrics.SetEndpointForTest(srv.URL + "/v1/metrics") + defer restoreEndpoint() + + t.Setenv("XDG_STATE_HOME", t.TempDir()) + + var notified []string + notify := func(s string) { notified = append(notified, s) } + + heartbeatCtx, cancel := context.WithCancel(context.Background()) + defer cancel() + + h, err := BuildProductMetrics(context.Background(), heartbeatCtx, true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, notify) + if err != nil { + t.Fatalf("BuildProductMetrics: %v", err) + } + defer h.Shutdown(context.Background()) + + if !h.FirstRun { + t.Fatal("want FirstRun=true on a fresh install-id state directory") + } + if len(notified) != 1 || notified[0] != ProductMetricsDisclosureNotice { + t.Fatalf("notify calls = %v, want exactly one call carrying ProductMetricsDisclosureNotice", notified) + } + + // A second call against the SAME state directory reads back the + // already-minted id: firstRun is false, and notify must NOT fire again. + notified = nil + heartbeatCtx2, cancel2 := context.WithCancel(context.Background()) + defer cancel2() + h2, err := BuildProductMetrics(context.Background(), heartbeatCtx2, true, false, + productmetrics.BinaryMecated, "test-version", 0, productmetrics.FeatureSnapshot{}, "", port.NopDiagnostics{}, notify) + if err != nil { + t.Fatalf("BuildProductMetrics (second call): %v", err) + } + defer h2.Shutdown(context.Background()) + + if h2.FirstRun { + t.Fatal("want FirstRun=false on the second call against an already-minted install id") + } + if len(notified) != 0 { + t.Fatalf("notify calls = %v, want none when firstRun is false", notified) + } +} diff --git a/internal/configgen/build.go b/internal/configgen/build.go index 8206e8e028..2afeed97c5 100644 --- a/internal/configgen/build.go +++ b/internal/configgen/build.go @@ -36,6 +36,7 @@ func BuildModel(docs Docs) *Model { steerSubtree(docs), modelsSubtree(docs), openRouterSubtree(docs), + telemetrySubtree(docs), mcpSubtree(docs), }} } @@ -398,6 +399,31 @@ func modelsSubtree(docs Docs) *Subtree { } } +func telemetrySubtree(docs Docs) *Subtree { + fields := fieldsOf("TelemetrySection", permconfig.TelemetrySection{}, docs) + for _, f := range fields { + if f.Key == "productMetrics" { + f.Nested = fieldsOf("ProductMetricsSection", permconfig.ProductMetricsSection{}, docs) + } + } + return &Subtree{ + Key: "telemetry", + Tier: TierOperator, + Doc: "OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). " + + "Honoured ONLY from the user-global + CLI tiers; a project-tier telemetry: block " + + "is IGNORED with a WARN (a project repo cannot flip a user's own telemetry choice " + + "in either direction). Omit entirely to fall through to the DO_NOT_TRACK env var " + + "and finally the enabled-by-default posture.", + CommentedOut: true, + Fields: fields, + Example: []string{ + "telemetry:", + " productMetrics:", + " enabled: false", + }, + } +} + // docFor returns the harvested doc for key, or a fallback when absent. func docFor(docs Docs, key, fallback string) string { if d := docs[key]; d != "" { diff --git a/internal/configgen/configgen_test.go b/internal/configgen/configgen_test.go index cbfd94175b..be02be99dc 100644 --- a/internal/configgen/configgen_test.go +++ b/internal/configgen/configgen_test.go @@ -315,6 +315,7 @@ func TestSubtreeTiersAreAsPinned(t *testing.T) { "steer": configgen.TierOperator, // operator-only: a project cannot flip the mid-run steer surface (issue #512) "models": configgen.TierProject, // operator + project (project within the operator allowlist) "openrouter": configgen.TierOperator, // operator-only: a project cannot steer the OpenRouter downstream provider (issue #480) + "telemetry": configgen.TierOperator, // operator-only: a project cannot flip a user's own telemetry choice "mcp": configgen.TierOperator, // operator-only: endpoints, auth, credentials, and egress policy } got := map[string]configgen.Tier{} diff --git a/internal/configgen/settings.skeleton.yaml b/internal/configgen/settings.skeleton.yaml index 013934c45d..86020027a2 100644 --- a/internal/configgen/settings.skeleton.yaml +++ b/internal/configgen/settings.skeleton.yaml @@ -358,6 +358,25 @@ #| order: ["anthropic", "google-vertex"] #| allow_fallbacks: false +#| === telemetry === (tier: operator) +#| OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). Honoured +#| ONLY from the user-global + CLI tiers; a project-tier telemetry: block is IGNORED +#| with a WARN (a project repo cannot flip a user's own telemetry choice in either +#| direction). Omit entirely to fall through to the DO_NOT_TRACK env var and finally the +#| enabled-by-default posture. +# telemetry: +# # ProductMetrics is the opt-out product/adoption metrics config. +# productMetrics: +# # Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit false: nil +# # = absent (composition falls through to DO_NOT_TRACK then the enabled-by-default +# # posture); a non-nil value is honoured exactly. +# enabled: false +#| +#| Example: +#| telemetry: +#| productMetrics: +#| enabled: false + #| === mcp === (tier: operator) #| Strict OPERATOR-TIER Streamable HTTP MCP authority configuration. Mode selects one #| mutually exclusive global or session-broker authority; broker mode carries its diff --git a/user-docs/building/what-you-get/observability.md b/user-docs/building/what-you-get/observability.md index 5ca572fb3d..e39128c383 100644 --- a/user-docs/building/what-you-get/observability.md +++ b/user-docs/building/what-you-get/observability.md @@ -232,6 +232,25 @@ The `jsonlstore` backend (selected with `--store-dir`) implements `ToolCallRecor --- +## Product / adoption metrics (opt-out) + +The four channels above are all **operator-facing**: they help you observe your own deployment. Separately, Mecatl reports a small set of **anonymous, aggregate community-adoption metrics** to Stacklok, over its own independent pipeline (`internal/adapter/productmetrics`) — a distinct concern from everything above, sharing no import, `MeterProvider`, or destination with the operator observability pipeline. Disabling your own OTLP/Prometheus setup has zero effect on this, and disabling this has zero effect on your own OTLP/Prometheus setup. + +**What's collected:** version, OS/arch, an anonymous per-install identifier (a random UUID, unrelated to any user, machine, or organization identity), which major features you have enabled (`memory`, `guardrails`, `mcp`, `scheduling`), your configured LLM provider family (`anthropic`/`openai`/`openrouter`/`other` — never a model id or alias), which binary you're running, and coarse counts — sessions started, runs completed (by stop reason and whether the run made at least one successful tool call), tool calls executed (by bounded category — a built-in tool's own name, or `mcp` for anything MCP-server-provided, never a real MCP server/tool name — and outcome), token counts by kind, run duration, tool calls per run, a one-time-per-install "time to first value" duration, and whether the Subagent/Team delegation families were used at least once. Never a prompt, file path, raw MCP tool/server name, session/run/model identifier, or any other free text. The full catalog and the privacy-guard test discipline that enforces it are recorded in [ADR 0338](https://github.com/stacklok/mecatl/blob/main/docs/adr/0338-product-metrics.md). + +**It's on by default (opt-out).** The first time a run is actually about to send product metrics, Mecatl prints a one-time, non-blocking disclosure to stderr naming what's collected and how to turn it off. To disable it, use any of: + +- `--product-metrics=false` on the command line (all four binaries). +- The `MECATL_PRODUCT_METRICS` environment variable set to `false`/`0` — a mecatl-specific override, checked before `DO_NOT_TRACK` below, so it can also opt you back **in** even if you have `DO_NOT_TRACK` set globally for other tools. +- The `DO_NOT_TRACK` environment variable set to a truthy value (`"0"`/`"false"` do not opt out) — the same convention other tools already respect. +- `telemetry.productMetrics.enabled: false` in your **operator-tier** `~/.config/mecatl/settings.yaml`. This setting is operator-tier only: a project repo's `.mecatl/settings.yaml` cannot change your telemetry choice in either direction. + +**Self-verify before trusting it.** `--product-metrics-dry-run` prints every observation this pipeline would have sent to stderr instead of exporting it, so you can check the "no PII" claim yourself rather than take the docs' word for it. + +**On Kubernetes.** The reported install identifier is an anonymous random UUID. `mecated`, `mecatui`, and `mecatequi` keep it in a local state file, but `mecak8s` is storage-free (no PVC), so a per-pod file would produce a brand-new id on every restart. The Helm chart therefore provisions the id once, into a `-mecak8s-install-id` ConfigMap that is reused across every `helm upgrade`, and mounts it into the container as `MECATL_PRODUCT_METRICS_INSTALL_ID`. Delete that ConfigMap to reset the id, or opt out entirely with any of the switches above — the env var only decides *which* id is used when reporting is on. + +--- + ## What's next To configure Mecatl for production, see the deployment guide for how to wire an OTLP collector, configure the admin listener, and set up session persistence with `jsonlstore`. diff --git a/user-docs/reference/configuration.md b/user-docs/reference/configuration.md index 729856803a..3d914b86f7 100644 --- a/user-docs/reference/configuration.md +++ b/user-docs/reference/configuration.md @@ -246,6 +246,17 @@ OPERATOR-TIER OpenRouter downstream-provider routing (issue #480): a per-model p | `openrouter.models..order` | `[]string` | `(absent)` | Order lists downstream provider slugs (lowercase-kebab, e.g. "anthropic", "google-vertex", "deepinfra/turbo") tried in order. Setting it disables OpenRouter's default price load-balancing. Base-slug matching applies: "google-vertex" matches all its regions/variants (service tiers excepted). | | `openrouter.models..allow_fallbacks` | `bool` | `(absent)` | AllowFallbacks, when explicitly false, pins the request to Order with no fallback to other downstreams. Omit the key to keep OpenRouter's default (true); set it to false to disable fallback. | +## `telemetry` + +Tier: **operator** + +OPERATOR-TIER opt-out product/adoption metrics (telemetry.productMetrics). Honoured ONLY from the user-global + CLI tiers; a project-tier telemetry: block is IGNORED with a WARN (a project repo cannot flip a user's own telemetry choice in either direction). Omit entirely to fall through to the DO_NOT_TRACK env var and finally the enabled-by-default posture. + +| Key | Type | Default | Description | +| --- | --- | --- | --- | +| `telemetry.productMetrics` | `productmetricssection` | `(absent)` | ProductMetrics is the opt-out product/adoption metrics config. | +| `telemetry.productMetrics.enabled` | `bool` | `(absent)` | Enabled is a *bool so ABSENT (nil) is distinguishable from an explicit false: nil = absent (composition falls through to DO_NOT_TRACK then the enabled-by-default posture); a non-nil value is honoured exactly. | + ## `mcp` Tier: **operator**