docs: propose a tiered pre-deployment regression gate

[amiddavid]

Docs only — one new file under docs/proposals/, which is excluded from the mkdocs
nav, so no nav edit and no site change. Nothing here is built yet; this is for
argument before any code.

Why

Commits and reviews are increasingly machine-authored and humans steer rather than
read every diff, so nothing currently asserts that a build is not worse than the last
one before it reaches /usr/local/bin/context-guru-proxy. CI proves the code
compiles, the tests pass and the binary starts. It proves nothing about configuration
drift, savings, or capability.

Running SWE-bench per deploy is not the answer, and the cost is the weaker half of
why. From docs/results/: two arms at one trial each is ~$60 and ~10 h of agent wall.
The stronger objection is that reward is the worst signal per dollar available — the
published arms separated 43 vs 44 of 50, so a 12-task subset (sd ≈1.2 tasks at p≈0.86)
cannot distinguish an 88% pipeline from a 78% one. A gate that samples the noisiest
metric and gets read as "no capability regression" is worse than no gate, because a
deploy proceeds on it.

So the proposal tiers by what each signal costs to measure, and gates hardest on the
lowest-variance quantity available.

Shape

• Tier 0 — free, every PR. Goldens over the effective config (main.go:983
  already assembles it for the dashboard), the preset set, the route/auth-class table,
  and an AST-derived env-var contract. This is the axis that has actually broken this
  deployment twice: key_env outranking the code on who pays, and
  TENANT_MONTHLY_CAP_USD left in the unit after the binary stopped reading it. A
  golden diff is also how an intended config change gets spotted rather than
  slipping through.
• Tier 1 — cents. Frozen-corpus replay through /compact, which runs the pipeline
  with no upstream call. deploy/harbor/replay.py already does the middle of this;
  what is missing is a pinned corpus and a committed baseline. Scope is limited — see
  below.
• Tier 2 — ~$10/RC. A ~12-task SWE-bench subset selected from the existing 50-task
  record by four stated criteria, gating hardest on hard failures and per-task cost,
  and only loosely on reward.
• Post-deploy verify-deployed, which closes the loop REDEPLOY-caller-pays.md
  walks by hand.

The two points most worth reviewing

Tier 1 measures content, not money, and only for some components. Both corrections
came from reading the code:

• The dollar valuation classifies each request into a billing tier from
  r.cache_read / r.cache_write / r.fresh_input (dash/readvalue.go:176), and
  those come from the upstream response's usage block. /compact never calls upstream,
  so all three are zero and every row classifies as fresh — the most expensive tier.
  Since 90.74% of input actually bills as cache reads at 12.6x less
  (measured-2026-08.md), that overstatement is large. Content tokens are exact here;
  dollars only under a frozen tier mix carried in the baseline, and explicitly not a
  claim about the bill; billed tiers and cache-write growth are tier 2 only.

• Of 26 registered components, 16 are a pure function of the request body and are
  exactly gateable; three call a model on it and are gateable only against recorded
  responses, which measures the plumbing rather than the savings; three depend on live
  cross-turn or cache state and cannot be measured at all. That last class passes
  vacuously rather than failing, which is why it needs naming: extract_llm_sweep
  needs the previous turn's forwarded body from the session stash
  (proxy/prefixask.go:112), a replay forwards nothing, so every record is a first
  turn — ErrNoPrefix, refused locally, no request made (extract_sweep.go:795). It
  declines on 100% of a corpus and removes 0 tokens, so a gate including it reports
  0 -> 0, unchanged, pass on a component that could be entirely broken. cachesplit
  is a marker that always skips (the edit is body-level in apply/prefixsplit.go) and
  cacheinject places breakpoints, so both have effects that exist only in provider
  cache accounting.

  Consequently tier 1 is described as the savings gate for the deterministic
  offloaders plus a universal integrity gate — not as "the expensiveness gate" — and
  the cache lever, which carries the largest measured effect in the product (−34.1%
  cost, 0% -> 96.7% hit in isolated A/B), sits outside it.

Baselines rot for reasons that are not us — model, gateway, task images, price
list. The recommendation is a single arm plus a cheap live drift canary that marks the
baseline stale when the upstream moves, rather than paying 2x for paired arms every
run; plus a ratchet rule so "progress moved the baseline" and "a regression moved the
baseline" do not look the same in the log.

The proposal also requires a cg-gate selftest that mutates each subject and asserts
the corresponding check fails, and three verdicts (pass / fail / unmeasured), so
a corpus that failed to load never reads as "nothing broke".

Two findings noted in passing

Recorded in the doc (§1, §9) so they are not lost, not fixed here:

• A grep suggests the tree reads 38 operator-facing environment variables, 10 of which
  appear nowhere under docs/ (CG_BASE, CG_FILTER_REMOVE, CG_LIVE*,
  CG_SMTP_LIVE_TO, CG_TRIGGER_CF_MAX, CHEAP_MODEL_AUTH, KVCACHE_PYTHON). That
  is a grep and not the AST inventory §3.2 proposes, so treat the exact list as
  indicative.
• deploy/eval-containers/sweep.py hardcodes ROOT to one engineer's laptop path, so
  it runs nowhere else as written.

Open decisions

§11 lists four. The first blocks phase 2: the corpus is real traffic, so where it is
hosted, what scrubbing it gets and who approves its use needs an answer before any of
it is built.