feat(domain): hourly per-service AI vendor discovery rollup - #453
Closed
JeremyFunk wants to merge 9 commits into
Closed
feat(domain): hourly per-service AI vendor discovery rollup#453JeremyFunk wants to merge 9 commits into
JeremyFunk wants to merge 9 commits into
Conversation
This was referenced Aug 13, 2026
Migration 0016 adds `service_ai_vendors_hourly` and the materialized view that writes it: one row per (org, service, vendor, hour), carrying span and session-key-state counters plus `uniqCombined(12)` states for traces and sessions. It backs service pruning, per-service vendor lookup, session-key health and sampling-exemption suggestions. The view filters `AiVendor != ''`, which is both the cost model and the semantics: the platform's ordinary HTTP and DB spans never enter MV processing, and after enablement "no rows" means "no AI spans". It groups on the stored, clamped `AiRollupHour`, never `toStartOfHour(Timestamp)`, so a skewed client cannot open a partition in 2038. No POPULATE and no backfill: correctness depends on the source rows having been classified, not on the view having existed. Because the view runs synchronously inside the INSERT pipeline, this target's part-count and merge-lag are trace-ingestion health — a `too many parts` here fails the INSERT into `traces`. `requiredForIngest` stays false all the same: the gateway never writes this table, and bumping the ingest gate would un-ready every BYO-ClickHouse org over a read-path change. Reader contract, pinned in the datasource doc and the tests: plain aggregates plus `uniqCombinedMerge(12)`, never `FINAL`, and never an assumption of one row per key. Headline coverage merges across vendor rows — a per-vendor ratio systematically understates co-occurring passthrough vendors. Hours before the recorded enablement hour do not exist for readers, and that boundary is an operator step, not something the migration can enforce; a partly-classified hour is internally consistent and looks healthy. 400-day retention over a 30-day source is deliberate and now the matrix's only 400-day tier: past the raw horizon these rows cannot be rebuilt. The migrated view body and the bootstrapped one come from one exported constant, and a test asserts neither copy drifted: a cluster migrated to 16 and a freshly bootstrapped one must compute the same coverage ratio. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Local schema v6: the rollup table and its view, and nothing else — asserted as a name delta against the frozen v5 manifest so a stray object cannot ride along. The v5 -> v6 module and a frozen v6 DDL snapshot keep an existing local store readable, and the manifest gate checks that snapshot's identity like the ones before it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rollup's output is a coverage percentage shown to a customer, and every way it can be wrong renders fine: a broken counter identity, an HLL state merged at the wrong grouping, a reader assuming merged parts. Only a real server over spans with known-by-construction answers catches those, so this suite runs against one — asserting the counter identities, the trace-level coverage ratio merged across vendors, the hour grouping, and that reading without `FINAL` over deliberately unmerged parts still gives the right answer. Wired into `ch:test` and into the ClickHouse CI job alongside the existing warehouse suites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JeremyFunk
force-pushed
the
ai2/03-vendors-rollup
branch
from
August 17, 2026 10:55
3584d63 to
68db7e7
Compare
Same pass as the earlier stack branches. Also drops migration 0017's stale claim that the ingest gate is "15" — index.test.ts pins 16. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… flag The flag is gone (ingest classifies unconditionally), so the boundary is the hour the classifying gateway binary reaches the whole fleet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml
02 dropped the AiRollupHour column, so the rollup now buckets on toStartOfHour(toDateTime(Timestamp)) — the same span-time hour every other hourly MV uses. The clock-skew clamp test went with it: a skewed client now lands in the rollup exactly where it lands in raw traces. Local-store v7 snapshot and pinned identity hashes regenerated on top of the new v6. Also corrects the two e2e migration-version assertions that still pointed one version back after the earlier renumbering (15→16 classification, 16→17 rollup); both passed vacuously before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🍁 Maple PR previewNote Preview resources were removed when this pull request closed. Final commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 3 of 3. Base:
ai2/02-ingest-write-path(writes the AI columns). This PR adds the aggregate that makes them queryable at dashboard speed.What this does
Adds
service_ai_vendors_hourly: an hourly rollup of AI activity per (org, service, vendor), fed by a materialized view overtraces. It answers "which AI frameworks does each service run, how many spans/traces, and how good is session-key coverage" without scanning raw traces — and it's retained 400 days, far past the 30-day raw-trace TTL, so vendor history survives the raw data.Per row (one per org × service × vendor × hour):
uniqCombined(12)sketches: traces total, traces with a session key, and distinct sessions (viaAiSessionKeyHash). Sketches merge across hours/services at query time; 12 gives ~1.6% error.The same definition lands in all three backends — ClickHouse migration
0017, the Tinybird datasource + materialization, and the CLI's local store (schema v7, generated snapshot). The MV SELECT is one shared constant (ai-vendors-rollup-sql.ts); a test asserts the migration and the Tinybird materialization stay byte-identical, so the two deployment paths can't drift.Rollout
0017is not ingest-gating — it touches nothing the gateway inserts, so BYO clusters aren't un-readied over a read-path table.AI_VENDORS_ROLLUP_ENABLEMENT_HOUR; readers must treat earlier hours as nonexistent, not zero — "no AI spans" and "wasn't recording" are different claims. Nothing consumes it yet; the read path is a later stack.ORDER BY (OrgId, …)prefix keeps prunable.Testing
@maple/domain490,@maple/cli435,cargo test136+70, repo typecheck 40/40 green.🤖 Generated with Claude Code
Update 2026-08-18 (
5678f325b): the classification flag no longer exists (see the base PR), so the rollout runbook is restated flag-free —AI_VENDORS_ROLLUP_ENABLEMENT_HOURis the first full hour after the classifying gateway binary reaches the whole fleet. The reader rule is unchanged: hours before the boundary are nonexistent, not zero.