feat(ingest): write AI classification columns on the ingest path - #452
Closed
JeremyFunk wants to merge 9 commits into
Closed
feat(ingest): write AI classification columns on the ingest path#452JeremyFunk wants to merge 9 commits into
JeremyFunk wants to merge 9 commits into
Conversation
This was referenced Aug 13, 2026
Migration 0015 adds five trailing columns to `traces` — vendor slug, session-key state, session-key hash, rules version, rollup hour — plus a `set(0)` skip index on the vendor and a token bloom filter on `ScopeName`, which the vendor rules match by prefix. Every column carries a DEFAULT, so the ALTER is metadata-only and rows written before the classifier existed still read back: `AiRulesVersion = 0` means "never examined", distinguishable from an examined-and-non-AI row. Nothing here materializes an index or column, and nothing mutates parts — the 30-day TTL retires the unindexed ones on its own. `requiredForIngest: true`, unlike the last two migrations: the gateway's INSERT now names all five columns, so a BYO-ClickHouse cluster that has not applied 0015 would reject every direct insert. Gating on it is the designed fallback — such an org resolves `clickhouse_ready = false` and routes to the managed pipeline until its schema syncs. The five columns declare snake_case JSONPaths rather than identity ones. That distinction is load-bearing and now also asserted: the insert-mapping generator drops a column that has a DEFAULT *and* an identity path, on the assumption the warehouse computes it. These are emitted on every span, so they must not match that shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The row builder now classifies each span and stamps the five columns. Inputs are built once per accepted payload, not per span: the migration-window flag is read once and the batch receive time is captured once, so every span in one payload clamps against the same instant. `AiRollupHour` is written unconditionally, flag on or off. It is the rollup's partition key and the span timestamp is attacker- and replay-controlled, so it is clamped at write time to `[receive - 7d, receive + 1d]`. Clamping in the view instead would need `now()`, which a later partition rebuild re-evaluates and which would silently relocate rows across hours. On the attribute-mapping path the classifier reads a first-occurrence-wins view of the wire attributes rather than the row's stored Map, which keeps last-wins canonicalization. The two rules only disagree on a span carrying a duplicate rule-key, and the verdict must not depend on whether the org happens to have mapping rules configured. Observability is batch-level, never per span — a span per classification on this path is what the self-observability rule forbids. The accept span carries whether the flag was on and how many spans were examined; `ingest_ai_spans_examined_total` is labeled by signal only, exactly like `native_rows`, so the two series are directly comparable and any divergence is a bug. Also here: - An adversarial fixture module driving `encode_traces` end to end, with a reproducibility check and a branch-coverage check over the written rows. - A ClickHouse E2E pinning `AiSessionKeyHash` to `cityHash64`. Without it a divergence returns zero rows and puts a permanent discontinuity in a 400-day-TTL sketch, with nothing else failing — so CI runs it, and the ClickHouse job's path filter now also watches the CityHash port. - A schema probe asserting the live `traces` columns against the generated schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Local schema v5: the same five defaulted columns and two skip indexes on `traces`, no new objects. The v4 -> v5 module and a frozen v5 DDL snapshot keep an existing local store readable after the generated current schema advances, and the manifest gate now checks that snapshot's identity the way it already checks v1 through v4. The local OTLP encoder stamps the same five fields, so a local store and the hosted warehouse hold the same shape for the same span. Asserted as a column and index delta against the frozen v4 manifest rather than a whole-manifest snapshot, so a stray table or a rewritten column cannot ride along on this version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same pass as the base branch: design-doc references, review-process narrative, and derivation history removed; constraints and traps kept. Comments only — code verified byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JeremyFunk
force-pushed
the
ai2/02-ingest-write-path
branch
from
August 17, 2026 10:48
ee03d73 to
64926a7
Compare
INGEST_AI_CLASSIFICATION_ENABLED is gone before it ever shipped: every trace batch now classifies, `AiClassificationSettings` shrinks to the batch receive time, and the flag-off row shape (`AiRowFields::UNEXAMINED`, rules_version 0) can no longer be written — version 0 now strictly means a pre-rollout row. The eve + vercel_ai_sdk session ladder joins the adversarial corpus while the vercel no-candidates cases move to their new honest state 2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
It only runs when packages/domain/src/ai/** or apps/ingest/src/cityhash102.rs change, but even then it's not worth a dedicated ClickHouse E2E leg on every touching commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The column was written on every span purely to pre-provision a future hourly rollup (service_ai_vendors_hourly, which does not exist). v1 keeps only the four classification columns that have readers; if the rollup ever ships, its migration adds the partition-hour column then. Removing it also deletes the whole receive-time clamp apparatus: the Rust rollup_hour_secs/format_datetime_secs pair, the TS port in the CLI encoder, and AiClassificationSettings, whose only field was the clamp anchor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🍁 Maple PR previewNote Preview resources were removed when this pull request closed. Final commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 2 of 3. Base:
ai2/01-ai-classifier(adds the classifier). Next:ai2/03-vendors-rollup(the hourly rollup that reads these columns).What this does
Runs the classifier on every span during ingest and stores its verdict in five new columns on
traces:AiVendorLowCardinality(String)''= not AI. Vendors are not enumerated in the schema — adding one is a code deploy, never a migration.AiSessionKeyStateUInt8AiSessionKeyHashUInt64cityHash64of the winning session key (0 below state 5) — the fixed-width grouping key for session counting.AiRulesVersionUInt32AiRollupHourDateTime('UTC')The columns land in all three backends the
tracestable lives in: ClickHouse migration0016, the Tinybird datasource, and the CLI's local store (schema v6 — the 1,746-line SQL file is a generated full-schema snapshot, the local store's existing convention).Rollout and safety
INGEST_AI_CLASSIFICATION_ENABLED(default off), read once per batch. Flag off still writes zeros plus a realAiRollupHour.MATERIALIZE INDEX) — applying it is instant, and pre-existing rows read the defaults.AiRollupHouris clamped to[receive − 7d, receive + 1d](else receive time). Span timestamps are client-controlled and this column becomes a partition key in PR 3 — unclamped, a misbehaving client could mint unbounded partitions whose TTL never fires.Details worth knowing
Testing
city_hash64of the winning key, so writer and classifier cannot drift apart silently.city_hash64== ClickHousecityHash64over the adversarial keys, covering the length bands where crates.io CityHash (1.1+) diverges from the 1.0.2 variant ClickHouse vendors.cargo test: 136 lib + 70 bin.@maple/domain487,@maple/cli435, schema checks green.🤖 Generated with Claude Code
Update 2026-08-18 (
d943f3ee4):INGEST_AI_CLASSIFICATION_ENABLEDis removed before ever shipping — classification is now unconditional on every trace batch.AiClassificationSettingsshrinks to the batch receive time, the flag-off row shape can no longer be written, andAiRulesVersion = 0now strictly means a pre-rollout row. The adversarial corpus gains eve + vercel_ai_sdk session-state ladders (artifact regenerated).