Skip to content

feat(aivision): enable managed query acceleration by default - #12

Merged
webup merged 8 commits into
mainfrom
feat/trace-query-read-model
Oct 25, 2026
Merged

webup merged 8 commits into
mainfrom
feat/trace-query-read-model

Conversation

@webup

@webup webup commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Provide standard revision-checked native Trace and complete-Session query acceleration for AI Vision. Batch selected Session summaries and read verified versioned STORED Token values without separate strategy switches. Missing/mismatched stored definitions retain the internal JSON strategy; database/permission/invalid-result failures remain errors. Known zero and missing usage retain distinct semantics.

Automatically migrate and maintain the derived schema under a shared generation/lease fence. The matching OceanBase provider is the initialization/retry owner; an internal marker prevents ingestion from repeating that initialization while preserving standalone bootstrap behavior. Backfill enrolls up to 25 sources with one UPSERT, retaining the same fence, revision checks, atomic cursor and soft work budget; this reduces statement count without a throughput guarantee. Publication shares one RC transaction/fence/Commit for at most 25 items, retains every per-Trace scope/revision/generation check, and returns success only after Commit; current-chunk SQL or Commit failure rolls back while earlier chunks can remain committed. Healthy startup preserves coverage; interrupted cleanup resumes in 256-row committed batches so raw writers can progress. Maintenance OFF is sticky across replicas; only explicit admin + deployment-key recovery can clear it. Snapshot/publication checks both source revision and maintenance generation, preventing a delayed pre-resume publication from passing after revisions restart.

Add bounded batch source/publication and one-repeatable-read window APIs. Recovery merges clean summaries with dirty/unprojectable/boundary raw Traces, or reads raw only, preserving full canonical content and physical authorization. Source bytes, per-Span bytes, Trace/Span counts and the actual compact JSON envelope are bounded before accumulation. No successful partial list is returned. Ordinary SQL/query deadlines remain unchanged; only the explicit Resume HTTP request gets a four-minute server budget; the management command also performs one normal maintenance iteration and can take longer. The product sends the existing timeout: 240 header for that action so the global API timeout middleware preserves its deadline; other requests remain unchanged.

Only SIGNOZ_AIVISION_PROJECTION_MAINTENANCE_ENABLED remains on this side (default true); product readers use AIVISION_QUERY_MODE=auto|raw. Retired strategy/authoritative flags no longer select runtime behavior. External raw SQL/retention/restore must suspend and rebuild maintenance.

Validation

Current SigNoz 22e936bf647ba2042810d71fa200e6cbbb468aab:

  • 158 offline tests passed; 9 DSN-gated SQL tests passed on seekdb and 9 on OceanBase CE. Both actual API entrypoint packages compile. Includes sticky OFF, schema repair, delayed-generation ABA, snapshot boundaries, batch source equality, Token unknown/zero/numeric behavior and payload limits.
  • Historical projection-only migration probes copied 36,612 summaries: seekdb 35.388s /CE 36.266s, with 11ms /8ms healthy restarts. Those do not measure full source/coverage cleanup. Current-source suspend/recovery checks cover all three derived tables and automatic rebuild; source generations, newer OFF ordering and bounded cleanup are verified separately. All 109,858 original raw rows remain byte-identical; ordinary global SQL deadline stays 10s.
  • Product HTTP replay uses the same frozen data and independent oracles: 126 correctness, three large raw recovery and 126 default indexed timing requests per engine, plus dirty/unprojectable recovery and actual operator command checks. No new model calls.
10k Sessions, P95 ms (n=20) seekdb OceanBase CE
List 863 588
Tokens ≥100 809 564
Tokens ≥0 893 617

The CE full-recovery observation includes a 327.9s management command and a separate 1450.6s later count-poll wait. This remains a material operational cost, separate from ready-list latency; the two timing phases do not measure pure cleanup or database throughput.

Full current-default report and sanitized samples. Earlier adverse distributions and opt-in paired comparisons are retained separately; no cross-time/cross-engine improvement is inferred. Independent implementation/operational reviews closed the confirmed findings. Substantive implementation/review used Codex agents.

Companion product and documentation

https://github.com/ob-labs/ai-vision-ob/pull/114 (2bdb1fbb4b5af0082211174334d32231bf901437)

https://github.com/ob-labs/ai-vision-docs/pull/20 (ff59069a5d9957e0925f141c62506391da72411f)

Tests use prepared Mac/Docker clones and real handlers/store/SQL, with fixed private auth replacing full SigNoz auth while loading its actual timeout middleware; full startup is bypassed. amd64-emulated seekdb and native-arm64 CE have different resource settings. Full service startup/authentication, released images, Kubernetes, cold/WAN latency and capacity are unverified. Matching product/data-plane builds are required for the new managed contract. Raw recovery is bounded but resource-heavy. This PR remains draft; no merge or deployment is performed.

Hosted query-regression passed on the current SigNoz head. The two real-database SQL matrices above are separate local evidence.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 24, 2026
@webup webup changed the title feat(aivision): add revision-checked Trace and Session read models feat(aivision): enable managed query acceleration by default Oct 25, 2026
@webup
webup marked this pull request as ready for review October 25, 2026 12:46
@webup
webup merged commit d6d5d94 into main Oct 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant