Skip to content

fix(health): add bounded telemetry-store readiness - #13

Merged
webup merged 1 commit into
mainfrom
codex/sql-readiness
Oct 25, 2026
Merged

webup merged 1 commit into
mainfrom
codex/sql-readiness

Conversation

@webup

@webup webup commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Description

When the telemetry database is unavailable, plain /api/v1/health still returns 200. Add anonymous GET /api/v1/ready, backed by the configured telemetry store's read-only SELECT 1 with a one-second request-derived deadline. Failures return fixed HTTP 503 JSON without database details. OceanBase uses the portable store API rather than the unsupported legacy ClickHouse handle.

Add native healthcheck --ready for scratch images, preserving default local liveness. Keep both root and prefixed probe URLs working with global.external_url paths. Document the endpoint and its limits. Implementation and validation were assisted by Codex.

Issues closed by this PR

Dependency of AI Vision deployment PR #94, addressing the readiness requirement in issue #88. This PR does not close that issue.

Screenshots / Screen Recordings

No visible UI changes. HTTP/native command assertions and real database fault injection cover the behavior.

Additional Information

Earlier isolated validation on 2026-10-25 (readiness commit d2c84788dcb4ce0bf8e9ee781153fb985aeadb88):

  • Go 1.25.7: go test ./cmd ./pkg/http/readiness ./pkg/apiserver/signozapiserver -count=1 passed against the actual packages.
  • Focused handler, native-command and prefix regressions also passed with race detection and shuffled order. Regression runs reproduced the missing readiness mode and prefixed-root 404 before fixes.
  • Full static Linux/arm64 community binary built with CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -tags timetzdata -ldflags '-s -w' ./cmd/community.
  • Real Docker fault/recovery on OrbStack 29.4.0 Linux/aarch64, with pinned oceanbase/seekdb@sha256:500910bda197c0bc5f8bbada10c6ad727807308826818c0ea0391844dedc5cc2 (Linux/amd64 under emulation; SQL reports seekdb v1.4.0.0). Rebuilt binaries ran in cached scratch-derived runtimes, not newly published images.
State Provider liveness Provider SQL readiness Gateway native readiness
Available HTTP 200 / native exit 0 HTTP 200 / native exit 0 Exit 0
Only database paused HTTP 200 / native exit 0 HTTP 503 in 1.004s / native exit 1 Exit 1 in 1.052s; gateway liveness stayed exit 0
Database recovered HTTP 200 / native exit 0 HTTP 200 in 0.003s / native exit 0 Exit 0 in 0.121s

Provider and gateway IDs remained unchanged, restart counts stayed zero, and a retained SQL canary matched after recovery. All task resources were removed after verification; unrelated deployments were preserved.

Tested source was the worktree based on d560a6ad5459220af64c851899aae8f696d5b9fe, now committed as d2c84788dcb4ce0bf8e9ee781153fb985aeadb88. Runtime source hashes were checked against that final worktree. Provider binary SHA-256: 13fa73798b274f80ff9c449b6b352fcc7965f5101ded9f6f3379b6824792faad.

Additional native full-stack qualification — 2026-10-25

The integrated candidate cafe91a3021829b4ccae499bd70f0ff934130971 includes this readiness implementation unchanged plus the shared-SQL-pool shutdown fix in #14. Its full original native Linux/amd64 Dockerfile built successfully; actual provider config ID is sha256:4c5313ac782dee939c6ca5cbfc7ecf2cc22edf43e4366b9137ba14492f808823. AI Vision source was 68c9ebb65b20be6fac378ca60878f533701d6ad0 with gateway image sha256:a45adbf9e5d0a7129a65596f0d14ee3ca4f31638b8d078652e069cb7da8adf73. These source/dev images use explicitly recorded SDK/base bindings; they were not published.

All seven final configurations passed on the existing native Linux/amd64 evaluation VM: two CLI managed Docker cases, four managed/external CE/seekdb Compose cases and one real K3s 1.34.11+k3s1 fixed-Operator CE installation. See AI Vision #94 for the exact inputs, maintained entry points and full evidence.

Native managed DB fault seekdb OceanBase CE
Only DB paused 3.4281 s 3.4831 s
Provider HTTP/native liveness 200 / exit 0 200 / exit 0
Provider SQL HTTP readiness 503 in 1.0017 s 503 in 1.0017 s
Provider native readiness Exit 1 in 1.1268 s Exit 1 in 1.1363 s
Gateway native readiness Exit 1 in 1.0524 s; live exit 0 Exit 1 in 1.0520 s; live exit 0
Recovery including Docker health polling 11.5169 s 10.7531 s

Database/provider/gateway IDs and restart counts were unchanged, no rescue timer fired, readiness recovered, and retained SQL/trace data matched. In real Kubernetes, a configured 110-second task-only SQL-network fault produced 19 strict HTTP/native samples and 11 Pod/EndpointSlice snapshots: SQL-dependent readiness failed while local liveness stayed healthy; all affected service endpoints became NotReady and recovered without consumer/observer UID or restart changes. Recast and real same-source FE replacement retained SQL, organization/Secret/PVC identities and three traces/nine exact spans. All 105 recorded Compose product stop checks, including 22 provider stops with #14, exited cleanly. Original source assertions and strict stop gates were retained; failed harness-only setup attempts remain separate. Test runtime resources were removed and original VM resources preserved.

Not qualified: real ClickHouse runtime, full repository suite, published release/image artifacts, all supported architectures/Kubernetes versions, full-component cross-release/code/schema upgrades, sustained/peak load, HA or request draining. No hosted CI success is claimed. These are functional bounded-fault observations, not benchmarks. SELECT 1 establishes connectivity; bootstrap and authenticated ingest/query acceptance are separate checks. Existing liveness behavior is unchanged; consumers still require a matching image before using the new readiness interface.

@github-actions github-actions Bot added the bug Something isn't working label Oct 25, 2026
@webup
webup merged commit b398103 into main Oct 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant