Repository navigation
fix(health): add bounded telemetry-store readiness - #13
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When the telemetry database is unavailable, plain
/api/v1/healthstill returns 200. Add anonymousGET /api/v1/ready, backed by the configured telemetry store's read-onlySELECT 1with a one-second request-derived deadline. Failures return fixed HTTP 503 JSON without database details. OceanBase uses the portable store API rather than the unsupported legacy ClickHouse handle.Add native
healthcheck --readyfor scratch images, preserving default local liveness. Keep both root and prefixed probe URLs working withglobal.external_urlpaths. Document the endpoint and its limits. Implementation and validation were assisted by Codex.Issues closed by this PR
Dependency of AI Vision deployment PR #94, addressing the readiness requirement in issue #88. This PR does not close that issue.
Screenshots / Screen Recordings
No visible UI changes. HTTP/native command assertions and real database fault injection cover the behavior.
Additional Information
Earlier isolated validation on 2026-10-25 (readiness commit
d2c84788dcb4ce0bf8e9ee781153fb985aeadb88):go test ./cmd ./pkg/http/readiness ./pkg/apiserver/signozapiserver -count=1passed against the actual packages.CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -tags timetzdata -ldflags '-s -w' ./cmd/community.oceanbase/seekdb@sha256:500910bda197c0bc5f8bbada10c6ad727807308826818c0ea0391844dedc5cc2(Linux/amd64 under emulation; SQL reports seekdb v1.4.0.0). Rebuilt binaries ran in cached scratch-derived runtimes, not newly published images.Provider and gateway IDs remained unchanged, restart counts stayed zero, and a retained SQL canary matched after recovery. All task resources were removed after verification; unrelated deployments were preserved.
Tested source was the worktree based on
d560a6ad5459220af64c851899aae8f696d5b9fe, now committed asd2c84788dcb4ce0bf8e9ee781153fb985aeadb88. Runtime source hashes were checked against that final worktree. Provider binary SHA-256:13fa73798b274f80ff9c449b6b352fcc7965f5101ded9f6f3379b6824792faad.Additional native full-stack qualification — 2026-10-25
The integrated candidate
cafe91a3021829b4ccae499bd70f0ff934130971includes this readiness implementation unchanged plus the shared-SQL-pool shutdown fix in #14. Its full original native Linux/amd64 Dockerfile built successfully; actual provider config ID issha256:4c5313ac782dee939c6ca5cbfc7ecf2cc22edf43e4366b9137ba14492f808823. AI Vision source was68c9ebb65b20be6fac378ca60878f533701d6ad0with gateway imagesha256:a45adbf9e5d0a7129a65596f0d14ee3ca4f31638b8d078652e069cb7da8adf73. These source/dev images use explicitly recorded SDK/base bindings; they were not published.All seven final configurations passed on the existing native Linux/amd64 evaluation VM: two CLI managed Docker cases, four managed/external CE/seekdb Compose cases and one real K3s 1.34.11+k3s1 fixed-Operator CE installation. See AI Vision #94 for the exact inputs, maintained entry points and full evidence.
Database/provider/gateway IDs and restart counts were unchanged, no rescue timer fired, readiness recovered, and retained SQL/trace data matched. In real Kubernetes, a configured 110-second task-only SQL-network fault produced 19 strict HTTP/native samples and 11 Pod/EndpointSlice snapshots: SQL-dependent readiness failed while local liveness stayed healthy; all affected service endpoints became NotReady and recovered without consumer/observer UID or restart changes. Recast and real same-source FE replacement retained SQL, organization/Secret/PVC identities and three traces/nine exact spans. All 105 recorded Compose product stop checks, including 22 provider stops with #14, exited cleanly. Original source assertions and strict stop gates were retained; failed harness-only setup attempts remain separate. Test runtime resources were removed and original VM resources preserved.
Not qualified: real ClickHouse runtime, full repository suite, published release/image artifacts, all supported architectures/Kubernetes versions, full-component cross-release/code/schema upgrades, sustained/peak load, HA or request draining. No hosted CI success is claimed. These are functional bounded-fault observations, not benchmarks. SELECT 1 establishes connectivity; bootstrap and authenticated ingest/query acceptance are separate checks. Existing liveness behavior is unchanged; consumers still require a matching image before using the new readiness interface.