Repository navigation
Debounce postgres top-N query metrics - #18
Merged
Merged
Conversation
- Reject non-positive deltas in top-N selection: with all-zero deltas (first scrape, idle database, pg_stat_statements reset) sort order was arbitrary and pulled random queryids into the top-set, minting fresh series on every scrape of an otherwise idle database - Emit per-query metrics only after a statement holds a top-20 spot on two consecutive scrapes, keeping one-off queries out of the per-day index; the meta push keeps its undebounced top-30 - Fix data race in the statements phase: the registry is Gathered by both the /metrics handler and the push ticker, so overlapping Collect calls wrote prevStmts and histBuckets concurrently, which is a fatal error that instrumentedCollector's recover cannot catch - Rank an index slice once for both widths instead of sorting copies of stmtCurrent five times per call, twice per scrape: 9.8ms and 10.4MB per scrape down to 1.5ms and 30KB at 5000 statements - Document the selection rule in docs/metrics.md: a flapping query never emits, and per-query metrics reappear on the third scrape after a restart
- Add release.prerelease=auto to .goreleaser.yml. Without it a -rc tag publishes as a full release and GitHub marks it "Latest", displacing the last stable one — agents on the stable channel would then auto-update onto a release candidate - The release job fires on any v* tag, so nothing else gated this. Two earlier rc tag series left no GitHub releases behind, which suggests they were published and then deleted by hand
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
topsrv_pg_query_*metric, and the top-set turned over on nearly every scrape. Two independent causes, both fixed here.selectTopStatementsnow rejects non-positive deltas. Ranking runs on per-scrape deltas, but nothing filtered them, and sort order among all-zero deltas is arbitrary (pdqsort is not stable). On a first scrape, an idle database, or right afterpg_stat_statements_reset(), that pulled up to 20 random queryids per dimension into the top-set on every scrape — minting fresh series from a database doing no work at all. The guard lives inside the shared helper, so both the metrics top-20 and the meta push top-30 get it.debounceTop). Covers the residual case the guard cannot: migrations, tests and ad-hoc statements legitimately top a dimension for exactly one scrape, and each one costs a permanent series. The meta push keeps its undebounced top-30 — query texts are not TSDB series, and the wider window still helps the UI resolve queries hovering on the boundary.a.registryis Gathered by both the/metricshandler (app.go:180) and the push ticker (push.go:167), soCollectcalls overlap and wroteprevStmtsandhistBucketsconcurrently. That isfatal error: concurrent map writes, whichinstrumentedCollector'srecover()cannot catch — the process dies. Pre-existing, but this PR adds state that depends on ordered once-per-scrape calls.stmtCurrentfive times per call and calling it twice per scrape. At 5000 statements: 9.8 ms / 10.4 MB → 1.5 ms / 30 KB per scrape.Design
TryLockrather thanLockon the statements phase: the loser skips instead of waiting. With a plainLockit would run immediately after the winner, recompute deltas over a near-zero window, produce an empty top-set, and wipeprevTopKeys— blinding the next two scrapes on every pull/push collision.prevTopKeysis replaced wholesale each scrape, so it is self-pruning and hard-bounded at 5 dimensions × 20 = 100 keys. It holds nostmtCurrent(208 B each, including query text) and no reference to the scan slice.<= 0guard is deliberately not a "fallback to cumulative values on first scrape". Cumulative ranking pulls in long-lived queries with large historical weight but no current activity — exactly the churn this PR removes.Behaviour change
Both documented in
docs/metrics.mdunder Howtopsrv_pg_query_*statements are selected:TestDebounceTopNeverEmitsAlternatingStatement.Interval = "30s"that is a 60–90 s gap. The aggregatetopsrv_pg_query_duration_secondshistogram is fed from every statement regardless of ranking, so latency percentiles have no such gap.Test plan
go test -race ./...— green in CI (testcheck, 2m17s)make test-integration— postgres / nginx / angie / smart / botlog green; also re-run under-race, no racesinternal/topsrv/postgres/statements_test.go— non-positive deltas,wal_bytesgating, metrics-vs-meta widths, debounce cycle, alternating statementTestIntegrationPostgresextended to 3 scrapes and now assertstopsrv_pg_query_calls_totalis present — without that assert the debounce could suppress per-query metrics forever and the test would stay greenWARNING: DATA RACEonprevStmts/histBuckets, gone once restored. Note CI runsgo test -racewithout theintegrationtag, soTestIntegrationPostgresConcurrentScrapedoes not run there — it needsmake test-integration-benchtime 200x -count 5: old 9.8 ms / 10.4 MB / 18 allocs, new 1.5 ms / 30 KB / 9 allocstopsrv_pg_query_*stops growing, and that a known-hot query still reports within 3 scrapes of an agent restartNote on
make lint: 51 issues (50goconst, 1sloglint), all pre-existing onmasterand all in files this PR does not touch — verified by running the same linter on a cleanmasterworktree. CI does not rungolangci-lint(it runsgo vet+staticcheck), and the local linter version is newer than what the repo was last cleaned against. Not addressed here to keep the diff scoped.Out of scope
Collect()calls, not wall-clock time. With both a listener and a pusher configured, the window depends on who Gathered last, and its strength scales withPush.Interval. Neighbouring state in the package (appNamesTTL,bloatRefreshInterval,settingRefreshInterval) is deliberately time-bounded; this should be too. The real fix is caching a collection for a minimum interval, which is an all-collectors change.topsrv_pg_statements_debouncedgauge would make it diagnosable, but that is new public metric surface for a patch release.