Include TSDB head max time in out of bounds and too old sample errors#7695
Open
yeya24 wants to merge 2 commits into
Open
Include TSDB head max time in out of bounds and too old sample errors#7695yeya24 wants to merge 2 commits into
yeya24 wants to merge 2 commits into
Conversation
When the ingester rejects a sample with an out of bounds or too old sample error, the error message now includes the TSDB head max time in addition to the rejected sample timestamp. This lets users directly see how far behind the accepted time range the rejected sample is, making these rejections self-diagnosable. Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by: Ben Ye <benye@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
When the ingester rejects a sample with
out of boundsortoo old sample, the error message only contains the rejected sample's own timestamp:Users cannot tell why the sample was too old — there is no reference point showing how far behind the acceptable time range the sample is. This PR includes the TSDB head max time in the error message for these two rejection reasons:
With this, users can directly self-diagnose these rejections (e.g. see that the sample is 1h behind the newest ingested data), instead of having to correlate with ingester internals. If the head max time is unset (empty head), the error falls back to the previous format.
The error is returned to the client (via the distributor) and also surfaced in the ingester's existing
partial failures to pushlog line, so the head max time is visible in both places.Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags