[common] Redact sensitive configuration values in logs and exceptions#8721
Open
XiaoHongbo-Hope wants to merge 13 commits into
Open
[common] Redact sensitive configuration values in logs and exceptions#8721XiaoHongbo-Hope wants to merge 13 commits into
XiaoHongbo-Hope wants to merge 13 commits into
Conversation
Add SensitiveConfigUtils with isSensitive / redactValue / redactMap / redactText. Sensitive values (password, secret, token, credential, access-key, authorization, ...) are masked, keeping the last 4 chars of long values to aid troubleshooting. Apply it on the REST transport: HttpClient no longer echoes an unparsed error response body verbatim into the exception, and RESTUtil no longer interpolates the request body value on encode failure (keeps the body type and Jackson's original reason instead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWKFWMMt1jgU76Fy9x1PgC
HadoopUtils no longer logs the value of forwarded hadoop.* entries at DEBUG. FlinkTableSource and ConfigRefresher route table/option maps through SensitiveConfigUtils.redactMap before logging, so credentials (fs.oss.*, fs.s3.*, ...) are masked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWKFWMMt1jgU76Fy9x1PgC
XiaoHongbo-Hope
force-pushed
the
log_prinnt_fix
branch
from
July 19, 2026 03:28
d12a1ab to
ba384d0
Compare
HadoopUtils now logs the forwarded hadoop.* value through SensitiveConfigUtils.redactValue, so non-sensitive entries keep their value while credentials are masked, instead of dropping every value. Use mock values in the redaction tests.
Address several credential-leak paths: - redactText masked only the first whitespace-delimited word, leaking the token in "Authorization: Bearer <token>"; the value now runs to a quote/comma/brace/ampersand/newline so multi-word values are masked whole. - Passwords, tokens and Authorization values are now fully masked (******) instead of keeping a 4-char tail. - Sensitive-key detection now also covers Azure/S3 credentials (fs.azure.account.key.*, fs.azure.sas.*, fs.s3a.encryption.key). - HttpClient.buildErrorResponse redacts the parsed ErrorResponse.message (previously only the unparsed fallback body was handled) and no longer echoes an arbitrary unparseable body into the exception. - A 2xx response that fails to deserialize now throws a type-only RESTException without the raw body or Jackson's source snippet, which could contain credentials (e.g. token responses).
… log/URI leaks Follow-up hardening after review: - Fully mask every true secret (secret, credential, private-key, encryption-key, api-key, account-key, sas, in addition to password / token / authorization); only identifier-like accessKeyId keeps a trailing hint, mirroring AWS/Azure where the access-key id is loggable but the secret is never shown. accessKeySecret normalizes to contain "secret" so it is fully masked; accessKeyId is not. - Replace redactText's regex with a linear all-or-nothing scan: it had a ReDoS (O(n^2) on ~20KB input) and leaked past commas / quotes / spaces and Azure sig. Any sensitive marker now redacts the whole text. - Add SensitiveConfigUtils.sanitizeUri and strip the query/user-info in LoggingInterceptor and HttpClientUtils.exists so signed-URL credentials (AWS/GCS signature, Azure SAS sig, presigned tokens) do not reach logs or exceptions. - RESTUtil encode-failure keeps only the body type: neither the cause nor Jackson's message is safe when a getter throws with a secret. - Redact dynamic table options in AbstractFlinkTableFactory and the hadoop.* config values logged by OSS/OBS/COSN/Jindo/GS FileIO.
More review follow-ups: - sanitizeUri now strips an embedded user:password@ on the parse-failure path too (previously only the query was removed), and LoggingInterceptor / RESTUtil no longer log or chain the raw URISyntaxException whose message echoes the full URL. - redactText normalizes the text before scanning, so separator variants (api_key, private.key, accessKey, X-Amz-Signature) are caught, plus literal markers for Azure SAS sig. - Sanitize the BLOB signed URL in BlobFormatWriter and FlinkRowWrapper failure logs so token/signature query params do not leak. - buildErrorResponse only labels a body "unparseable" when it truly failed to parse; a parsed ErrorResponse with an empty message is reported as such, and the embedded status matches the resolved code.
…tions sanitizeUri guessed at malformed URIs with a regex, which left user-info behind when the password contains '/' or '@'; it now returns a fixed <invalid-uri> placeholder on parse failure, and also when the host is unresolved but an '@' is present (credential fragments can land in other components). HttpGet/HttpHead construction in HttpClientUtils is wrapped so an invalid URI throws a safe IllegalArgumentException without the original cause/message (which echo the full raw URL); isInvalidUriException recognizes it so the blob write-NULL path is unchanged.
UriReaderFactory.create calls URI.create(input) on the blob URI before it reaches HttpClientUtils. On a malformed signed URL that threw an IllegalArgumentException whose message and cause echo the full URL (token/signature). Parse it defensively and rethrow a safe exception with only the sanitized URI and no cause.
…cation Redirect/protocol failures at the execute stage (e.g. "Circular redirect to <Location>") echo the target URL, which for a signed BLOB URL is a credential. Wrap the execute boundary in HttpClientUtils so these become an IOException with no original message/cause, reporting only the sanitized request URI; also drop the leaking cause in HttpClient.exec and the leaking message in SimpleHttpClient. Move the invalid-URI exception factory into SensitiveConfigUtils so UriReaderFactory and HttpClientUtils share one message prefix, fixing isInvalidUriException misclassifying UriReaderFactory failures. Add circular-redirect and cross-module classification tests.
descriptorFileExists now surfaces the sanitized "Invalid URI: <invalid-uri>" instead of the raw "Illegal character in ...: <uri>" that echoed the URI. Assert the sanitized prefix and that the raw path is not leaked.
… raw URL SimpleHttpClient and HttpClient built the request (new HttpGet/HttpPost/HttpDelete) before entering the safe exec()/execute() boundary, so a malformed URL threw an IllegalArgumentException from the constructor that echoed the full raw URL (query, user-info, signature). Route all construction through HttpClientUtils safe factories that convert a construction-stage failure into SensitiveConfigUtils.invalidUri(url). Add malformed GET/POST coverage in HttpClientTest and a new SimpleHttpClientTest.
A malformed ECS metadata token response makes RESTApi.fromJson throw a JsonProcessingException (which extends IOException) whose message embeds the raw token JSON (AccessKeyId/AccessKeySecret/SecurityToken). It was wrapped keeping the message and cause, leaking the credentials. Parse in a separate try that reports a generic message with no body, snippet, or cause.
XiaoHongbo-Hope
marked this pull request as ready for review
July 19, 2026 13:31
DLFLocalFileTokenLoader had the same leak as the ECS loader: a malformed token file makes RESTApi.fromJson throw a JsonProcessingException embedding the raw token JSON (AK/SK/STS), which was kept as the cause of the final RuntimeException. Catch the parse failure separately and report a generic message with no body or cause; retry is preserved. Add a malformed-token-file test via a test-only retry-count seam.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Credentials (
fs.oss.accessKeySecret,fs.s3a.secret.key, STS tokens, ...) can leak in cleartext into logs/exceptions:HadoopUtilslogshadoop.*askey=value,FlinkTableSource/ConfigRefresherdump whole option maps, and the REST client (HttpClient/RESTUtil) echoes response/request bodies on thetoken path.
Add
SensitiveConfigUtils(inpaimon-api) —redactMap/redactTextmask only sensitive values, keeping keys and the last 4 chars of long values (****alue). Applied at all sites above.Covers Paimon output only; third-party logs need runtime log config. PyPaimon follow-up in a later PR.
Tests
SensitiveConfigUtilsTest,HadoopUtilsSecretLoggingTest.