Skip to content

fix(logs): deliver raw gzip payload to Firehose subscription destinations - #2720

Merged
vieiralucas merged 1 commit into
mainfrom
fix/logs-firehose-raw-gzip
Oct 16, 2026
Merged

vieiralucas merged 1 commit into
mainfrom
fix/logs-firehose-raw-gzip

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

CloudWatch Logs subscription filters with a Firehose destination passed encoded.as_bytes() (the base64 text of the gzip payload) as the Firehose record, so the delivered S3 object contained base64 text rather than gzip. In AWS the Firehose record is the gzip-compressed CloudWatch Logs payload itself, and the object is readable with zcat. Base64 is only the wire encoding for SQS/Lambda (and the Kinesis path already decodes it back to bytes).

Found while reviewing #2715. Adds a unit test asserting the record starts with the gzip magic and decodes to the DATA_MESSAGE payload.


Summary by cubic

Fixes CloudWatch Logs subscription filters delivering base64 text instead of the gzip payload to Firehose destinations, so delivered S3 objects now contain gzip data readable with zcat as in AWS.

  • Sends the raw compressed payload to Firehose; base64 remains only the SQS/Lambda/Kinesis wire encoding.
  • Adds a unit test asserting the record starts with the gzip magic and decodes to a DATA_MESSAGE payload.

Written for commit 38e705e. Summary will update on new commits.

View guided diff Turn on auto-fix

…ions

CloudWatch Logs subscription filters targeting Firehose passed the
base64 text of the gzip payload as the record, so delivered S3 objects
held base64 instead of gzip. In AWS the Firehose record is the gzip
payload itself; base64 is only the SQS/Lambda/Kinesis wire encoding.
@vieiralucas
vieiralucas force-pushed the fix/logs-firehose-raw-gzip branch from 32d51a5 to 38e705e Compare October 16, 2026 10:20
@vieiralucas
vieiralucas merged commit cce49ba into main Oct 16, 2026
255 of 256 checks passed
@vieiralucas
vieiralucas deleted the fix/logs-firehose-raw-gzip branch October 16, 2026 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant