[aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.0

[brijesh-elastic]

Proposed commit message

aws: backport cloudtrail ingest pipeline improvements and bugfixes to 7.0.x

Backport of the CloudTrail ingest pipeline quality work (elastic/integrations pull 20403) to the
backport-aws-7.0 branch so the 7.0.x line reaches parity with main. The change is limited to
the cloudtrail data stream plus the changelog, root manifest (version 7.0.1) and validation.yml;
the aws-s3 input template and policy tests keep their 7.0 state so external_id is retained.

Enhancements: every ignore_failure is replaced with an ignore_missing flag, a condition guard,
or an explicit on_failure handler that reports to error.message; previously dropped CloudTrail fields
(userIdentity session context, inScopeOf, invokedByDelegate, addendum, tlsDetails.keyExchange,
vpcEndpointAccountId) are now mapped; cloud.provider, cloud.service.name and related.* are
populated. The assumed-role session name is no longer stored in user.changes.*; it is added to
related.user (full email plus local-part prefix when it is an email) while user.name keeps
the IAM role name.

Bugfixes: file.hash.sha256 is no longer set on digest records (the value remains in
aws.cloudtrail.digest.previous_hash_value and related.hash); DeleteObject uses event.type
deletion; Publish, SendCommand and Converse now set event.type; retain: none drops both
keyword and flattened copies; retain: keyword and retain: minimal no longer discard digest
and Insights records; aws.cloudtrail.service_event_details is governed by retain like the
other payload fields.

Relates elastic/integrations issue 20040.

What

Backport of #20403 to backport-aws-7.0 so users on the AWS package 7.0.x
line receive the CloudTrail ingest pipeline improvements and bugfixes. This
brings the cloudtrail data stream to parity with main.

Why

Per #20040, the CloudTrail ingest-pipeline quality work (#20403) is being
backported across every maintained aws line. The 6.x (#20972) and 7.1.x
(#21129) backports are already in place; this PR closes the gap for the
7.0.x line.

What this includes vs. main

• The cloudtrail ingest pipeline, fields.yml, sample_event.json,
  data-stream manifest, docs, and pipeline test fixtures are byte-identical to
  the merged main change (a3291a9 / [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403).
• Package version bumped to 7.0.1 (format_version kept at 3.6.1).

Scope

• Limited to the cloudtrail data stream and package-level files (changelog,
  manifest, validation). No other data streams are modified.
• Like the 6.x backport ([aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-6.x #20972) — and unlike 7.1.x ([aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1 #21129) — the
  assumed-role session-name change ([aws] cloudtrail: move assumed-role session name from user.changes.* to related.user #20318) is folded in here, because it
  was never released on the 7.0.x line. It gets its own changelog entry.
• external_id is intentionally retained in the aws-s3 input template.
  It was only removed in 7.2.0 ([aws] Enable Identity Federation for all remaining policy templates #20527), so it must remain on 7.0.x. The
  input templates and policy tests are deliberately left at their 7.0 state;
  everything else in the data stream matches main.

Enhancements

• Replace every ignore_failure: true with an ignore_missing flag, a
  condition guard, or an explicit on_failure handler reporting to
  error.message, so real faults surface instead of being silently swallowed.
• Add support for CloudTrail fields that were previously dropped.
• Populate the cloud.provider, cloud.service.name and related.* ECS fields.
• Stop storing the assumed-role ARN session name in user.changes.*; the
  session name is now added to related.user for cross-source correlation (when
  it is an email, both the full email and the local-part prefix are added), and
  user.name continues to hold the IAM role name for detection rules ([aws] cloudtrail: move assumed-role session name from user.changes.* to related.user #20318).

Bugfixes

• Stop populating file.hash.sha256 on digest records, where it held the hash
  of the previous digest file while file.path pointed at the current one. The
  value is still available as aws.cloudtrail.digest.previous_hash_value and in
  related.hash.
• Fix DeleteObject using event.type: delete, which is not a valid ECS value.
  It is now deletion.
• Fix Publish, SendCommand and Converse producing no event.type. They
  now use info, start and info respectively.
• Fix retain: none behaving like retain: all. It now drops both the keyword
  and the flattened copies as documented.
• Fix retain: keyword and retain: minimal discarding digest and Insights
  records. The setting governs API call payloads and no longer affects either
  record type.
• Fix aws.cloudtrail.service_event_details surviving retain: flattened and
  retain: none. It is now governed like the other payload fields.

How to test this PR locally

• Check out this branch and start a stack on a supported 9.4+ version, e.g.
  elastic-package stack up -d --version 9.4.0.
• From packages/aws, run: elastic-package test pipeline --data-streams cloudtrail -v

Related

• Backport of [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403
• 6.x equivalent: [aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-6.x #20972
• 7.1.x equivalent: [aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1 #21129
• Relates aws.cloudtrail: Ingest pipeline & ECS mapping improvements #20040
• Closes [aws.cloudtrail] Fix ARN partition grok pattern for aws-cn/aws-iso* (7.0.x backport) #21178

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[efd6]

Suggested change

	value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
	value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

(throughout)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

✅ Package owners are in sync with main.

[vera-review-bot]

Review summary

Issues found across the latest commits 446ad6a — 1 medium

Package-level:

• 🟡 PR description says SVR00009 was added to packages/aws/validation.yml, but the file is unchanged

  Either commit the missing validation.yml change, matching main and the 6.x backport, or drop the sentence from the PR description if the check does not apply on this branch:

  errors:
    exclude_checks:
      - SVR00004 # References in dashboards.
      - SVR00009 # Pipeline on_failure error message format; pre-existing pipeline predates this check.

Issues found across earlier commits 8287d11 — 1 low

Package-level:

• 🔵 Proposed commit message

  aws: backport cloudtrail ingest pipeline improvements and bugfixes to 7.0.x
  
  Backport of the CloudTrail ingest pipeline quality work (elastic/integrations pull 20403) to the backport-aws-7.0 branch so the 7.0.x line reaches parity with main. The change is limited to the cloudtrail data stream plus the changelog, root manifest (version 7.0.1) and validation.yml; the aws-s3 input template and policy tests keep their 7.0 state so external_id is retained.
  
  Enhancements: every ignore_failure is replaced with an ignore_missing flag, a condition guard, or an explicit on_failure handler that reports to error.message; previously dropped CloudTrail fields (userIdentity session context, inScopeOf, invokedByDelegate, addendum, tlsDetails.keyExchange, vpcEndpointAccountId) are now mapped; cloud.provider, cloud.service.name and related.* are populated. The assumed-role session name is no longer stored in user.changes.*; it is added to related.user (full email plus local-part prefix when it is an email) while user.name keeps the IAM role name.
  
  Bugfixes: file.hash.sha256 is no longer set on digest records (the value remains in aws.cloudtrail.digest.previous_hash_value and related.hash); DeleteObject uses event.type deletion; Publish, SendCommand and Converse now set event.type; retain: none drops both keyword and flattened copies; retain: keyword and retain: minimal no longer discard digest and Insights records; aws.cloudtrail.service_event_details is governed by retain like the other payload fields.
  
  SVR00009 is excluded in validation.yml because the pre-existing pipeline predates that check.
  
  Relates elastic/integrations issue 20040.
  

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 446ad6a

History

• 💚 Build #50007 succeeded 8287d11

cc @brijesh-elastic

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[github-actions]

Changelog sync PR created: #21444

[elastic-vault-github-plugin-prod]

Package aws - 7.0.1 containing this change is available at https://epr.elastic.co/package/aws/7.0.1/