aws.cloudtrail: Ingest pipeline & ECS mapping improvements

[kcreddy]

Summary

Improve the cloudtrail data stream ingest pipeline (packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml, ~2190 lines) so events are fully normalised and enriched, focusing on ECS related.* coverage and ECS version currency.

Current state

• Pipeline already applies geoip to source.ip → source.geo and source.as (tags geoip_source_ip_to_source_geo_*, geoip_source_ip_to_source_as_*). Enrichment for source IP is present.
• Every processor already carries a tag: (99 tags) and _ingest.on_failure_processor_tag is surfaced in error handling (10 on_failure blocks). The pipeline-tag modernisation is already done — do not re-do it.
• user_agent parsing is present.
• ECS related.user (6) and related.hash (1) are populated, but related.ip and related.hosts are not populated at all (0 occurrences).

Proposed changes

ECS & enrichment

• Populate related.ip — at minimum append source.ip (from CloudTrail sourceIPAddress) and any resolved destination/target IPs.
• Populate related.hosts where a hostname/host identifier is available (e.g. from user_identity/resources or EC2 instance context).
• Confirm related.user covers all principal identities (root, IAM user, assumed role, federated).
• Verify source.ip is only geo-enriched when it is a real IP — sourceIPAddress is often an AWS service DNS name (e.g. cloudtrail.amazonaws.com); ensure those are not fed to geoip and don't produce failures.

Robustness (verify, likely already fine)

• Spot-check the 23 ignore_missing usages for redundant ignore_missing: true + if guards and simplify where found.
• Confirm partial-failure paths append to error.message with the processor tag (already wired in 4 places) across all on_failure blocks.

Exception-handling hygiene

The pipeline has 50 ignore_failure, 10 script, and 2 grok processors. The expensive part of a failure is the exception being thrown; ignore_failure: true and on_failure both catch an already-thrown exception, so converting between them is not a performance win (on_failure is marginally heavier — it allocates a wrapper exception and writes failure metadata). Optimise by not throwing, and by matching the handler to intent:

• Audit the 50 ignore_failure: true usages. Where the only expected failure is a missing field, replace with ignore_missing: true or an if guard so the processor doesn't throw on the common path.
• For the script/grok processors that fail on expected input, add if guards (or restructure) so the common path doesn't throw. Keep ignore_failure: true on these as a last-resort safety net: the guard removes the frequent throw cost, while ignore_failure catches the rare unexpected throw so the pipeline continues instead of bubbling to the global on_failure handler.
• Where a failure is genuinely unexpected and a user would want to see it, prefer an on_failure that appends to error.message over silent ignore_failure. Do not attach per-doc error.message to grok/script whose failures are expected and frequent — that adds noise and cost.

Rationale & impact

related.* fields power cross-source pivoting in Security; missing related.ip/related.hosts means CloudTrail events don't join IP/host-centric investigations. An ECS bump keeps mappings current and unlocks newly standardised fields.

Acceptance criteria

• Pipeline tests pass and new expectations assert related.ip/related.hosts population.
• elastic-package check passes with no ECS mapping errors.
• Changelog entry added and aws package version bumped.

References

• Skills applied from https://github.com/elastic/integration-skills: ingest-pipelines, maintain-integration, ecs-field-mappings.
• ECS: https://www.elastic.co/guide/en/ecs/current/index.html
• SIEM field reference: https://www.elastic.co/guide/en/security/current/siem-field-reference.html

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

Backport plan for #20403

On the scope of the #20403 backports, the team's decision is to make all the backport branches in parity.

What we agreed

The full change from #20403 (enhancements and bugfixes together, cloudtrail data stream only) was already merged to backport-aws-6.x in #20972. To avoid an asymmetric state where 6.x users have the full pipeline rewrite but 7.x users don't, we agreed to carry the same change into every maintained 7.x line as an explicit, one-off exception to the bugfix-only backport rule:

Branch	Kibana	PR
backport-aws-6.x	8.19.x, 9.2–9.3	#20972 (merged)
backport-aws-7.0	9.4.x	TODO
backport-aws-7.1	9.4.x–9.5.x	#21129
backport-aws-7.2	9.6.x	TODO
backport-aws-7.3	9.6.x	TODO

@brijesh-elastic will open the remaining PRs following the same shape as #21129.

Why this is an exception

The repo's backport process is bugfix-only with a patch version bump. #20403 shipped its bugfixes and its enhancements as a single pipeline rewrite, so a bugfix-only backport would mean re-implementing each fix against the older pipeline rather than cherry-picking tested code. Given that 6.x already had the full change, carrying it to 7.x was judged lower risk than diverging. This is not a precedent.

Rule going forward

• By default, only bugfixes are backported.
• Bugfixes found during quality or enhancement work land as their own PR, or at minimum their own commit with a separate type: bugfix changelog entry, before the enhancement lands on top. This keeps backports as clean cherry-picks.
• Enhancements may still be backported when a customer needs them, but only as an explicit exception that goes through approval, stated in the PR description, and not as a side effect of a mixed PR.

[kcreddy]

Follow-up question from @brijesh-elastic that I think we should settle before opening the remaining backports.

Looking at the Kibana constraints at each base commit: 6.x is ^8.19.4 || ^9.2.1, 7.0 and 7.1 are both ^9.4.0, and 7.2 and 7.3 are both ^9.6.0 (same as main). There is no Kibana bump or breaking change between 7.0 and 7.1, nor between 7.2, 7.3 and 7.4, so users on 7.0.x can take 7.1.x and users on 7.2.x/7.3.x can take 7.4.x or 8.x. Fleet also only offers the latest compatible version, so a 7.0.1 or 7.2.1 is only reachable by installing an explicit version via the API.

@efd6, what population are backport-aws-7.0, -7.2 and -7.3 (#21161) meant to serve? If it's a specific customer pinned to one of those minors, could we record that on the .backports.yml entry with a maintained_until? If not, I'd propose:

• Full [aws.cloudtrail] Ingest Pipeline Improvments and Bugfixes #20403 backport to 7.1 only ([aws.cloudtrail] Backport ingest pipeline improvements and bugfixes to backport-aws-7.1 #21129), since that's the only stranded population (Kibana 9.4–9.5).
• Your narrow ARN fix ([aws.cloudtrail] Fix ARN partition grok pattern for aws-cn/aws-iso* (7.0.x backport) #21178, [aws.cloudtrail] Fix ARN partition grok pattern for aws-cn/aws-iso* (7.2.x backport) #21179, [aws.cloudtrail] Fix ARN partition grok pattern for aws-cn/aws-iso* (7.3.x backport) #21180) to 7.0/7.2/7.3, then archive those three branches.
• Going forward, create a backport branch only when main crosses a Kibana/agent constraint bump or a breaking change that strands users, based on the last minor before the boundary. That's how 6.x was done and is what the backport doc describes. Under that rule we don't create 7.4, 8.0, 8.1 and so on.

Happy to be corrected if there's a reason for the per-minor branches that I'm missing.

[brijesh-elastic]

Backport PRs of #20403

Branch	Kibana	PR
backport-aws-6.x	8.19.x, 9.2–9.3	#20972 (merged)
backport-aws-7.0	9.4.x	#21393
backport-aws-7.1	9.4.x–9.5.x	#21129
backport-aws-7.2	9.6.x	#21394
backport-aws-7.3	9.6.x	#21395