[panw] Fix audit log parsing on the tcp and udp syslog inputs

[robester0403]

Proposed commit message

The tcp and udp inputs run the syslog processor on the agent, which strips the
RFC 3164/5424 header before the ingest pipeline sees the event. The header grok in
default.yml only had a pattern for the audit CSV with that header still attached,
so every AUDIT event collected over syslog failed at the header and never reached
audit.yml. Only the logfile input, which does not run the syslog processor, could
parse audit logs. Audit logs are also the only PAN-OS type that omits the leading
FUTURE_USE and Receive Time fields, so they cannot fall back to the first pattern.

Add a third header pattern for the bare audit CSV. It is constrained to a literal
audit type token so it cannot claim lines belonging to any other log type,
regardless of pattern order.

Audit logs also report the result as none for read-only actions, which is not a
valid ECS event.outcome. The vendor value now lands in panw.panos.result and
event.outcome is derived from it, following the same approach already used in
config.yml. Existing documents keep their current event.outcome and only gain the
new field.

Audit was missing from the list of collected log types in the docs, and has been
added.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
  - [] I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
  - [] I have verified that Kibana version constraints are current according to guidelines.
  - [ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• The new grok pattern is appended last and is constrained to the audit type token, so it does not shadow the existing traffic/threat pattern.
• event.outcome for audit logs is now derived rather than copied. Existing values are unchanged; previously invalid values become unknown.
• panw.panos.result is an existing mapped field; its description was widened to cover audit values.

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[vera-review-bot]

Severity: 🔵 Low confidence: medium path: packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/default.yml:54

The new AUDIT grok token is case-insensitive but the audit router only accepts 'AUDIT' or 'audit'; align the two so a mixed-case type token cannot pass the header grok and then skip the audit sub-pipeline.

Details

The AUDIT pattern definition matches any casing (e.g. Audit, AUDIT), and the captured text lands in panw.panos.type unchanged. The routing processor with tag pipeline_e69a9012 only fires for exactly AUDIT or audit (the uppercase normalisation lives inside audit.yml, so it never runs if routing fails). A bare audit line with a mixed-case token therefore clears the header grok, is routed nowhere, and ends up with no CSV fields, no event.outcome and a mixed-case panw.panos.type, silently. The PR description states the token is meant to be a literal audit, so the pattern is wider than the design intends.

Recommendation:

Either restrict the token to the two spellings the router accepts, or route with a case-insensitive comparison. The narrower pattern keeps the change local to the new line:

      pattern_definitions:
        TIMESTAMP: "%{PANW_DATE}|%{TIMESTAMP_ISO8601}"
        PANW_DATE: "%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}"
        FIELD: "[^,]*"
        AUDIT: "(?:AUDIT|audit)"

Alternatively widen the router so all three header shapes behave the same:

  - pipeline:
      tag: pipeline_e69a9012
      if: ctx.panw?.panos?.type instanceof String && ctx.panw.panos.type.equalsIgnoreCase('AUDIT')
      name: '{{ IngestPipeline "audit" }}'

---

🤖 AI-Generated Review | Vera Review Bot - v0.4.1 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Severity: 🔵 Low confidence: high path: packages/panw/data_stream/panos/elasticsearch/ingest_pipeline/audit.yml:32

The description on the fallback event.outcome processor is cut off mid-sentence; complete it so the intent of the unknown mapping is documented.

Details

The description reads "Any result the two processors above did not recognise, such as the none reported" and stops. It is the only place that explains why none becomes unknown, so the truncated sentence leaves the maintainer without the rationale.

Recommendation:

Finish the sentence so it states what happens to unrecognised results.

Suggested change

	description: >-
	Any result the two processors above did not recognise, such as the none reported
	description: >-
	Any result the two processors above did not recognise, such as the none reported
	for read-only actions, is mapped to unknown.

---

🤖 AI-Generated Review | Vera Review Bot - v0.4.1 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[vera-review-bot]

🟢 No issues across the latest commits fecbb42.

⚠️ 2 comments still unresolved from earlier commits — 2 low

• 🔵 The new AUDIT grok token is case-insensitive but the audit router only accepts 'AUDIT' or 'audit' (link)
• 🔵 The description on the fallback event.outcome processor is cut off mid-sentence (link)

Review summary

Issues found across earlier commits f565123 — 3 low

• 🔵 The new AUDIT grok token is case-insensitive but the audit router only accepts 'AUDIT' or 'audit' (link) (Outdated)
• 🔵 The description on the fallback event.outcome processor is cut off mid-sentence (link)

Package-level:

• 🔵 Proposed commit message

  panw: parse audit logs received over the tcp and udp syslog inputs
  
  The tcp and udp inputs run the syslog processor on the agent, which strips the
  RFC 3164/5424 header before the ingest pipeline sees the event. The header grok
  in default.yml only had a pattern for the audit CSV with that header still
  attached, so every audit event collected over syslog failed at the header and
  never reached audit.yml. Only the logfile input could parse audit logs. Audit
  logs are also the only PAN-OS type that omits the leading FUTURE_USE and
  Receive Time fields, so they cannot fall back to the first pattern.
  
  Add a third header pattern for the bare audit CSV, constrained to the audit
  type token so it cannot claim lines belonging to any other log type.
  
  Audit logs report the result as none for read-only actions, which is not a
  valid ECS event.outcome. The vendor value now lands in panw.panos.result and
  event.outcome is derived from it, following the approach already used in
  config.yml. Existing documents keep their current event.outcome and only gain
  the new field.
  
  Add Audit to the list of collected log types in the README.
  

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: fecbb42

History

• 💚 Build #49875 succeeded f565123

[Copilot]

Copilot review overview

🟡 Changes recommended

The failure outcome branch remains untested, and one processor description is incomplete.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Add audit fixture covering failed result mapping · New
• Clarify incomplete processor metadata fallback description · New

What changed in this PR

Fixes PAN-OS audit log parsing for TCP/UDP syslog inputs and normalizes audit outcomes to ECS values.

Changes:

• Parses bare audit CSV after syslog header removal.
• Preserves vendor results while deriving event.outcome.
• Adds audit documentation and test fixtures.

File	Description
packages/​panw/​manifest.yml	Bumps package version.
packages/​panw/​changelog.yml	Records the bug fix.
packages/​panw/​docs/​README.md	Documents audit logs and results.
packages/​panw/​_dev/​build/​docs/​README.md	Updates source documentation.
packages/​panw/​data_stream/​panos/​fields/​fields.yml	Expands result field semantics.
packages/​panw/​data_stream/​panos/​elasticsearch/​ingest_pipeline/​default.yml	Recognizes bare audit CSV headers.
packages/​panw/​data_stream/​panos/​elasticsearch/​ingest_pipeline/​audit.yml	Derives ECS outcomes from vendor results.
packages/​panw/​data_stream/​panos/​_dev/​test/​pipeline/​test-panw-panos-audit-sample.log	Adds bare audit samples.
packages/​panw/​data_stream/​panos/​_dev/​test/​pipeline/​test-panw-panos-audit-sample.log-expected.json	Adds expected parsed events.
packages/​panw/​_dev/​deploy/​docker/​syslog_logs/​panw-panos-other.log	Adds TCP/UDP system fixtures.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ilyannn]

Note: This review has been drafted with 🤖 Cursor/Claude Fable 5.1 under my supervision.

I simulated bare success/none/failed audit lines and the syslog-header form against the PR, main and 5.5.0. On main/5.5.0 every bare line fails at the header grok; on this branch all route to audit.yml with the right event.outcome and panw.panos.result preserved. LGTM with two nits:

1. The description on set_event_outcome_f5a8c413 is cut off mid-sentence ("such as the none reported").
2. No fixture exercises failed → failure. One bare line ending in ,failed covers it (I confirmed the mapping works).

Bot triage: Vera's router finding doesn't hold — AUDIT is (?:AUDIT|audit), exactly the two spellings the router accepts, and a mixed-case Audit fails the header grok before routing. Copilot's two points are the nits above.

Out of scope, for a follow-up: audit events never get @timestamp from the log. default.yml derives it only from high_resolution_timestamp or received_time, which audit records don't have, so they keep the agent/syslog-header time while panw.panos.generated_time holds the real one. Now that audit parses over TCP/UDP it may be worth promoting it. Also minor: the bare-audit pattern is listed after the generic CSV pattern and wins only because column 7 of an audit line isn't a timestamp; listing the more specific pattern first would be cleaner.

How this was tested. Log Shape ran the same lines through the worktree, main and the release (5.5.1-local.b3378b6b / 5.5.0-local.debb3c23 / 5.5.0). Provenance shows panw.panos.result @120-127 feeding both panw.panos.result and event.outcome, and zero provenance entries for @timestamp on any audit line on any version — which is where the follow-up note comes from. The Grok Visualizer, given the three header patterns and definitions verbatim, shows mixed-case Audit stopping at pattern 2's AUDIT token (offset 33) with the other two patterns failing earlier, and shows the generic CSV pattern being attempted first and giving up at column 7 on bare audit lines.

[ilyannn]

Suggest: "Any result the two processors above did not recognise, such as the none PAN-OS reports for read-only GUI navigation, maps to unknown; the vendor value stays in panw.panos.result."

[ilyannn]

Add one bare line for the failure branch, e.g.

003001000000,2024/04/18 18:40:20,audit,2561,cli,Mustang,"commit",failed

and regenerate the expected output (panw.panos.result: failed, event.outcome: failure).

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request