[aws] Add processor tags to ingest pipelines by andrewkroh · Pull Request #19824 · elastic/integrations

andrewkroh · 2026-06-29T12:10:54Z

Proposed commit message

[aws] Add processor tags to ingest pipelines

Add `tag` fields to every processor in all AWS ingest pipelines using
`elastic-package modify --modifiers pipeline-tag`. Also update
`on_failure` error messages to include `_ingest.pipeline` per the
SVR00009 linter rule. Both are requirements that will be enforced when
the AWS package format_version is bumped to 3.6.0.

Also fix two pre-existing duplicate processors exposed by the tagging:
- cloudfront_logs/default.yml: duplicate x-edge-response-result-type rename
- rds/default.yml: duplicate FreeStorageSpace rename

Fixes cloudfront_logs/pipeline_process_ip.yml which was missing a
pipeline-level on_failure block (SVR00008/SVR00009).

Created with Claude Code using Sonnet 4.6.

Background

This PR pre-lands the ingest pipeline changes required by the
elastic-package linter when format_version is bumped to 3.6.0:

SVR00006: every processor must have a tag field
SVR00008: pipeline-level on_failure must set event.kind: pipeline_error
SVR00009: pipeline-level on_failure error message must include _ingest.pipeline

By landing these changes separately, the upcoming format_version bump PR
will be smaller and easier to review. These changes are also safe to
backport since they have no functional effect at format_version 3.4.0.

The changes were generated with:

elastic-package modify --modifiers pipeline-tag

followed by manual fixes for the on_failure message template updates
and two duplicate processor removals that were surfaced by the tagging.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

infra-vault-gh-plugin-prod · 2026-06-29T12:18:16Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

infra-vault-gh-plugin-prod · 2026-06-29T12:32:17Z

💔 Build Failed

Buildkite Build
Commit: 0c07b52

Failed CI Steps

Check integrations aws

History

vera-review-bot · 2026-06-29T12:34:59Z

👀 I have started reviewing the PR

vera-review-bot · 2026-06-29T12:48:15Z

Vera Review Bot

For the current commit state, I did not find any issues.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Add `tag` fields to every processor in all AWS ingest pipelines using `elastic-package modify --modifiers pipeline-tag`. Also update `on_failure` error messages to include `_ingest.pipeline` per the SVR00009 linter rule. Both are requirements that will be enforced when the AWS package format_version is bumped to 3.6.0. Also fix two pre-existing duplicate processors exposed by the tagging: - cloudfront_logs/default.yml: duplicate x-edge-response-result-type rename - rds/default.yml: duplicate FreeStorageSpace rename Fixes cloudfront_logs/pipeline_process_ip.yml which was missing a pipeline-level on_failure block (SVR00008/SVR00009).

elastic-vault-github-plugin-prod · 2026-06-29T12:57:49Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-06-29T13:35:11Z

👀 I have started reviewing the PR

vera-review-bot · 2026-06-29T13:44:04Z

Vera Review Bot

For the current commit state, I did not find any issues.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

elastic-vault-github-plugin-prod · 2026-06-29T15:09:05Z

🚀 Benchmarks report

Package `aws` 👍(12) 💚(4) 💔(6)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`guardduty`	1893.94	1408.45	-485.49 (-25.63%)	💔
`inspector`	1308.9	1026.69	-282.21 (-21.56%)	💔
`securityhub_findings`	2087.68	1709.4	-378.28 (-18.12%)	💔
`securityhub_findings_full_posture`	2178.65	1795.33	-383.32 (-17.59%)	💔
`securityhub_insights`	1709.4	1443	-266.4 (-15.58%)	💔
`cloudwatch_logs`	200000	142857.14	-57142.86 (-28.57%)	💔

To see the full report comment with /test benchmark fullreport

elasticmachine · 2026-06-29T15:09:09Z

💚 Build Succeeded

Buildkite Build
Commit: 1541dd3

History

💔 Build #45196 failed 0c07b52

andrewkroh · 2026-06-29T16:14:05Z

      field: json.resource.containerDetails.name
      target_field: aws.guardduty.resource.container_details.name
      ignore_missing: true
  - convert:
+      tag: convert_json_resource_containerDetails_securityContext_privileged_to_aws_guardduty_resource_container_details_security_context_privileged_8d0f193a


These auto-generated tags can be lengthy.

ishleenk17 · 2026-06-29T18:46:15Z

      target_field: aws.apigateway.connection_id
      ignore_missing: true
  - rename:
+      tag: rename_aws_apigateway_eventType_to_aws_apigateway_event_type_2a463e2f


@andrewkroh : Do these numbers 2a463e2f have any relevance here ?

Based on the source code for the elastic-package modify --modifiers pipeline-tag command, it looks like its an additional hash of the processor config to help make the ID distinct or unique.

https://github.com/elastic/elastic-package/blob/283983181e1ee42834c1a78ce20788383d55813a/internal/modify/pipelinetag/pipelinetag.go#L139-L146

ShourieG

LGTM

mergify · 2026-07-01T11:50:55Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

Add tag fields to every processor in all aws_securityhub ingest pipelines using elastic-package modify --modifiers pipeline-tag, and update pipeline-level on_failure error messages to include _ingest.pipeline per the SVR00009 linter rule. Both are requirements that will be enforced when the package format_version is bumped to 3.6.x for Identity Federation support. Also fix a pre-existing copy-paste bug surfaced by the tagging: the pipeline_object_evidence pipeline processor in default.yml was tagged pipeline_object_device, duplicating the device processor's tag. Landing these changes separately keeps the upcoming format_version bump PR small, following the precedent of elastic#19824 for the aws package. No functional effect at format_version 3.5.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

andrewkroh requested review from a team as code owners June 29, 2026 12:10

andrewkroh force-pushed the aws/feat/add-processor-tags branch from 550a1be to 0c07b52 Compare June 29, 2026 12:12

andrewkroh added Integration:aws AWS enhancement New feature or request labels Jun 29, 2026

This comment has been minimized.

Sign in to view

This comment was marked as resolved.

Sign in to view

andrewkroh force-pushed the aws/feat/add-processor-tags branch from 0c07b52 to 1541dd3 Compare June 29, 2026 12:52

andrewkroh mentioned this pull request Jun 29, 2026

[aws] Add var_groups for credential type selection with Identity Federation #19828

Merged

2 of 4 tasks

andrewkroh commented Jun 29, 2026

View reviewed changes

ishleenk17 reviewed Jun 29, 2026

View reviewed changes

ishleenk17 approved these changes Jun 29, 2026

View reviewed changes

MichaelKatsoulis approved these changes Jul 1, 2026

View reviewed changes

ShourieG approved these changes Jul 1, 2026

View reviewed changes

kcreddy approved these changes Jul 1, 2026

View reviewed changes

andrewkroh merged commit 910a126 into elastic:main Jul 1, 2026
12 checks passed

This was referenced Jul 31, 2026

[aws_securityhub] Add processor tags to ingest pipelines #20435

Closed

Add federate-aws-integration skill elastic/integration-skills#17

Closed

Conversation

andrewkroh commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Background

Checklist

Related

Uh oh!

github-actions Bot commented Jun 29, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jun 29, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

vera-review-bot Bot commented Jun 29, 2026

Uh oh!

This comment has been minimized.

This comment was marked as resolved.

vera-review-bot Bot commented Jun 29, 2026

Vera Review Bot

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 29, 2026

Uh oh!

vera-review-bot Bot commented Jun 29, 2026

Uh oh!

vera-review-bot Bot commented Jun 29, 2026

Vera Review Bot

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jun 29, 2026

🚀 Benchmarks report

Package aws 👍(12) 💚(4) 💔(6)

Uh oh!

elasticmachine commented Jun 29, 2026

💚 Build Succeeded

History

Uh oh!

andrewkroh Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

andrewkroh Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

ShourieG left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

andrewkroh commented Jun 29, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Jun 29, 2026 •

edited

Loading

Package `aws` 👍(12) 💚(4) 💔(6)