[AWS] Expose default region config by legoguy1000 · Pull Request #4158 · elastic/integrations

legoguy1000 · 2022-09-07T20:30:55Z

What does this PR do?

Expose default region config to the UI. This allows users that utilize non-regular AWS environments to set the initial region which sets the correct endpoints to use with such as IAM which will be called prior to service specific endpoints. For the SaaS solutions such as Cloudflare, Crowdstrike, Cisco Umbrella, this change allows for integration to AWS Gov-CLoud or AWS China. The changes to AWS and AWS Logs integrations allow for AWS Gov-Cloud, AWS China, US Gov AWS isolated regions.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates Refactor endpoint resolver for AWS services beats#32921

Screenshots

elasticmachine · 2022-09-07T20:34:24Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-19T19:12:06.195+0000
Duration: 43 min 0 sec

Test stats 🧪

Test	Results
Failed	0
Passed	229
Skipped	2
Total	231

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

francescayeye · 2022-09-08T02:34:35Z

that's amazing @legoguy1000 , thank you very much!

I was just discussing about making this change. :)

If you don't mind I will go further and remove endpoint but for where non_aws_bucket_name is defined (it should be only in packages/aws_logs/data_stream/generic/agent/stream/aws-s3.yml.hbs)

francescayeye · 2022-09-08T02:36:37Z

this should be done also for:

thanks!

francescayeye · 2022-09-08T02:36:44Z

/test

francescayeye

missing packages/aws/data_stream/ecs_metrics/agent/stream/stream.yml.hbs

francescayeye · 2022-09-08T03:07:23Z

kibana.version must be bumped to at least ^8.0.0

default_region is available since then https://github.com/elastic/beats/blob/v8.0.0/x-pack/libbeat/common/aws/credentials.go#L43

elasticmachine · 2022-09-08T03:12:09Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`23/23`)	💚
Files	96.774% (`30/31`)	👎 -0.71
Classes	96.774% (`30/31`)	👎 -0.71
Methods	90.099% (`364/404`)	👎 -0.143
Lines	95.164% (`10568/11105`)	👍 3.851
Conditionals	100.0% (`0/0`)	💚

legoguy1000 · 2022-09-08T11:40:31Z

@aspacca Do u think I should just remove endpoint entirely from the manifest for the AWS integration? As i went through removing endpoint from the agent configs, realized that we shouldn't need it at all in the entire AWS package.

francescayeye · 2022-09-08T11:50:33Z

Do u think I should just remove endpoint entirely from the manifest for the AWS integration? As i went through removing endpoint from the agent configs, realized that we shouldn't need it at all in the entire AWS package.

yes, let's go for it. also in the other integrations where is not needed anymore

legoguy1000 · 2022-09-08T11:55:21Z

ok, I left it in the aws_logs since it could be used to pull logs in via non-AWS S3 compatible services. I also left it in crowdstrike and Cisco Umbrella as I suspect AWS is the only S3 compatible service they support but figured that might not always be the case so having it be in the config doesn't hurt, especially since it defaults to blank.

francescayeye · 2022-09-08T11:57:16Z

I also left it in crowdstrike and Cisco Umbrella as I suspect AWS is the only S3 compatible service they support but figured that might not always be the case so having it be in the config doesn't hurt, especially since it defaults to blank.

for me it's fine.

we can ping someone from @elastic/security-external-integrations to validate on this anyway :)

cc @P1llus , any hint ? thanks

kaiyan-sheng

/test

legoguy1000 · 2022-09-16T19:43:24Z

Seems good to merge??

francescayeye · 2022-09-19T05:50:02Z

@legoguy1000 we need a final approval from @elastic/security-external-integrations

efd6

Tentatively (and modulo the change in version number); it would be nice to have a more clear PR description with 'what' and 'why' that could be the basis of the commit message.

bump version

legoguy1000 · 2022-09-19T18:33:56Z

@aspacca @efd6 updated.

elasticmachine · 2022-09-19T19:12:37Z

🚀 Benchmarks report

Package `aws` 👍(7) 💚(4) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`cloudfront_logs`	2164.5	1845.02	-319.48 (-17.32%)	💔
`cloudwatch_logs`	500000	333333.33	-166666.67 (-50%)	💔

Package `cisco_umbrella` 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`log`	9009.01	7246.38	-1762.63 (-24.32%)	💔

Package `cloudflare_logpush` 👍(6) 💚(1) 💔(0)

Package `crowdstrike` 👍(1) 💚(1) 💔(0)

To see the full report comment with /test benchmark fullreport

* [AWS] Expose default region config * update changelog * remove endpoint option where not needed * Update other integrations that use S3 for logs * remove endpoint completely * Update changelog.yml bump version * bump minor versions * update changelogs & add cloudflare_logpush

legoguy1000 requested a review from a team as a code owner September 7, 2022 20:30

francescayeye reviewed Sep 8, 2022

View reviewed changes

Comment thread packages/aws/manifest.yml Resolved

Comment thread packages/aws_logs/manifest.yml Resolved

francescayeye mentioned this pull request Sep 8, 2022

Refactor endpoint resolver for AWS services elastic/beats#32921

Merged

legoguy1000 requested a review from a team as a code owner September 8, 2022 11:30

legoguy1000 force-pushed the aws-default-region branch from b265e36 to ee4fe87 Compare September 8, 2022 11:41

kaiyan-sheng approved these changes Sep 13, 2022

View reviewed changes

francescayeye approved these changes Sep 14, 2022

View reviewed changes

This was referenced Sep 14, 2022

[filebeat] Credentials error with S3+SQS input in 8.4.0 elastic/beats#32888

Closed

Set default endpoint for AWS to empty string #4207

Merged

francescayeye suggested changes Sep 19, 2022

View reviewed changes

Comment thread packages/aws_logs/manifest.yml Resolved

Comment thread packages/crowdstrike/manifest.yml Resolved

efd6 approved these changes Sep 19, 2022

View reviewed changes

legoguy1000 added 6 commits September 19, 2022 18:31

[AWS] Expose default region config

cb37f72

update changelog

413fa5e

remove endpoint option where not needed

ac41aeb

Update other integrations that use S3 for logs

3ab4b66

remove endpoint completely

6a9d08c

Update changelog.yml

1424ff9

bump version

bump minor versions

57935c6

legoguy1000 force-pushed the aws-default-region branch from 6c8f95f to 57935c6 Compare September 19, 2022 18:33

update changelogs & add cloudflare_logpush

81ef01f

francescayeye approved these changes Sep 20, 2022

View reviewed changes

francescayeye merged commit dd12a71 into elastic:main Sep 20, 2022

legoguy1000 deleted the aws-default-region branch September 20, 2022 05:43

andrewkroh mentioned this pull request Sep 29, 2022

[production] Promote many packages from snapshot elastic/package-storage#5760

Merged

github-actions Bot mentioned this pull request Apr 8, 2026

Multiple integrations fail on Agent 8.19.12+ when using non-AWS S3-compatible buckets — missing region field #18279

Closed

Conversation

legoguy1000 commented Sep 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Sep 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

francescayeye commented Sep 8, 2022

Uh oh!

francescayeye commented Sep 8, 2022

Uh oh!

francescayeye commented Sep 8, 2022

Uh oh!

francescayeye left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

francescayeye commented Sep 8, 2022

Uh oh!

elasticmachine commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

legoguy1000 commented Sep 8, 2022

Uh oh!

francescayeye commented Sep 8, 2022

Uh oh!

legoguy1000 commented Sep 8, 2022

Uh oh!

francescayeye commented Sep 8, 2022

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

legoguy1000 commented Sep 16, 2022

Uh oh!

francescayeye commented Sep 19, 2022

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

legoguy1000 commented Sep 19, 2022

Uh oh!

elasticmachine commented Sep 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package aws 👍(7) 💚(4) 💔(2)

Package cisco_umbrella 👍(0) 💚(0) 💔(1)

Package cloudflare_logpush 👍(6) 💚(1) 💔(0)

Package crowdstrike 👍(1) 💚(1) 💔(0)

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

legoguy1000 commented Sep 7, 2022 •

edited

Loading

elasticmachine commented Sep 7, 2022 •

edited

Loading

elasticmachine commented Sep 8, 2022 •

edited

Loading

elasticmachine commented Sep 19, 2022 •

edited

Loading

Package `aws` 👍(7) 💚(4) 💔(2)

Package `cisco_umbrella` 👍(0) 💚(0) 💔(1)

Package `cloudflare_logpush` 👍(6) 💚(1) 💔(0)

Package `crowdstrike` 👍(1) 💚(1) 💔(0)