Skip to content

[AWS] Remove quotes from VPC flow log message and move dot_expander to top - #4857

Merged
tommyers-elastic merged 2 commits into
elastic:mainfrom
kaiyan-sheng:vpcflow
Dec 19, 2022
Merged

[AWS] Remove quotes from VPC flow log message and move dot_expander to top#4857
tommyers-elastic merged 2 commits into
elastic:mainfrom
kaiyan-sheng:vpcflow

Conversation

@kaiyan-sheng

@kaiyan-sheng kaiyan-sheng commented Dec 16, 2022 •
edited
Loading

Copy link
Copy Markdown

What does this PR do?

Turned out the incoming VPC flow log messages look like {"message":"2 428961148399 eni-0e0bf7be352692297 - - - - - - - 1671029698 1671029728 - NODATA"} instead of {\"message\":\"2 428961148399 eni-0e0bf7be352692297 - - - - - - - 1671029698 1671029728 - NODATA\"}.

Also moving the dot_expander processor to the top of the ingest pipeline so all fields with dot are expanded into an object field before further processing.

For testing the dot_expander processor change, here is what I did using the dev tools:

POST _ingest/pipeline/logs-aws.vpcflow-1.28.3/_simulate
{
  "docs": [
    {
      "_source": {
        "message": "5 64111117617 eni-069xxxxxb7a490 89.160.20.156 10.200.0.0 50041 33004 17 52 1 164000066 1640000297 REJECT OK vpc-09676f97xxxxxb8a7 subnet-02d645xxxxxxxdbc0 i-0axxxxxx1ad77 1 IPv4 89.160.20.156 10.200.0.80 us-east-1 use1-az5 - - AMAZON CLOUDFRONT ingress 1",
        "@timestamp": "2022-03-31T22:33:50Z",
        "event.id": "36768711127022897352489210216814512733652448397667467264",
        "cloud.provider": "aws",
        "cloud.account.id": "428152502467",
        "cloud.region": "us-east-1",
        "aws.firehose.arn": "arn:aws:firehose:us-east-1:428152502467:deliverystream/vpc-flow-log-stream-http-endpoint",
        "aws.firehose.request_id": "a70eae51-0629-4c78-b0dc-799a19642585",
        "aws.kinesis.type": "deliverystream",
        "aws.kinesis.name": "vpc-flow-log-stream-http-endpoint"
      }
    }
    ]
}

Note: you have to use a message field that contains the region.
Without the change in this PR, I got:

    "cloud": {
            "availability_zone": "use1-az5",
            "instance": {
              "id": "i-0axxxxxx1ad77"
            },
            "provider": "aws",
            "region": [
              "us-east-1",
              "us-east-1"
            ],
            "account": {
              "id": "64111117617"
            }
          }

With the change in this PR, I got:

    "cloud": {
            "availability_zone": "use1-az5",
            "instance": {
              "id": "i-0axxxxxx1ad77"
            },
            "provider": "aws",
            "region": "us-east-1",
            "account": {
              "id": "64111117617"
            }
          }

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

@kaiyan-sheng
kaiyan-sheng requested a review from a team as a code owner December 16, 2022 17:19
@kaiyan-sheng kaiyan-sheng self-assigned this Dec 16, 2022
@kaiyan-sheng kaiyan-sheng added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Dec 16, 2022
@elasticmachine

elasticmachine commented Dec 16, 2022 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-12-16T17:20:28.873+0000

  • Duration: 43 min 31 sec

Test stats 🧪

Test Results
Failed 0
Passed 178
Skipped 3
Total 181

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine

elasticmachine commented Dec 16, 2022

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (14/14) 💚
Files 93.333% (14/15) 👎 -4.416
Classes 93.333% (14/15) 👎 -4.416
Methods 85.214% (219/257) 👎 -6.285
Lines 96.022% (5914/6159) 👍 4.363
Conditionals 100.0% (0/0) 💚

zmoog
zmoog approved these changes Dec 19, 2022
View reviewed changes

@zmoog zmoog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tommyers-elastic
tommyers-elastic merged commit 3a0e866 into elastic:main Dec 19, 2022
@elasticmachine

elasticmachine commented Dec 19, 2022

Copy link
Copy Markdown

Package aws - 1.28.3 containing this change is available at https://epr.elastic.co/search?package=aws

orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
kaiyan-sheng
[AWS] Remove quotes from VPC flow log message and move dot_expander t…
b4ce643 
…o top (elastic#4857)

* Remove quotes from VPC flow log message field and move dot_expander processor to top

* add changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zmoog zmoog zmoog approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kaiyan-sheng @elasticmachine @zmoog @tommyers-elastic