[AWS][ELB] Split ELB data dashboard into 3 different data dashboard by anil-elastic · Pull Request #10444 · elastic/integrations

anil-elastic · 2024-07-10T11:00:03Z

Proposed commit message

Split ELB data dashboard into 3 different data dashboard

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Fields that are not shown on the dashboard

ALB Metrics

Source Field	Description
Consumed LCUs	The number of load balancer capacity units (LCU) used by your load balancer
HTTP Code ELB 500 Count	The number of HTTP 500 error codes that originate from the load balancer
HTTP Code ELB 502 Count	The number of HTTP 502 error codes that originate from the load balancer
HTTP Code ELB 503 Count	The number of HTTP 503 error codes that originate from the load balancer
HTTP Code ELB 504 Count	The number of HTTP 504 error codes that originate from the load balancer
HTTP Fixed Response Count	The number of fixed-response actions that were successful
HTTP Redirect Url Limit Exceeded Count	The number of redirect actions that couldn't be completed because the URL in the response location header is larger than 8K
IPv6 Processed Bytes	The total number of bytes processed by the load balancer over IPv6
IPv6 Request Count	The number of IPv6 requests received by the load balancer
Processed Bytes	The total number of bytes processed by the load balancer over IPv4 and IPv6
Rejected Connection Count	The number of connections that were rejected because the load balancer had reached its maximum number of connections
Rule Evaluations	The number of rules processed by the load balancer given a request rate averaged over an hour

NLB Metrics

Source Field	Description
Active Flow Count TLS	The total number of concurrent TLS flows (or connections) from clients to targets
Consumed LCUs	The number of load balancer capacity units (LCU) used by your load balancer
New FlowCount TLS	The total number of new TLS flows (or connections) established from clients to targets in the time period
Processed Bytes	The total number of bytes processed by the load balancer, including TCP/IP headers
Processed Bytes TLS	The total number of bytes processed by TLS listeners
TCP Client Reset Count	The total number of reset (RST) packets sent from a client to a target
TCP ELB Reset Count	The total number of reset (RST) packets generated by the load balancer
TCP Target Reset Count	The total number of reset (RST) packets sent from a target to a client

ELB Metrics

Source Field	Description
Active Flow Count TLS	The total number of concurrent TLS flows (or connections) from clients to targets
Consumed LCUs	The number of load balancer capacity units (LCU) used by your load balancer
New Flow Count TLS	The total number of new TLS flows (or connections) established from clients to targets in the time period
Processed Bytes	The total number of bytes processed by the load balancer, including TCP/IP headers
Processed Bytes TLS	The total number of bytes processed by TLS listeners
TCP Client Reset Count	The total number of reset (RST) packets sent from a client to a target
TCP ELB Reset Count	The total number of reset (RST) packets generated by the load balancer
TCP Target Reset Count	The total number of reset (RST) packets sent from a target to a client

Screenshots

ELB

ALB

NLB

Updated snapshot

ALB

ELB

NLB

Existing ELB metric dashboard

AWS-Load-Balancer-Overview

anil-elastic · 2024-07-10T13:01:34Z

/test

ghost

After looking into how to properly display the data across dashboards, I believe we can utilize the aws.cloudwatch.namespace field.

We can add a filter in each dashboard (similar to data_stream.dataset filter) to ensure aws.cloudwatch.namespace corresponds to each specific load balancer type:

AWS/NetworkELB namespace for the NLB dashboard
AWS/ApplicationELB namespace for the ALB dashboard
AWS/ELB namespace for the ELB (classic load balancer) dashboard

We also need to use different breakdown fields for lenses based on the load balancer type:

AWS/NetworkELB and AWS/ApplicationELB: breakdown by aws.dimensions.LoadBalancer
AWS/ELB: breakdown by aws.dimensions.LoadBalancerName

@agithomas WDYT about this approach?

elasticmachine · 2024-07-10T14:37:44Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Co-authored-by: Gabriel Pop <94497545+gpop63@users.noreply.github.com>

agithomas · 2024-07-11T02:40:13Z

@agithomas WDYT about this approach?

I think it is a good approach, especially from the performance front.

agithomas · 2024-07-11T02:50:46Z

Anil, below is my quick feedback looking at the attached screenshots

Please fix the duplicate metric panel title such as Healthy Host Count
Kindly update the package screenshots as part of this PR
Please keep related panels closer. For example, Keep Connection count & New connection count together.
There exist panel with duplicate title (Request count). Please see the screenshot attached first.
Try to avoid usage of count in the panel title, instead make use of Y-axis labels.
You can always rename the control drop. Please refer to control dropdown named cloud.region
Please fix the label 3xx_count_sum

agithomas · 2024-07-11T02:59:11Z

@anil-elastic / @gpop63 ,

I think it is important to have an input control to use the values of aws.dimensions.LoadBalancerName such that the user can select from multiple Loadbalancers of the same type (ALB / ELB / NLB).

WDYT?

ali786XI · 2024-07-12T10:05:39Z

@anil-elastic Please have a look at these review comments

For Healthy/Unhealthy host count it's showing decimal values. Can we remove them to show it as an integer?
For Http 3xx count check the X-axis label. It's @timestamp per minute whereas rest all are 5 minutes. Generally the convention is to keep horizontal axis timestamp interval to Auto. Also please check y axis label it is showing /s.
Related to point 1, a general comment for all visualizations. It's not always necessary to show metrics as double values if they are coming as integer. Like for example you can remove decimals for 3xx count, 4xx count visualizations. Please check for similar if the decimals are really required.
(Minor) It would be great to update the screenshot of ELB as it is distorted.

…ions into integrations-10151

anil-elastic · 2024-07-24T09:28:24Z

@agithomas @aliabbas-elastic I just pushed the changes and updated the snapshot also. Please review.

anil-elastic · 2024-07-24T09:29:20Z

/test

…ions into integrations-10151

elasticmachine · 2024-07-24T10:41:52Z

💔 Build Failed

Buildkite Build
Commit: 2258034

Failed CI Steps

Check integrations aws

History

💔 Build #13914 failed dbe1c5d
💔 Build #13913 failed dbe1c5d
💔 Build #13506 failed 85cf2dd
💚 Build #13495 succeeded 0ff78a9
💔 Build #13484 failed 0ff78a9

cc @anil-elastic

mbondyra

Visualizations LGTM 👌🏼

ali786XI

I didn't just check in the Updated for the visualizations screenshots :) Thanks for addressing all the comments!

efd6

Purely due to codeowners.

anil-elastic · 2024-08-01T06:40:16Z

Updated the existing ELB metrics dashboard with links of new LB's dashboard. Snapshot attached.

agithomas · 2024-08-01T12:08:24Z

@anil-elastic , can you revisit the content for `Network Load Balancer (NLB) in the modified (existing) overview dashboard ?

agithomas · 2024-08-01T12:11:06Z

Please update the screenshot of the existing ELB overview dashboard to reflect the new layout.

ghost · 2024-08-01T13:48:10Z

Overall, it looks good to me. A few nitpicks:

If we use a static color in the overview dashboard, the background colors of the lenses won't change dynamically based on the Kibana color theme.
For some lenses, we have Y-axis labels, while for others we don't. The current labels seem ok.
Since the old dashboard has become an overview dashboard with navigation, can we rename it to something like [Metrics AWS] Load Balancers Overview? There are many dashboards where we do this, and it would avoid having to deprecate it?

@agithomas

anil-elastic · 2024-08-05T18:24:35Z

@agithomas Updated as per the review comments. Please review once.

elasticmachine · 2024-08-07T10:56:02Z

💚 Build Succeeded

Buildkite Build
Commit: fe1cf54

History

💔 Build #14405 failed 03c51b8
💚 Build #14342 succeeded 8fed11d
💚 Build #14311 succeeded 70ec669
💔 Build #14307 failed b81070d
💔 Build #14305 failed dc9e50f
💚 Build #14296 succeeded 6a00b44

cc @anil-elastic

elastic-sonarqube · 2024-08-07T10:56:04Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

agithomas

LGTM!

elasticmachine · 2024-08-13T09:51:33Z

Package aws - 2.23.0 containing this change is available at https://epr.elastic.co/search?package=aws

…lastic#10444) * split ELB data dashboard into 3 different dashboard * Update packages/aws/changelog.yml Co-authored-by: Gabriel Pop <94497545+gpop63@users.noreply.github.com> * Review comment fixed * ALB snapshot updated * fomatting fixed * Updated existing ELB dashboard * Updated with deprecrated and New dashboard link * dashboard size and text updated * version updated * file name corrected * dashboard word removed from link and NLB description updated. * snapshot updated * removed deprecated elb dashboard screenshot --------- Co-authored-by: Gabriel Pop <94497545+gpop63@users.noreply.github.com>

split ELB data dashboard into 3 different dashboard

0ff78a9

anil-elastic requested review from a team as code owners July 10, 2024 11:00

anil-elastic requested a review from agithomas July 10, 2024 11:04

agithomas requested a review from a user July 10, 2024 11:34

ghost reviewed Jul 10, 2024

View reviewed changes

Comment thread packages/aws/changelog.yml Resolved

ghost assigned anil-elastic Jul 10, 2024

Update packages/aws/changelog.yml

85cf2dd

Co-authored-by: Gabriel Pop <94497545+gpop63@users.noreply.github.com>

agithomas changed the title ~~Split ELB data dashboard into 3 different data dashboard~~ [AWS][ELB] Split ELB data dashboard into 3 different data dashboard Jul 11, 2024

agithomas requested a review from ali786XI July 11, 2024 07:48

This was referenced Jul 12, 2024

[AWS] AWS ELB metrics integration OOTB Dashboard does not reflect ALBs data for Inbound and Outbound Traffic Visualisations #10149

Closed

[Meta] AWS ELB Integration Improvements #10151

Closed

andrewkroh added the Integration:aws AWS label Jul 19, 2024

anil-elastic added 2 commits July 24, 2024 14:53

Review comment fixed

f8cdd83

Merge branch 'integrations-10151' of github.com:anil-elastic/integrat…

dbe1c5d

…ions into integrations-10151

anil-elastic added 3 commits July 24, 2024 15:55

Merge branch 'main' into integrations-10151

ba0ed15

ALB snapshot updated

824b90e

Merge branch 'integrations-10151' of github.com:anil-elastic/integrat…

2258034

…ions into integrations-10151

fomatting fixed

95dfb09

mbondyra reviewed Jul 25, 2024

View reviewed changes

Comment thread packages/aws/kibana/dashboard/aws-34021716-10b1-4338-807c-1a9bcb3da222.json Resolved

mbondyra approved these changes Jul 25, 2024

View reviewed changes

ali786XI approved these changes Jul 25, 2024

View reviewed changes

efd6 approved these changes Jul 29, 2024

View reviewed changes

Updated existing ELB dashboard

ac3e2d8

anil-elastic added 4 commits August 5, 2024 14:51

Updated with deprecrated and New dashboard link

6a00b44

dashboard size and text updated

dc9e50f

Merge branch 'main' into integrations-10151

b81070d

version updated

70ec669

agithomas requested a review from tommyers-elastic August 6, 2024 04:29

agithomas reviewed Aug 6, 2024

View reviewed changes

Comment thread packages/aws/manifest.yml Resolved

anil-elastic added 5 commits August 6, 2024 11:30

file name corrected

5340868

dashboard word removed from link and NLB description updated.

24cba15

snapshot updated

8fed11d

removed deprecated elb dashboard screenshot

03c51b8

Merge branch 'main' into integrations-10151

fe1cf54

agithomas approved these changes Aug 8, 2024

View reviewed changes

anil-elastic merged commit e6362aa into elastic:main Aug 13, 2024

agithomas mentioned this pull request Aug 13, 2024

[aws] AWS ELB metrics integration should contain OOTB dashboards for each of the 3 load balancer types #5953

Closed

anil-elastic deleted the integrations-10151 branch August 22, 2024 05:23

Conversation

anil-elastic commented Jul 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Fields that are not shown on the dashboard

ALB Metrics

NLB Metrics

ELB Metrics

Screenshots

ELB

ALB

NLB

Updated snapshot

ALB

ELB

NLB

Existing ELB metric dashboard

AWS-Load-Balancer-Overview

Uh oh!

anil-elastic commented Jul 10, 2024

Uh oh!

ghost left a comment • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Jul 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

agithomas commented Jul 11, 2024

Uh oh!

agithomas commented Jul 11, 2024

Uh oh!

agithomas commented Jul 11, 2024

Uh oh!

ali786XI commented Jul 12, 2024

Uh oh!

anil-elastic commented Jul 24, 2024

Uh oh!

anil-elastic commented Jul 24, 2024

Uh oh!

elasticmachine commented Jul 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Uh oh!

mbondyra left a comment

Choose a reason for hiding this comment

Uh oh!

ali786XI left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

anil-elastic commented Aug 1, 2024

Uh oh!

agithomas commented Aug 1, 2024

Uh oh!

agithomas commented Aug 1, 2024

Uh oh!

ghost commented Aug 1, 2024

Uh oh!

anil-elastic commented Aug 5, 2024

Uh oh!

Uh oh!

elasticmachine commented Aug 7, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube Bot commented Aug 7, 2024

Quality Gate passed

anil-elastic commented Jul 10, 2024 •

edited

Loading

ghost left a comment •

edited by ghost

Loading

elasticmachine commented Jul 10, 2024 •

edited

Loading

elasticmachine commented Jul 24, 2024 •

edited

Loading

ali786XI left a comment •

edited

Loading