Skip to content

optimise prometheus scraping workflow#13650

Open
DaanHoogland wants to merge 2 commits into
4.20from
ghi13586-prometheusDrainage-20
Open

optimise prometheus scraping workflow#13650
DaanHoogland wants to merge 2 commits into
4.20from
ghi13586-prometheusDrainage-20

Conversation

@DaanHoogland

@DaanHoogland DaanHoogland commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

This PR...

Fixes: #13586

  1. Removes alertManager.recalculateCapacity() from PrometheusExporterImpl.updateMetrics(). This is the actual fix — it stops the exporter from forcing a full, thread-pool-churning, all-zone capacity recalculation on every single scrape. The exporter should just read CapacityDao, which AlertManagerImpl's own timer already keeps current. This alone should flatten the scrape_duration curve.
  2. Reuses a single long-lived ExecutorService in AlertManagerImpl instead of creating/tearing one down on every recalculateHostCapacities()/recalculateStorageCapacities() call. Cheap, low-risk, and removes the thread-churn contributor even for the timer-driven path.

this should solve the issue, but claude suggested the following improvements as well:

  1. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
  2. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
  3. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

if anyone wants to, these are nice to haves/good first issue ;)

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@DaanHoogland DaanHoogland added this to the 4.20.4 milestone Jul 20, 2026
@DaanHoogland
DaanHoogland requested review from NuxRo and shwstppr July 20, 2026 10:48
@codecov

codecov Bot commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.26%. Comparing base (e8df87e) to head (7835881).

Files with missing lines Patch % Lines
...rc/main/java/com/cloud/alert/AlertManagerImpl.java 70.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff            @@
##               4.20   #13650   +/-   ##
=========================================
  Coverage     16.26%   16.26%           
- Complexity    13435    13437    +2     
=========================================
  Files          5667     5667           
  Lines        500731   500732    +1     
  Branches      60803    60804    +1     
=========================================
+ Hits          81430    81436    +6     
+ Misses       410197   410189    -8     
- Partials       9104     9107    +3
Flag Coverage Δ
uitests 4.14% <ø> (ø)
unittests 17.11% <70.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@DaanHoogland DaanHoogland linked an issue Jul 20, 2026 that may be closed by this pull request
Scrape duration using prometheus is increasing continuously until restart #13586
Open
@DaanHoogland

DaanHoogland commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

blueorangutan commented Jul 20, 2026

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jul 20, 2026

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18617

@DaanHoogland

DaanHoogland commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@blueorangutan test

@blueorangutan

blueorangutan commented Jul 20, 2026

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

blueorangutan commented Jul 21, 2026

Copy link
Copy Markdown

[SF] Trillian test result (tid-16593)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 70079 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13650-t16593-kvm-ol8.zip
Smoke tests completed. 127 look OK, 14 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_01_events_resource Error 9.52 test_events_resource.py
test_01_events_resource Error 9.52 test_events_resource.py
test_replace_acl_of_network Error 4.65 test_global_acls.py
test_02_deploy_vm_with_extraconfig_kvm Error 1.49 test_deploy_vm_extra_config_data.py
test_03_update_vm_with_extraconfig_kvm Error 1.42 test_deploy_vm_extra_config_data.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Error 225.98 test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Error 226.00 test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces Failure 7.01 test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces Failure 7.88 test_internal_lb.py
ContextSuite context=TestDeployVMFromISO>:setup Error 0.00 test_deploy_vm_iso.py
ContextSuite context=TestDeployVMsInParallel>:setup Error 0.00 test_deploy_vms_in_parallel.py
ContextSuite context=TestDeployVmWithVariedPlanners>:setup Error 0.00 test_deploy_vms_with_varied_deploymentplanners.py
ContextSuite context=TestDomainsServiceOfferings>:setup Error 1520.97 test_domain_service_offerings.py
test_updating_nics_on_two_shared_networks Error 1.70 test_gateway_on_shared_networks.py
ContextSuite context=TestGatewayOnSharedNetwork>:teardown Error 3.90 test_gateway_on_shared_networks.py
ContextSuite context=TestImportAndUnmanageVolumes>:setup Error 0.00 test_import_unmanage_volumes.py
test_01_invalid_upgrade_kubernetes_cluster Failure 0.01 test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster Failure 0.01 test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster Failure 0.07 test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster Failure 0.00 test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle Error 0.01 test_kubernetes_clusters.py
ContextSuite context=TestListVolumes>:setup Error 0.00 test_list_volumes.py
test_01_add_primary_storage_disabled_host Error 39.12 test_primary_storage.py
ContextSuite context=TestStorageTags>:setup Error 38.52 test_primary_storage.py
test_01_primary_storage_scope_change Error 0.14 test_primary_storage_scope.py

Damans227
Damans227 reviewed Jul 21, 2026
View reviewed changes
Comment thread ...egrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java Outdated
public void updateMetrics() {
final List<Item> latestMetricsItems = new ArrayList<Item>();
try {
// NOTE: capacity data is refreshed independently by AlertManagerImpl's own

@Damans227 Damans227 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this inline comment is quite long (7 lines), reads more like a commit message / issue writeup than a code comment. Could trim to a line or two referencing the issue.

@DaanHoogland DaanHoogland Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, makes sense. I’ll move this to the javadoc in the interface/declaration of updateMetrics() (and edit it.)

@sonarqubecloud

sonarqubecloud Bot commented Jul 22, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
46.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shwstppr shwstppr Awaiting requested review from shwstppr
@NuxRo NuxRo Awaiting requested review from NuxRo
1 more reviewer
@Damans227 Damans227 Damans227 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

4.20.4

Development

Successfully merging this pull request may close these issues.

Scrape duration using prometheus is increasing continuously until restart

3 participants

@DaanHoogland @blueorangutan @Damans227