Human readable sizes in logs by Spaceman1984 · Pull Request #4207 · apache/cloudstack

Spaceman1984 · 2020-07-10T09:58:58Z

Description

This PR adds outputting human readable byte sizes in the management server logs, agent logs, and usage records. A non-dynamic global variable is added (display.human.readable.sizes) to control switching this feature on and off. This setting is sent to the agent on connection and is only read from the database when the management server is started up. The setting is kept in memory by the use of a static field on the NumbersUtil class and is available throughout the codebase.

Instead of seeing things like:
2020-07-23 15:31:58,593 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 8-1863645820801253428: Processing: { Ans: , MgmtId: 52238089807, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-224-VM","bytesSent":"106496","bytesReceived":"0","result":"true","details":"","wait":"0",}}] }

The KB MB and GB values will be printed out:

2020-07-23 15:31:58,593 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 8-1863645820801253428: Processing: { Ans: , MgmtId: 52238089807, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-224-VM","bytesSent":"(104.00 KB) 106496","bytesReceived":"(0 bytes) 0","result":"true","details":"","wait":"0",}}] }

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Human+Readable+Byte+sizes

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

This has been tested by observing the logs on the management server and on the agent while doing things like the following (Also inspecting the events table and cloud_usage table after the usage aggregation job has run.):

Upload an iso.
Deploy a vm.
Start a vm.
Stop a vm.
Migrate a vm.
Add a storage volume to a zone.
Resize the volume.

Tested on a VMWare, Xenserver and a KVM cluster.

Spaceman1984 · 2020-07-14T08:58:38Z

@blueorangutan package

blueorangutan · 2020-07-14T08:59:40Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-07-14T09:32:18Z

Packaging result: ✖centos7 ✖debian. JID-1575

Spaceman1984 · 2020-07-14T09:51:33Z

@blueorangutan package

blueorangutan · 2020-07-14T09:52:44Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-07-14T10:20:19Z

Packaging result: ✔centos7 ✔debian. JID-1576

Spaceman1984 · 2020-07-23T14:11:46Z

@blueorangutan package

blueorangutan · 2020-07-23T14:13:01Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-07-23T14:40:15Z

Packaging result: ✔centos7 ✔debian. JID-1599

GabrielBrascher · 2020-07-23T18:26:54Z

@Spaceman1984 is this PR ready for review?

Spaceman1984 · 2020-07-24T07:20:54Z

@Spaceman1984 is this PR ready for review?

@GabrielBrascher not yet, just a few more tweaks.

Spaceman1984 · 2020-07-24T10:29:30Z

@blueorangutan test

blueorangutan · 2020-07-24T10:30:02Z

@Spaceman1984 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Spaceman1984 · 2020-07-27T06:37:10Z

@blueorangutan package

blueorangutan · 2020-07-27T06:38:13Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-07-27T07:09:10Z

Packaging result: ✔centos7 ✔debian. JID-1602

yadvr · 2020-07-27T09:02:26Z

@blueorangutan test

blueorangutan · 2020-07-27T09:03:12Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland

lggtm (save a space)

blueorangutan · 2020-07-27T19:49:26Z

Trillian test result (tid-2203)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37059 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4207-t2203-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 79 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_kubernetes_cluster	`Error`	0.06	test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster	`Error`	0.02	test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster	`Error`	0.03	test_kubernetes_clusters.py
ContextSuite context=Test01DeployVM>:setup	`Error`	0.00	test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup	`Error`	0.00	test_vm_life_cycle.py
ContextSuite context=Test03SecuredVmMigration>:setup	`Error`	0.00	test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	328.01	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	310.74	test_vpc_redundant.py
test_hostha_enable_ha_when_host_disabled	`Error`	1.39	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	303.50	test_hostha_kvm.py

Spaceman1984 · 2020-07-28T11:08:06Z

@blueorangutan package

blueorangutan · 2020-07-28T11:09:22Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-08-07T00:31:10Z

Trillian test result (tid-2278)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49892 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4207-t2278-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 77 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1807.80	test_kubernetes_supported_versions.py
test_11_migrate_vm	`Error`	61.06	test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	478.83	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3799.13	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3799.16	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	0.01	test_vpc_redundant.py
ContextSuite context=TestVPCRedundancy>:teardown	`Error`	0.01	test_vpc_redundant.py

blueorangutan · 2020-08-07T01:32:06Z

Trillian test result (tid-2279)
Environment: xenserver-70 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52977 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4207-t2279-xenserver-70.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Smoke tests completed. 82 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1807.44	test_kubernetes_supported_versions.py
test_04_extract_template	`Failure`	1.17	test_templates.py

blueorangutan · 2020-08-07T02:41:15Z

Trillian test result (tid-2284)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47386 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4207-t2284-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 80 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_kubernetes_cluster	`Error`	0.06	test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster	`Error`	0.04	test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_01_reset_vm_on_reboot	`Error`	7.20	test_reset_vm_on_reboot.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	374.11	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3857.87	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	167.28	test_hostha_kvm.py

borisstoyanov · 2020-08-07T07:14:00Z

@blueorangutan package

blueorangutan · 2020-08-07T07:15:02Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-08-07T07:45:32Z

Packaging result: ✔centos7 ✔debian. JID-1670

Spaceman1984 · 2020-08-07T07:53:21Z

@blueorangutan package

blueorangutan · 2020-08-07T07:53:58Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-08-07T08:32:23Z

Packaging result: ✔centos7 ✔debian. JID-1672

Spaceman1984 · 2020-08-07T08:36:17Z

@blueorangutan package

blueorangutan · 2020-08-07T08:36:57Z

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-08-07T09:08:03Z

Packaging result: ✔centos7 ✔debian. JID-1673

Spaceman1984 · 2020-08-07T09:14:07Z

@blueorangutan test

blueorangutan · 2020-08-07T09:14:57Z

@Spaceman1984 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov

LGTM

blueorangutan · 2020-08-07T23:47:16Z

Trillian test result (tid-2294)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50425 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4207-t2294-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 81 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_reset_vm_on_reboot	`Error`	7.25	test_reset_vm_on_reboot.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3811.18	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	370.82	test_hostha_kvm.py

alexandremattioli · 2020-08-11T09:43:07Z

LGTM

yadvr · 2020-08-11T10:24:37Z

@blueorangutan package

blueorangutan · 2020-08-11T10:25:56Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-08-11T11:06:50Z

Packaging result: ✔centos7 ✔debian. JID-1709

yadvr · 2020-08-11T11:24:32Z

@blueorangutan test

blueorangutan · 2020-08-11T11:25:57Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-08-12T00:23:13Z

Trillian test result (tid-2364)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44721 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4207-t2364-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_add_delete_kubernetes_supported_version	`Error`	1809.79	test_kubernetes_supported_versions.py
test_hostha_kvm_host_degraded	`Error`	5.78	test_hostha_kvm.py
test_hostha_kvm_host_fencing	`Error`	214.92	test_hostha_kvm.py

GabrielBrascher assigned Spaceman1984 Jul 23, 2020

GabrielBrascher added the component:logging label Jul 23, 2020

GabrielBrascher added this to the 4.15.0.0 milestone Jul 23, 2020

Spaceman1984 marked this pull request as ready for review July 24, 2020 11:05

Spaceman1984 closed this Jul 24, 2020

Spaceman1984 reopened this Jul 24, 2020

Spaceman1984 marked this pull request as draft July 24, 2020 11:07

Spaceman1984 marked this pull request as ready for review July 24, 2020 11:09

DaanHoogland approved these changes Jul 27, 2020

View reviewed changes

Comment thread core/src/main/java/com/cloud/agent/transport/Request.java Outdated

Added attributes to test

d643114

Added some more conversions

f8ae28d

fixed some typos and added some more conversions

fa34560

borisstoyanov approved these changes Aug 7, 2020

View reviewed changes

yadvr merged commit b586eb2 into apache:master Aug 13, 2020

shwstppr mentioned this pull request Sep 1, 2020

cks: assorted fixes, test refactoring #4269

Merged

5 tasks

DaanHoogland added the type:new-feature label Oct 21, 2020

ACSGitBot added the type:new_feature label Nov 8, 2020

Conversation

Spaceman1984 commented Jul 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Spaceman1984 commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

Spaceman1984 commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

blueorangutan commented Jul 14, 2020

Uh oh!

Spaceman1984 commented Jul 23, 2020

Uh oh!

blueorangutan commented Jul 23, 2020

Uh oh!

blueorangutan commented Jul 23, 2020

Uh oh!

GabrielBrascher commented Jul 23, 2020

Uh oh!

Spaceman1984 commented Jul 24, 2020

Uh oh!

Spaceman1984 commented Jul 24, 2020

Uh oh!

blueorangutan commented Jul 24, 2020

Uh oh!

Spaceman1984 commented Jul 27, 2020

Uh oh!

blueorangutan commented Jul 27, 2020

Uh oh!

blueorangutan commented Jul 27, 2020

Uh oh!

yadvr commented Jul 27, 2020

Uh oh!

blueorangutan commented Jul 27, 2020

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blueorangutan commented Jul 27, 2020

Uh oh!

Spaceman1984 commented Jul 28, 2020

Uh oh!

blueorangutan commented Jul 28, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

borisstoyanov commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

Spaceman1984 commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

Spaceman1984 commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

blueorangutan commented Aug 7, 2020

Uh oh!

Spaceman1984 commented Aug 7, 2020

Uh oh!

Spaceman1984 commented Jul 10, 2020 •

edited

Loading