Skip to content

SystemVM optimizations#5831

Merged
nvazquez merged 78 commits into
apache:mainfrom
shapeblue:sysvm-optimizations-v2
Apr 21, 2022
Merged

SystemVM optimizations#5831
nvazquez merged 78 commits into
apache:mainfrom
shapeblue:sysvm-optimizations-v2

Conversation

@Pearl1594
Copy link
Copy Markdown
Contributor

@Pearl1594 Pearl1594 commented Jan 6, 2022 •
edited
Loading

Description

This PR addresses ideas put forth in: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVM+Optimisations

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Screenshots (if appropriate):

How Has This Been Tested?

@yadvr yadvr added this to the 4.16.1.0 milestone Jan 6, 2022
@weizhouapache
Copy link
Copy Markdown
Member

weizhouapache commented Jan 6, 2022

@rohityadavcloud @Pearl1594 @sureshanaparti
I suggest to add this to 4.17.0.0 milestone instead of 4.16.1.0

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Jan 7, 2022

@weizhouapache these are largely operational changes with no DB changes, do you see any issues with getting the optimisations in 4.16.1? cc @sureshanaparti

@weizhouapache
Copy link
Copy Markdown
Member

weizhouapache commented Jan 7, 2022

@weizhouapache these are largely operational changes with no DB changes, do you see any issues with getting the optimisations in 4.16.1? cc @sureshanaparti

@rohityadavcloud
let's discuss internally.

according to https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS

This LTS release cycle has the following goals:

Stability over new features: Deliver releases that only address defects and CVEs. No features or enhancements** would be backported to LTS release branches. This greatly reduces the upgrade risk / operational impact.
Reliable Release Lifetimes: The LTS release cycle will provide users with reliable support and upgrade time frames.
** LTS may accept enhancement if it adds support for an important component upgrade such as support for a new hypervisor version to allow users to upgrade their infra without requiring them to perform a full CloudStack major upgrade.

@Pearl1594 Pearl1594 modified the milestones: 4.16.1.0, 4.17.0.0 Jan 14, 2022
@Pearl1594 Pearl1594 force-pushed the sysvm-optimizations-v2 branch from be131f3 to 6a7d148 Compare January 19, 2022 06:38
@Pearl1594 Pearl1594 changed the base branch from 4.16 to main January 19, 2022 06:39
@Pearl1594
Copy link
Copy Markdown
Contributor Author

Pearl1594 commented Jan 19, 2022

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 19, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 19, 2022

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 2253

@Pearl1594 Pearl1594 force-pushed the sysvm-optimizations-v2 branch from d0245df to c19b7e8 Compare January 20, 2022 04:05
@Pearl1594
Copy link
Copy Markdown
Contributor Author

Pearl1594 commented Jan 20, 2022

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 20, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 20, 2022

Packaging result: ✔️ el7 ✔️ el8 ✖️ debian ✔️ suse15. SL-JID 2258

@Pearl1594
Copy link
Copy Markdown
Contributor Author

Pearl1594 commented Jan 20, 2022

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 20, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 20, 2022

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2259

@Pearl1594 Pearl1594 force-pushed the sysvm-optimizations-v2 branch from a95c337 to 3fdd7cf Compare January 21, 2022 08:54
@Pearl1594
Copy link
Copy Markdown
Contributor Author

Pearl1594 commented Jan 21, 2022

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 21, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 21, 2022

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2267

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 21, 2022

Trillian test result (tid-2931)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46504 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t2931-kvm-centos7.zip
Smoke tests completed. 89 look OK, 3 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestKubernetesCluster>:setup Error 0.00 test_kubernetes_clusters.py
test_01_deployVMInSharedNetwork Failure 3676.31 test_network.py
test_02_verifyRouterIpAfterNetworkRestart Failure 301.36 test_network.py
test_03_destroySharedNetwork Failure 1.06 test_network.py
ContextSuite context=TestSharedNetwork>:teardown Error 2.14 test_network.py
test_01_nic Error 2121.51 test_nic.py

@Pearl1594
Copy link
Copy Markdown
Contributor Author

Pearl1594 commented Jan 24, 2022

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 24, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 24, 2022

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2280

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 24, 2022

Trillian test result (tid-2951)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37976 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t2951-vmware-67u3.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
test_02_cancel_host_maintenace_with_migration_jobs Error 1.42 test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure Error 1.50 test_host_maintenance.py

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Jan 24, 2022

Trillian test result (tid-2949)
Environment: kvm-centos8 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40451 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t2949-kvm-centos8.zip
Smoke tests completed. 83 look OK, 9 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
test_delete_account Error 16.95 test_network.py
test_delete_network_while_vm_on_it Error 2.35 test_network.py
test_delete_network_while_vm_on_it Error 2.35 test_network.py
test_deploy_vm_l2network Error 2.31 test_network.py
test_deploy_vm_l2network Error 2.31 test_network.py
test_l2network_restart Error 3.40 test_network.py
test_l2network_restart Error 3.40 test_network.py
ContextSuite context=TestL2Networks>:teardown Error 4.51 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 6.24 test_network.py
ContextSuite context=TestPublicIP>:setup Error 2.38 test_network.py
test_reboot_router Error 1.78 test_network.py
test_releaseIP Error 1.74 test_network.py
ContextSuite context=TestRouterRules>:setup Error 3.51 test_network.py
test_01_deployVMInSharedNetwork Failure 1.22 test_network.py
test_02_verifyRouterIpAfterNetworkRestart Failure 2.09 test_network.py
test_03_destroySharedNetwork Failure 1.07 test_network.py
ContextSuite context=TestSharedNetwork>:teardown Error 2.17 test_network.py
ContextSuite context=TestDeployVM>:setup Error 0.00 test_vm_life_cycle.py
test_01_secure_vm_migration Error 78.60 test_vm_life_cycle.py
test_02_unsecure_vm_migration Error 217.79 test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration Error 148.24 test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration Error 148.11 test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup Error 1.79 test_vm_life_cycle.py
test_network_acl Error 4.25 test_network_acl.py
test_01_nic Error 49.54 test_nic.py
ContextSuite context=TestCpuCapServiceOfferings>:setup Error 0.00 test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup Error 1.88 test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
test_02_routervm_iptables_policies Error 1.21 test_routers_iptables_default_policy.py
test_01_single_VPC_iptables_policies Error 5.30 test_routers_iptables_default_policy.py
test_01_VPC_nics_after_destroy Error 4.70 test_vpc_router_nics.py
test_02_VPC_default_routes Error 4.67 test_vpc_router_nics.py
test_02_cancel_host_maintenace_with_migration_jobs Error 1.44 test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure Error 1.49 test_host_maintenance.py
test_01_cancel_host_maintenance_ssh_enabled_agent_connected Failure 15.49 test_host_maintenance.py
test_03_cancel_host_maintenance_ssh_disabled_agent_connected Failure 21.51 test_host_maintenance.py
test_04_cancel_host_maintenance_ssh_disabled_agent_disconnected Failure 26.22 test_host_maintenance.py
ContextSuite context=TestHostMaintenanceAgents>:teardown Error 27.43 test_host_maintenance.py

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 20, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 20, 2022

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5831 (SL-JID-1414)

@apache apache deleted a comment from blueorangutan Apr 20, 2022
@acs-robot
Copy link
Copy Markdown

acs-robot commented Apr 20, 2022

PR Coverage Report

CLASS INSTRUCTION MISSED INSTRUCTION COVERED BRANCH MISSED BRANCH COVERED LINE MISSED LINE COVERED
VirtualMachineGuru 78 0 4 0 15 0
NetworkOrchestrationService 0 101 0 0 0 10
VirtualMachineManagerImpl 15659 0 1500 0 3079 0
NetworkOrchestrator 9806 0 1206 0 1928 0
SystemVmTemplateRegistration 1773 0 86 0 376 0
DomainRouterVO 116 50 0 0 41 14
ConsoleProxyDaoImpl 635 0 20 0 144 0
DomainRouterDaoImpl 1600 0 22 0 228 0
LibvirtComputingResource 8877 1958 1086 140 1982 443
LibvirtPatchSystemVmCommandWrapper 228 15 16 0 42 4
LibvirtStartCommandWrapper 75 295 16 22 15 72
VmwareResource 20395 0 2274 0 4397 0
CitrixResourceBase 14658 557 1456 34 3169 122
CitrixCheckSshCommandWrapper 15 51 3 3 3 14
CitrixNetworkElementCommandWrapper 0 14 0 0 0 4
CitrixPatchSystemVmCommandWrapper 224 14 14 0 42 4
CitrixRebootRouterCommandWrapper 28 25 3 1 5 7
CitrixStartCommandWrapper 690 62 81 1 116 16
KubernetesClusterManagerImpl 4724 0 480 0 760 0
KubernetesClusterActionWorker 1540 0 114 0 281 0
KubernetesClusterResourceModifierActionWorker 1843 0 140 0 325 0
ApiDBUtils 2367 0 210 0 590 0
ResponseObjectTypeAdapter 161 8 14 0 37 2
DomainRouterJoinDaoImpl 801 0 80 0 195 0
DomainRouterJoinVO 237 0 0 0 80 0
ConsoleProxyManagerImpl 3717 0 423 0 726 0
LibvirtServerDiscoverer 994 0 116 0 218 0
NetworkServiceImpl 13383 0 1864 0 2551 0
NetworkHelperImpl 2073 0 274 0 441 0
VirtualNetworkApplianceManagerImpl 7585 0 780 0 1521 0
VpcManagerImpl 6883 0 762 0 1314 0
ConfigurationServerImpl 2061 0 176 0 495 0
ManagementServerImpl 12159 0 1076 0 2428 0
UserVmManagerImpl 20798 0 2566 0 3869 0
CAManagerImpl 633 0 80 0 129 0
MockNetworkManagerImpl 485 0 22 0 83 0
SecondaryStorageManagerImpl 3544 149 343 11 629 32

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 20, 2022

Trillian Build Failed (tid-3953)

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 20, 2022

Trillian Build Failed (tid-3954)

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 20, 2022

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3250

@nvazquez
Copy link
Copy Markdown
Contributor

nvazquez commented Apr 20, 2022

@blueorangutan test ubuntu20 kvm-ubuntu20

1 similar comment
@nvazquez
Copy link
Copy Markdown
Contributor

nvazquez commented Apr 20, 2022

@blueorangutan test ubuntu20 kvm-ubuntu20

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 20, 2022

@nvazquez a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3949)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 25714 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3949-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_templates.py

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3950)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26620 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3950-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3951)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26868 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3951-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py

borisstoyanov
borisstoyanov approved these changes Apr 21, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@borisstoyanov borisstoyanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,
Screenshot 2022-04-21 at 9 10 38

@yadvr
Copy link
Copy Markdown
Member

yadvr commented Apr 21, 2022

That UI/router view now looks really good, well done @Pearl1594

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3962)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41408 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3962-kvm-centos7.zip
Smoke tests completed. 96 look OK, 0 have errors
Only failed tests results shown below:

Test Result Time (s) Test File

@acs-robot
Copy link
Copy Markdown

acs-robot commented Apr 21, 2022

Found UI changes, kicking a new UI QA build
@blueorangutan ui

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3961)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44187 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3961-xenserver-71.zip
Smoke tests completed. 95 look OK, 1 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestListIdsParams>:teardown Error 1.12 test_list_ids_parameter.py

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/5831 (SL-JID-1431)

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3959)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 45277 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3959-kvm-ubuntu20.zip
Smoke tests completed. 95 look OK, 1 have errors
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_redundant_vpc_site2site_vpn Error 0.14 test_vpc_vpn.py

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Trillian test result (tid-3960)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45037 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5831-t3960-vmware-67u3.zip
Smoke tests completed. 96 look OK, 0 have errors
Only failed tests results shown below:

Test Result Time (s) Test File

@Pearl1594
Copy link
Copy Markdown
Contributor Author

Pearl1594 commented Apr 21, 2022

@blueorangutan package

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

@Pearl1594 a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@acs-robot
Copy link
Copy Markdown

acs-robot commented Apr 21, 2022

PR Coverage Report

CLASS INSTRUCTION MISSED INSTRUCTION COVERED BRANCH MISSED BRANCH COVERED LINE MISSED LINE COVERED
VirtualMachineGuru 78 0 4 0 15 0
NetworkOrchestrationService 0 101 0 0 0 10
VirtualMachineManagerImpl 15659 0 1500 0 3079 0
NetworkOrchestrator 9806 0 1206 0 1928 0
SystemVmTemplateRegistration 1773 0 86 0 376 0
Upgrade41520to41600 262 17 16 0 62 5
Upgrade41610to41700 159 7 6 0 43 2
DomainRouterVO 116 50 0 0 41 14
ConsoleProxyDaoImpl 635 0 20 0 144 0
DomainRouterDaoImpl 1600 0 22 0 228 0
LibvirtComputingResource 8877 1958 1086 140 1982 443
LibvirtPatchSystemVmCommandWrapper 228 15 16 0 42 4
LibvirtStartCommandWrapper 75 295 16 22 15 72
VmwareResource 20395 0 2274 0 4397 0
CitrixResourceBase 14658 557 1456 34 3169 122
CitrixCheckSshCommandWrapper 15 51 3 3 3 14
CitrixNetworkElementCommandWrapper 0 14 0 0 0 4
CitrixPatchSystemVmCommandWrapper 224 14 14 0 42 4
CitrixRebootRouterCommandWrapper 28 25 3 1 5 7
CitrixStartCommandWrapper 690 62 81 1 116 16
KubernetesClusterManagerImpl 4724 0 480 0 760 0
KubernetesClusterActionWorker 1540 0 114 0 281 0
KubernetesClusterResourceModifierActionWorker 1843 0 140 0 325 0
ApiDBUtils 2367 0 210 0 590 0
ResponseObjectTypeAdapter 161 8 14 0 37 2
DomainRouterJoinDaoImpl 801 0 80 0 195 0
DomainRouterJoinVO 237 0 0 0 80 0
ConsoleProxyManagerImpl 3717 0 423 0 726 0
LibvirtServerDiscoverer 994 0 116 0 218 0
NetworkServiceImpl 13383 0 1864 0 2551 0
NetworkHelperImpl 2073 0 274 0 441 0
VirtualNetworkApplianceManagerImpl 7585 0 780 0 1521 0
VpcManagerImpl 6883 0 762 0 1314 0
ConfigurationServerImpl 2061 0 176 0 495 0
ManagementServerImpl 12159 0 1076 0 2428 0
StorageManagerImpl 8580 0 976 0 1710 0
UserVmManagerImpl 20798 0 2566 0 3869 0
CAManagerImpl 633 0 80 0 129 0
MockNetworkManagerImpl 485 0 22 0 83 0
SecondaryStorageManagerImpl 3544 149 343 11 629 32

@blueorangutan
Copy link
Copy Markdown

blueorangutan commented Apr 21, 2022

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3262

@nvazquez nvazquez merged commit 830f306 into apache:main Apr 21, 2022
@vishesh92 vishesh92 deleted the sysvm-optimizations-v2 branch September 15, 2025 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DaanHoogland DaanHoogland DaanHoogland left review comments

@nvazquez nvazquez nvazquez left review comments

@yadvr yadvr yadvr approved these changes

@NuxRo NuxRo NuxRo approved these changes

@borisstoyanov borisstoyanov borisstoyanov approved these changes

@sureshanaparti sureshanaparti Awaiting requested review from sureshanaparti

@weizhouapache weizhouapache Awaiting requested review from weizhouapache

Assignees

@Pearl1594 Pearl1594

Labels

component:agent component:api component:build component:compute component:networking component:orchestration component:packaging component:systemvm-template component:vpc distro:centos distro:ubuntu type:enhancement type:new-feature

Projects

No open projects
Apache CloudStack 4.17.0
Status: Done

Milestone

4.17.0.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Pearl1594 @weizhouapache @yadvr @blueorangutan @DaanHoogland @nvazquez @borisstoyanov @acs-robot @NuxRo