Description

Every time there is a major CloudStack version that requires a new systemvmtemplate, admins need to restart VRs which can take few (sometimes several) minutes. The aim is to make it easier to perform a rolling restart (or blue-green deployment) of the network with acceptable downtime. In this strategy, a new VR is provisioned before the old VR is destroyed/cleaned for non-redundant isolated VRs. For redundant networks, a rolling replacement of routers is performed.

The network downtime is usually the short-time (less than 10s) after the new VR starts (nics are up) and old VR is destroyed. The downtime may be affected by several factors as the new VR has a different mac address, stale ARP cache may take upto 30seconds (default gc_timeout on Linux) to renew in guests making egress traffic downtime slightly higher than ingress traffic (ingress, i.e. public network requesting access to VR's public ip). As an illustration, ingress traffic such as a http request or ssh request to the VR's public IP that port-forwards to guest VM may face lower downtime than egress traffic (say a ping or curl from guest VM to public Internet). Among several environment factors, the following interval/timeouts affect how the old VR's ip-mac address is cached:

/proc/sys/net/ipv4/neigh/default/gc_interval
/proc/sys/net/ipv4/neigh/default/gc_stale_time
/proc/sys/net/ipv4/route/gc_interval
/proc/sys/net/ipv4/route/gc_timeout

To make things faster, a gratituous ARP notification is sent by the new VR (again) after the old VR is successfully destroyed.

Project Goals:

• Reduce systemvmtemplate size, see CLOUDSTACK-10341: Reduce systemvmtemplate size, install nftables #2506. Smaller the image size, faster it will be to copy template to primary storage to provision new VR.
• Rolling reboot/deployment of VRs for isolated network with non-rVRs
• Rolling reboot/deployment of VRs for isolated network with rVRs
• Rolling reboot/deployment of VRs for VPCs with non-rVRs and rVRs

Future:

• Speed up iptables rules application
• Move away from iptables, move to nftables
• Move away from Python2 codebase to Python3 or something else like Go? TBD

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

Non-Redundant Isolated Network VRs

Rolling restart in action, a new VR is deployed lifing the VR as a redundant VR:

The old VR is stopped and destroyed:

The new VR's redundancy is removed and is reprogrammed:

Meanwhile, the following screenshot shows ping drops from a guest VM, in a non-redundant isolated network: (usually 2-10 ping drops observed)

Redundant Isolated Network VR

Rolling restart starts by stopping old backup VR:

New backup VR deployed:

Old master stopped, new backup becomes new master:

Old master destroyed and another backup VR deployed:

Redundant VRs after rollling reboot:

In the meanwhile, usually 0-4 ping drops observed in guest VM:

How Has This Been Tested?

Tested with following, benchmarks show rounded off avg/max downtime seconds for restarting non-redundant isolated VRs when cleanup=true:

** The downtime seconds may be higher in real-world environment under heavy load. Redundant routers show a downtime of 0-4 seconds.
*** All used a CentOS7 based management server, tested in environments deployed by Trillian

Ingress traffic was tested using ssh and http services hosted on the guest VM. Example, sshping like utility I wrote for calculating ingress traffic downtime: (number of fail count = downtime seconds)

sshping() {                                                                                                                                              
  ip=$1                                                                            
  total_count=0                                                                    
  fail_count=0                                                                     
  while true; do                                                                   
    start_time=$(date +%s)                                                         
    timeout 1.5 sshpass -p 'password' ssh -o StrictHostKeyChecking=no root@$1 "uptime" > /dev/null 2>&1
    if [[ $? -gt 0 ]]; then                                                        
        fail_count=$((fail_count+1))                                               
    fi                                                                             
    end_time=$(date +%s)                                                           
    diff=$(($end_time - $start_time))                                              
    if [[ $diff -eq 0 ]]; then                                                     
        sleep 1                                                                    
    fi                                                                             
                                                                                   
    total_count=$((total_count+1))                                                 
    echo "Fail/Total = $fail_count/$total_count"                                   
  done                                                                             
}  

Checklist:

• I have read the CONTRIBUTING document.
• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing tests passed.

@blueorangutan package