Redundant VRs in ROUTED networks/VPCs both become PRIMARY: nftables INPUT drops VRRP and conntrackd #14445

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR (systemvm scripts, cs/CsAddress.py), ROUTED network mode

CLOUDSTACK VERSION

4.20.3.1 (systemvm template 4.20.2, scripts from cloudstack-common 4.20.3.1)
The same code is on the 4.20 and 4.22 branches and main.

CONFIGURATION

Advanced zone, KVM. Isolated networks and VPCs with network mode ROUTED (Static and Dynamic routing) using a redundant router offering.

OS / ENVIRONMENT

Management server 4.20.3.1, KVM hosts on CentOS 8 (libvirt 8.0), VR on Debian 12.

SUMMARY

When a ROUTED isolated network or VPC uses a redundant router offering, both VRs become PRIMARY: both hold the guest gateway VIP and the public IP.

In ROUTED mode the VR flushes iptables and uses nftables (table ip4_firewall for isolated networks, ip4_acl for VPCs) with an INPUT policy of drop. The INPUT whitelist built in CsAddress.py (fw_router_routing / fw_vpcrouter_routing) accepts DHCP, DNS, userdata and SSH, but not VRRP (224.0.0.18; keepalived uses auth_type AH, so these are IP protocol 51) or conntrackd sync (225.0.0.50). The non-routed iptables path accepts both (-A INPUT -d 224.0.0.18/32 -j ACCEPT, -A INPUT -d 225.0.0.50/32 -j ACCEPT in fw_router / fw_vpcrouter). Each VR drops the other's advertisements and promotes itself.

Related: #11565 asks for redundant VR support in routed networks. Part of this is already in place. The Dynamic and Static Routing feature (#9470) added handling of the RedundantRouter capability on the Gateway service for ROUTED offerings: in ConfigurationManagerImpl and VpcManagerImpl, in the capabilities of VirtualRouterElement, and in the UI offering forms, which send it when the redundant router switch is on. So redundant ROUTED networks and VPCs can be created (via the API), but the VRs come up dual PRIMARY because of the missing rules above.

As a side note, the UI switch is shown only when SourceNat is selected, so it never appears for ROUTED offerings.

I am filing this as a bug against 4.20 because the configuration can be created on 4.20 and leaves the network with duplicate gateway and public IPs. If you would rather treat it as part of #11565 on main, I can retarget the PR; the fix applies cleanly to main as well. With the two missing accept rules added, redundancy worked in our tests (see the linked PR).

STEPS TO REPRODUCE

# 1. A redundant ROUTED offering (API only, see above)
cmk create networkoffering name=Isolated-Routed-Static-HA displaytext=Isolated-Routed-Static-HA \
  guestiptype=Isolated traffictype=GUEST networkmode=ROUTED routingmode=Static \
  supportedservices=Dhcp,Dns,UserData,Firewall specifyvlan=false \
  serviceproviderlist[0].service=Dhcp serviceproviderlist[0].provider=VirtualRouter \
  serviceproviderlist[1].service=Dns serviceproviderlist[1].provider=VirtualRouter \
  serviceproviderlist[2].service=UserData serviceproviderlist[2].provider=VirtualRouter \
  serviceproviderlist[3].service=Firewall serviceproviderlist[3].provider=VirtualRouter \
  servicecapabilitylist[0].service=Gateway servicecapabilitylist[0].capabilitytype=RedundantRouter \
  servicecapabilitylist[0].capabilityvalue=true
cmk update networkoffering id=<id> state=Enabled

# 2. Add an IPv4 subnet for routed networks to the zone, create a network with this
#    offering (cidrsize=24) and deploy a VM in it.

# 3. On both VRs
/opt/cloud/bin/checkrouter.sh
ip -4 -br addr
nft list chain ip ip4_firewall INPUT
tcpdump -ni eth0 host 224.0.0.18

EXPECTED RESULTS

One VR PRIMARY, the other BACKUP (public interface down, no guest gateway VIP).

ACTUAL RESULTS

r-563-VM  Status: PRIMARY  eth0 10.105.0.73/24 10.105.0.1/24   eth2 UP 10.73.3.208/24
r-564-VM  Status: PRIMARY  eth0 10.105.0.169/24 10.105.0.1/24  eth2 UP 10.73.3.208/24

keepalived: (inside_network) Entering BACKUP STATE
keepalived: (inside_network) Entering MASTER STATE        <- 4 s later, on both routers

tcpdump -ni eth0 host 224.0.0.18 (on either VR) shows the advertisements of both routers:
IP 10.105.0.169 > 224.0.0.18: AH(spi=...): VRRPv2, Advertisement, vrid 51, prio 100, authtype ah
IP 10.105.0.73 > 224.0.0.18: AH(spi=...): VRRPv2, Advertisement, vrid 51, prio 100, authtype ah

table ip ip4_firewall { chain INPUT { type filter hook input priority filter; policy drop;
  ... no rule accepting 224.0.0.18 / 225.0.0.50 ...

The same happens for a ROUTED VPC with a redundant VPC offering (ip4_acl table).