Skip to content

cluster_update_slot failing if new master added #2740

@vadimonus

Description

@vadimonus

Summary: failing to get key from slot migrated to new master which not existed on connection time.

Steps to reproduce:

  • create 5 redis nodes but add only 3 to cluster
redis-server --daemonize yes --port 7000 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7000.conf
redis-server --daemonize yes --port 7001 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7001.conf
redis-server --daemonize yes --port 7002 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7002.conf
redis-server --daemonize yes --port 7003 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7003.conf
redis-server --daemonize yes --port 7004 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7004.conf
redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 --cluster-replicas 0 --cluster-yes
  • start script that connects to cluster and get random key
// note - here we pass only one node address, but phpredis will resolve all other masters, when connected to this node
$redisCluster = new RedisCluster(null, ['127.0.0.1:7000']);
while (true) {
            $masters = $redisCluster->_masters(); // so we will get 3 master here
            foreach ($masters as $master) {
                $randomKey = $redisCluster->randomKey($master);
                $redisCluster->get($randomKey);
            }
           sleep(1);
}
  • add 2 nodes to cluster, and rebalance it, causing to move some slots to new masters.
redis-cli --cluster add-node 172.27.0.199:7003 172.27.0.199:7000 
redis-cli --cluster add-node 172.27.0.199:7004 172.27.0.199:7000
sleep 3
redis-cli --cluster rebalance 172.27.0.199:7000 --cluster-use-empty-masters
redis-cli -h 172.27.0.199 -p 7000 cluster slots

connection will keep info only about 3 masters, that was known on connection time.
when trying to get key from moved slot, it will search new master, but this master is not in array.

Before 6.3.0RC1 (fixed in b0ba827) there was a segfault in this case.
In 6.3.0 it will cause Redis Cluster's master data is incomplete in this case. But cluster is in valid state.
https://github.com/phpredis/phpredis/blob/develop/cluster_library.c#L1419 here should check, if new master is in masters list, and add it' if it is not there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions