[bitnami/redis] Sentinel old replicas remain and replicate from wrong master

[davidtinker]

Which chart:
Bitname/redis 15.3.2

Describe the bug
Redis Sentinel keeps old replicas around after a replica pod is deleted (and respawns). If a new Redis pod uses one of these ip addresses it becomes a slave of the wrong master and loses its own data.

Our Redis's are deployed as follows:

helm upgrade --install butter-redis bitnami/redis --version=15.3.2 \
    --set global.storageClass=openebs-hostpath,auth.enabled=false,auth.sentinel=false,sentinel.enabled=true \
    --set sentinel.downAfterMilliseconds=10000,sentinel.failoverTimeout=5000,sentinel.livenessProbe.initialDelaySeconds=120

To Reproduce
Deploy Redis as above. Kill one of the slave nodes. Check the list of replicas with one of the Sentinels:

k exec -it butter-redis-node-0 -- redis-cli -p 26379 SENTINEL replicas mymaster

This will contain the old dead slave with "flags": "s_down,slave" as well as the new one and others.

It is difficult to get another Redis pod to come up with the same IP but eventually it happens and that Redis becomes a replica of the wrong master and loses its data.

Expected behavior
The dead replicas should disappear from the replicas list. The workaround is to do this manually:

k exec -it butter-redis-node-0 -- redis-cli -p 26379 SENTINEL reset mymaster
k exec -it butter-redis-node-1 -- redis-cli -p 26379 SENTINEL reset mymaster
k exec -it butter-redis-node-2 -- redis-cli -p 26379 SENTINEL reset mymaster

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.6.3", GitCommit:"d506314abfb5d21419df8c7e7e68012379db2354", GitTreeState:"clean", GoVersion:"go1.16.5"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:59:07Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

[juan131]

Duplicate of #5418