[bitnami/redis] Module crashes when master failover when a write is done #56378
Assignees
Labels
redis
solved
stale
15 days without activity
tech-issues
The user has a technical issue about an application
Comments
wgnathanael
added
the
tech-issues
wgnathanael
changed the title
|
As an attempt to isolate the issue a bit further, I've checked out the bitnami/containers repo, and tried using debian:bullseye instead of minideb:bullseye. Same crash occurs. How can we find out how redis is being compiled? The Dockerfile is just downloading pre-compiled binaries as far as I can tell. |
|
Also, where/how is redis being compiled? I can't find any repo that details how this is being compiled. Redis being opensource, doesn't that need to be provided somewhere so people can see the provenance and any patches applied etc? (Not meant in an accusatory tone, just generally curious) |
|
Hi @wgnathanael, I'm trying to reproduce this, could you tell us if the sentinel:test is bitnami:redis-sentinel or if you made any changes? |
|
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback. |
|
Hello @Mauraza, Not sure why I didn't get a notification when you wrote. The It's using the base redis image. We weren't using the bitnami image because failover there is plagued by issues with sentinel going into tilt mode and the sentinel servers taking forever to be able to come to agreement about the master node being down. bitnami/charts#9689 is the issue we found describing the tilt issue in the bitnami container. The work arounds essentially mean if you disable tls and hostnames there's no tilt issue but then there are connectivity and encryption issues.. We're still evaluating that aspect. |
|
Did you test it with https://github.com/bitnami/containers/tree/main/bitnami/redis-cluster? |
|
For weird implementation reasons we don't always use the cluster mode where it does its own sharding etc. Our module handles a node affinity type situation differently depending on if its cluster mode or not. W do have a version using redis-cluster, but we also support the non redis-cluster mode and so need failover in that situation to work as well. (We haven't gotten to testing redis-cluster as of yet). |
|
Could you add to the environment variable in the bitnami images |
|
Doesn't seem to provide any additional information that I can see |
|
Hi @wgnathanael, The error maybe is related to |
|
I don't 100% know if I'm understanding what you are asking but if slave1 isn't the one that becomes master then it changes, It depends on which node becomes master - but whichever node does crashes. |
|
Hi @wgnathanael, Could you check the events of the pod and the PVC? |
|
Hello Mauraza, These tests are being run in podman after seeing the issue in k8s. There's no PVC and the pod running under podman has crashed and I'm not sure there are events I could check in this testing scenario. At least I'm not aware of any... It should be really easy to replicate this on your end using either podman or docker by compiling that really small custom module. Fire up a multi pod non-k8s deployment and pause the master. It'll crash. No k8s involvement required. Same with the reference tilt issue. |
|
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback. |
|
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Name and Version
bitnami/redis
What architecture are you using?
amd64
What steps will reproduce the bug?
What is the expected behavior?
A replica/slave node/container becomes master, no crashes
What do you see instead?
Additional information
We originally saw this behaviour in our actual custom redis module. We wondered if it was because when a node became master we were causing a few SET calls to be made on the master in the handler function. So we created an extremely simple module to test the theory and sure enough it was crashing. We were deploying using the helm chart and tried the bitnami redis containers with 6.2 and 7.2.4. Both crashed. We filed a bug with redis here. However they were unable to reproduce it. So we used podman to run a cluster locally, this time using
redis:latestinstead of bitnami. The master failover worked perfectly. So we used podman with thedocker.io/bitnami/redis:7.2.4-debian-11-r2container. It crashed identically as it did in k8s. Here's everything needed to reproduce:The module was taken from here.
Compiled with
The
module.cbelow has been very much simplified by essentially doing nothing. We tested the failover with just log messages and it worked perfectly. Once we added the SET calls it crashes every time.The Dockerfile
overrides.conf:
We were using https://github.com/AliyunContainerService/redis-cluster as a basis to setup the redis+sentinel in podman. So had built a sentinel image using their
sentinel/Dockerfileto be used in the followingdocker-compose.yamlThis brought up redis, we could see all redis instances sync with the master. Checking replication info on the master showed 3 slaves. running
podman pause redis-masterthe sentinels noticed it went down, a slave was chosen to become master but you'll see in the test set message it crashes. If we remove the call to the SET function, there's no crash.What is super odd is that this does not occur if we use
redis:7.2.4instead of bitnami for our redis master/slave containers or nodes.The text was updated successfully, but these errors were encountered: