Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FulcrumAdmin to the docker container to be able to gracefully shutdown Fulcrum #253

Closed
antonr-p2p opened this issue Jun 6, 2024 · 6 comments

Comments

@antonr-p2p
Copy link

Please add FulcrumAdmin to the docker container to be able to gracefully shutdown Fulcrum.
Faced issue with corrupted DB, because used kubectl delete pod fulcrum-0 today.
Many times before it survived, but today faced corrupted data dir.
Once you would add FulcrumAdmin it would be possible to configure shutdown hook.

@cculianu
Copy link
Owner

cculianu commented Jun 6, 2024

Fulcrum responds to graceful shutdown signals that are standard such as SIGTERM and SIGINT.

just don’t kill it with SIGKILL.

Also provide a timeout to wait for graceful shutdown.

@cculianu
Copy link
Owner

cculianu commented Jun 6, 2024

I'll add the FulcrumAdmin script to the built image. The fact that it's missing is indeed an error. I think python3 also needs to be installed for it to work but that shouldn't be a problem.

@cculianu
Copy link
Owner

cculianu commented Jun 6, 2024

Ok, added to image. I will do a release very soon (v. 1.11.0 -- good stuff coming!). I will see if I can do a github action now to publish the new image tho.

@antonr-p2p
Copy link
Author

@cculianu thanks a lot for a quick response and actions!
In a meantime I would try to use terminationGracePeriodSeconds option, I hope it would help.

@cculianu
Copy link
Owner

cculianu commented Jun 6, 2024

You're welcome. Yeah I'm no kubernetes/docker expert but it should be possible to gracefully shut down the container rather than nuke it.. like sending all processes a SIGTERM seems customary. Pretty sure if you dig a bit as to how to wait for graceful shutdown, it should work ok.

But yeah I just pushed a new image and FulcrumAdmin now lives alongside Fulcrum now in /usr/bin inside the image.

@antonr-p2p
Copy link
Author

BTW, Kubernetes send SIGTERM and waits 30 seconds by default, then only SIGKILL. This delay can be configured via terminationGracePeriodSeconds . My issue was that I provided too small memory for both DB and container itself, so it crashed and had DB corrupted. Once I gave enough memory - I had no issue so far.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants