-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proper HA setup #34
Comments
Hi, I know you're not under any obligation to develop this and appreciate all you've done already but are there any works in progress towards this or #33? At the moment this prevents being able to upgrade nodes without hard downtime. I really wish I could contribute but low level custom Kubernetes add-ons is still above my pay grade. It may not be forever but it is in the foreseeable future. |
Hi, |
Hi, Definitely not rushing you, but: I'd love to be able to continue using this tool but not being able to upgrade nodes on our cluster without downtime is becoming an issue around not being able to apply security patches to the underlying EC2 boxes by creating new node groups without introducing downtime on our apps. With that said, are you open to receiving USD donations to help expedite this ticket and any related tickets to reach an outcome of being able to do zero down time node upgrades while using pod-graceful-drain on your pods? If so feel free to email me at Also if you need any assistance in testing anything (regardless of your decision) let me know. Happy to do anything I can do. |
Hi, Thank you for your consideration! But I don't think I'm able to receive your donation. |
Here's some news: |
It's all good. It is nice to see you're using this at work too. |
Hi, Where do you stand on this since it's been a bit? I'm just probing for information, not trying to push you into anything! On the bright side, your project is so wildly useful that we're happily using it and would love to continue using it. |
I've been busy launching a new service on my team, and I recently got some time to work on this project. |
@nickjj potential workaround: Using selectors/taints, you can run this on separate nodes from where the workloads requiring the graceful shutdown run. For example, on AWS we have all our controllers running in one EKS ASG, and then our services running in another. For upgrades or scaling events, we can do these two groups independently. |
@nkkowa We have a small cluster with (2) nodes with no auto-scaling and we run (2) replicas of most of our services. This tool itself ends up always running on 1 of the nodes. We haven't yet configured anything at the K8s level to guarantee 1 replica runs on each node because K8s has handled this pretty well out of the box. Were you suggesting something that could still work with that in mind? I also update the cluster usually with Terraform (EKS module) which handles the whole taint+drain+etc. process. |
@nickjj Ah, without an architecture that has dedicated nodes for controllers/operators I don't think my setup would work for you 😞 |
It's taking more time to write tests and polish up non-critical features. |
Sounds great. This is a critical tool in general, don't feel rushed! I'd personally be happy to wait a bit longer to have extensive test coverage and features that you deem are worthy. What are the non-critical features btw? |
Thank you. I was trying some leader election based minor optimizations. |
Hi, Any updates on this? Not trying to be pushy in the slightest. EKS 1.23 will be end of life in a few months and I was wondering if I should upgrade the cluster with a minute of app downtime or potentially wait until this patch is available to do it without downtime. |
Hi, I'm really sorry that I haven't replied to you too long. I think you might be able to mitigate the downtime risk with carefully orchestrating the process until that.
|
It's all good, hope things are going well. I ended up taking the downtime approach a few months ago, it was less than a minute or so of downtime but it's very uncontrollable. Thanks for the checklist, hopefully in a few months it can go away! |
Hi, Any updates on this? Sorry for following up again. |
Hi, sorry to reply too late. |
Thanks, when PR #40 is released, how do you see the upgrade process being for existing installations? Would it be bumping the Helm version for pod-graceful-drain and deploying it or would it be more involved? Would all deployments need to be re-deployed too, etc.? |
Just bumping up the version and redeploying the pod-graceful-drain only, or re-installing pod-graceful-drain only would be okay. |
Hi. I've released
|
Hi, That's great news, thank you so much for working on this project. It has been a staple for us since we started using Kubernetes. For rolling this out to production, do you suggest giving it a bit more time? For context, we don't have a dev or staging environment that runs our normal workload. We deploy straight to production. I can spin up a few dummy services in a test environment but as you know, that's never a great indicator of how something may work with your "real" workload. Did the rough edges cause any side effects that could impact a deployment or service's availability? |
I don't think there will be a drastic change until the release. I meant rough edges are something that bothers me with my confidence, such as logging, testing, redundant API requests, and some mild generic 'it works on my cluster' and 'what if' kind of concerns. Our dev cluster is for a team with around ~15 deploying developers, and ~100 non-dev members. I'll let you know in a week or two, or if something happens. |
Right, the more you know how something works, the more you're surprised how any of it works. Thanks for the details, I'll check it in a week or so and start using it. |
During node draining,
pod-graceful-drain
itself may also be evicted.I deliberately chosen to ignore webhook admission failures since otherwise deployments would fail to progress.
Because of this, pods that are evicted/deleted at that time can suffer downtime even with multiple replicas.
To fix this,
pod-graceful-drain
needs a HA setup.However, simply giving it
replicas: 2
on the deployment won't work.It would not behave correctly when there are multiple replicas of it.
The text was updated successfully, but these errors were encountered: