-
-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Leader election failure to restart #783
Conversation
…events in LeaderAwareResourceWatcher
Thank you @exextatic for your contribution! It's fine to have unit tests, testing network loss is pretty hard to setup, as such I trust your message that you tested it ;-) The linter is complaining about some white spaces though. Can you run dotnet format and push the commit again? I'd love to merge these changes. |
Thank you for looking at the changes @buehler. |
Fixes #677.
This pull request fixes the following behavioral issues noted when testing a leader-aware operator with transient network issues:
LeaderElectionBackgroundService
, ifelector.RunUntilLeadershipLostAsync()
throws, the exception is not observed in the library and no further attempts to become the leader occur. The library now logs any unexpected exceptions and tries to become the leader again.LeaderAwareResourceWatcher<TEntity>.StoppedLeading
would erroneously pass a cancelled cancellation token toResourceWatcher<TEntity>
. The library now passes theIHostApplicationLifetime.ApplicationStopped
token to theResourceWatcher<TEntity>
- we can assume thatApplicationStopped
is a good indication that the stop should no longer be graceful.Due to the nature of these issues I've only been able to add test coverage in the form of unit tests.
I've tested these changes in practice using clumsy and an AKS cluster.