Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PIP-118: Do not restart brokers when ZooKeeper session expires #13304

Closed
merlimat opened this issue Dec 14, 2021 · 0 comments · Fixed by #13341
Closed

PIP-118: Do not restart brokers when ZooKeeper session expires #13304

merlimat opened this issue Dec 14, 2021 · 0 comments · Fixed by #13341
Labels

Comments

@merlimat
Copy link
Contributor

merlimat commented Dec 14, 2021

Motivation

After all the work done for PIP-45 that was already included in 2.8 and 2.9
releases, it enabled the concept of re-acquirable resource locks and leader
election.

Another important change was to avoid doing any deferrable metadata operation
when we know that we are not currently connected to the metadata service.

Finally, that enabled to stabilize in 2.9 the configuration setting that allows
brokers to continue operating in a safe mode when the session with ZooKeeper
expires.

The way it works is that, when we lose a ZooKeeper session, the data plane will
continue to work undisturbed, relying on the BookKeeper fencing to avoid any
inconsistencies.

New topics are not able to get started, but existing topics will see no
impact.

The original intention for shutting down the brokers was to ensure that we
would automatically go back to a consistent state, with respect to which
resources are "owned" in ZooKeeper by a given broker.

With the re-acquirable resource locks, that problem was solved and thoroughly
tested to be robust.

Proposed changes

In 2.10 release, for the setting:

# There are two policies to apply when broker metadata session expires: session expired happens, "shutdown" or "reconnect".
# With "shutdown", the broker will be restarted.
# With "reconnect", the broker will keep serving the topics, while attempting to recreate a new session.
zookeeperSessionExpiredPolicy=shutdown

Deprecate the old setting and rename it to metadataSessionExpiredPolicy, with default value set to reconnect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants