You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Raft protocol 3 is a requirement for autopilot, so we should default
to it in 1.0. We should avoid putting too much effort into managing
this protocol upgrade, it's unlikely to be repeated. We should put
enough effort into it that upgrading to 1.0 should not be unusually
manual or dangerous.
The Bug
Follower B is stopped
Follower C is stopped
B' is restarted on version 1.0
It's serf tags are gossiped to the leader A
A removes B. The future returns when the configuration change is committed
A adds B'
A is stopped
C' is restarted. C' only knows about servers {A, B} both of which are now gone
The Fix
The bug won't happen if a rolling upgrade waits until the previous
membership change has been committed and then next upgrades a machine
that has an up-to-date log. For simplicity of tooling, we could just
wait until all cluster members have caught up before marking the
cluster safe to upgrade.
There are two implementation options: provide operator tooling to
allow the cluster administrator to safely perform a manual rolling
upgrade, or handle the protocol upgrade internally so that no
additional operator burden is imposed.
Operator Tooling
We could bundle a tool with 1.0 that checks the commit index of every
machine in the cluster to determine if it's safe to perform the next
upgrade. This could just be a query that hits every server in the
cluster to ensure that the index changing the configuration has been
accepted on all servers in the cluster. In an ordinary operator
upgrade where the servers are updated with a rolling upgrade, this
check will be true in less time than it takes for stats dashboards to
catch up, so a typical operator would just experience a fairly simple
but necessary check at every step of the upgrade process.
This check could be generalized to be a new requirement for the
upgrade process, where it may be a good place to ship data
version/consistency preflight checks in the future.
The main risk of operator tooling is that this 1.0 release, which
communicates stability, will require a new manual upgrade step and
could cause a cluster to need to be recovered if upgraded too quickly.
Automatic Upgrade
upgrade all the servers to 1.0
the leader waits until ServersMeetMinimumVersion shows that all servers are running 1.0
the leader calls new UpgradeProtocol RPC on a follower
the leader's raft library commits to raft a RemoveServer configuration change for that follower
that follower receives the RPC and
disconnects
upgrades it's raft instance
rejoins the cluster
the leader adds the protocol 3 instance to the raft cluster
the leader waits until follower has its log up to date
the leader repeats 3-7 for all remaining protocol 2 followers
the leader removes itself from the cluster and upgrades locally
More notes:
the raft protocol 2 instance must leave the cluster first to avoid
the illegal state of two raft servers having the same Addr
the raft protocol 3 instance is added to the raft cluster with raft.AddStaging, but that method is a stub that just adds does an AddVoter (there's a todo comment in the raft libarary). Checking
the instance's log index is the only way to ensure that it's caught
up.
waiting for follower to become integrated with the cluster before
moving on to the next server is necessary to prevent RemoveServer
from decreasing the quorum size. We want to keep the quorum size
constant while upgrading.
The text was updated successfully, but these errors were encountered:
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Background
Raft protocol 3 is a requirement for autopilot, so we should default
to it in 1.0. We should avoid putting too much effort into managing
this protocol upgrade, it's unlikely to be repeated. We should put
enough effort into it that upgrading to 1.0 should not be unusually
manual or dangerous.
The Bug
B
is stoppedC
is stoppedB'
is restarted on version 1.0A
A
removesB
. The future returns when the configuration change is committedA
addsB'
A
is stoppedC'
is restarted.C'
only knows about servers{A, B}
both of which are now goneThe Fix
The bug won't happen if a rolling upgrade waits until the previous
membership change has been committed and then next upgrades a machine
that has an up-to-date log. For simplicity of tooling, we could just
wait until all cluster members have caught up before marking the
cluster safe to upgrade.
There are two implementation options: provide operator tooling to
allow the cluster administrator to safely perform a manual rolling
upgrade, or handle the protocol upgrade internally so that no
additional operator burden is imposed.
Operator Tooling
We could bundle a tool with 1.0 that checks the commit index of every
machine in the cluster to determine if it's safe to perform the next
upgrade. This could just be a query that hits every server in the
cluster to ensure that the index changing the configuration has been
accepted on all servers in the cluster. In an ordinary operator
upgrade where the servers are updated with a rolling upgrade, this
check will be true in less time than it takes for stats dashboards to
catch up, so a typical operator would just experience a fairly simple
but necessary check at every step of the upgrade process.
This check could be generalized to be a new requirement for the
upgrade process, where it may be a good place to ship data
version/consistency preflight checks in the future.
The main risk of operator tooling is that this 1.0 release, which
communicates stability, will require a new manual upgrade step and
could cause a cluster to need to be recovered if upgraded too quickly.
Automatic Upgrade
ServersMeetMinimumVersion
shows that all servers are running 1.0UpgradeProtocol
RPC on a followerRemoveServer
configuration change for that followerMore notes:
the illegal state of two raft servers having the same
Addr
raft.AddStaging
, but that method is a stub that just adds does anAddVoter
(there's a todo comment in the raft libarary). Checkingthe instance's log index is the only way to ensure that it's caught
up.
moving on to the next server is necessary to prevent
RemoveServer
from decreasing the quorum size. We want to keep the quorum size
constant while upgrading.
The text was updated successfully, but these errors were encountered: