Backport of keyring: fixes for keyring replication on cluster join into release/1.4.x #15010
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This PR is auto-generated from #14987 to be assessed for backporting due to the inclusion of the label backport/1.4.x.
The below text is copied from the body of the original PR.
Fixes #14981. Note to reviewers: these are definitely bugs but I don't have 100% confidence we've solved the problem because there's a log line missing in the reports from both the issue author and our internal user that I would expect to see. So I'm still trying to repro exactly, but it's worth getting some early eyes on this PR anyways.
Don't unblock early if rate limit burst exceeded. The rate limiter returns an error and unblocks early if its burst limit is exceeded (unless the burst limit is Inf). Ensure we're not unblocking early, otherwise we'll only slow down the cases where we're already pausing to make external RPC requests.
Set
MinQueryIndex
on stale queries. When keyring replication makes a stale query to non-leader peers to find a key the leader doesn't have, we need to make sure the peer we're querying has had a chance to catch up to the most current index for that key. Otherwise it's possible for newly-added servers to query another newly-added server and get a non-error nil response for that key ID.Note that the "not found" case does not return an error, just an empty key. Update the handling of empty responses so that we don't break the loop early if we hit a server that doesn't have the key. (Peers aren't shuffled so we'd expect to hit the same server repeatedly.)
Move the keyring initialize step to wait until we're sure the FSM is current.
If a key is rotated immediately following a leader election, plans that are in-flight may get signed before the new leader has the key. Allow for a short timeout-and-retry to avoid rejecting plans