Wait for cluster fully online in cluster_config_consistent #272
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Fix `cluster_config_consistent` proc of tcl tests by verifying the node status to be online. We expect the start_server to create the desired primaries and replicas before the start of the tests. With the current setup, the replicas may not complete the sync with primaries and can be in loading state. In some cases, the role of replicas can still be master with the delay of propagation of replicate command. The tests can show flaky behavior in such cases. Add a check that verifies the health status 'online' to consider the cluster as consistent. Signed-off-by: Ram Prasad Voleti <ramvolet@amazon.com>
hpatro
approved these changes
Apr 8, 2024
|
@zuiderkwast Here's the fix + tcl test coverage for #265 |
madolson
approved these changes
Apr 9, 2024
madolson
changed the title
PatrickJS
pushed a commit
to PatrickJS/placeholderkv
that referenced
this pull request
Apr 24, 2024
…#272) Wait for cluster to be in a fully consistent and online state in `cluster_config_consistent`. We expect the `start_server` to create the desired primaries and replicas before the start of the tests. With the current setup, the replicas may not complete the sync with primaries and can be in loading state. In some cases, the role of replicas can still be master with the delay of propagation of replicate command. The tests can show flaky behavior in such cases. Add a check that verifies the nodes health status 'online' for the cluster consistency. Leverage the deterministic order of `CLUSTER SLOTS` to consider the cluster as consistent along with the nodes health status. --------- Signed-off-by: Harkrishn Patro <harkrisp@amazon.com> Signed-off-by: Ram Prasad Voleti <ramvolet@amazon.com> Co-authored-by: Harkrishn Patro <harkrisp@amazon.com> Co-authored-by: Ram Prasad Voleti <ramvolet@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Wait for cluster to be in a fully consistent and online state in
cluster_config_consistent. We expect thestart_serverto create the desired primaries and replicas before the start of the tests. With the current setup, the replicas may not complete the sync with primaries and can be in loading state. In some cases, the role of replicas can still be master with the delay of propagation of replicate command. The tests can show flaky behavior in such cases. Add a check that verifies the nodes health status 'online' for the cluster consistency. Leverage the deterministic order ofCLUSTER SLOTSto consider the cluster as consistent along with the nodes health status.