Develop 3.0 ensemble #1331
Merged
Develop 3.0 ensemble #1331
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
See - https://docs.riak.com/riak/kv/latest/configuring/strong-consistency/index.html#setting-the-target-n-val-parameter Previously ensemble_util would not set the target_n_val, and it would default to 4 when nval is 5 in many riak_ensemble tests. This setting could mean that a non-quorum minority partition (2 nodes) could contain a majority of peers (3 peers) - and so it would not be possible for all ensembles to prove quorum during such a partition. Where tests are going to partition the cluster they need to have at least nval + 1 nodes - as without this the target_n_val will not be achievable. Also stops over-riding backend. This runs the risk that the tests assume that the backend has the features of the in-memory backend (e.g. non-persisting) - but that isn't documented. The tests appear to be more stable with bitcask backend. All tests still at least past intermittently with the in-memory backend. The test also uses more relaistic defaults. It increases the super-fast anti-entropy tick, and normalises the vnode_management_timer. This appeasr to increase stability in tests (at a cost that more sleeps required to give time for completion).
Currently riak develop-3.0 does not support basho-patches in the code path. Bodge this so we can still add and save a patch (maybe - or perhaps just adding the path this way won't have the intercept survive the reboot?). basho/riak#994
ThomasArts
approved these changes
Oct 29, 2019
Wait for the replace and the leave to complete - test now fails 80% of the time.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These changes are primarily about correcting the test conditions so that the target_n_val is set correctly, and that the number of nodes and n-vals used in the tests will consistently work - without there being a chance that sometimes a minority of nodes might contain a majority of partitions.
There is still an outstanding issue with ensemble_remove_node2 - basho/riak_core#943
There is still a very intermittent failure in
ensemble_sync. Very rarely one of the PUTs will timeout https://github.com/basho/riak_test/blob/develop-2.9/tests/ensemble_sync.erl#L109-L110. It is not the first put. On one occasion it was associated with a vnode unexpectedly trying to handoff via raw_put and the bang returning a badarg.Note also - basho/riak#994