Remove 2.x backward compatibility of mappings. #21670
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpountz
added
:Search Foundations/Mapping
jpountz
force-pushed
the
remove_mapping_bwc
branch
from
365fc1d to
9f1b4f1
Compare
|
@elasticmachine retest this please |
|
I am not sure I can review this but I will give it a glance |
|
For the record, there are things that probably need to be done as follow-ups, like adding deprecation warnings to 5.x for the case that eg. a timestamp and/or a ttl are provided as part of indexing requests. |
jpountz
force-pushed
the
remove_mapping_bwc
branch
from
7562b12 to
f474344
Compare
|
I just rebased to fix conflicts with master. |
jpountz
added a commit
to jpountz/elasticsearch
that referenced
this pull request
Nov 28, 2016
`timestamp` and `ttl` already emit deprecation warnings when they are set in the mappings, but not when they are used in index requests. We should fix it so that users are not caught by surprise when we stop accepting these parameters. Relates elastic#21670
|
I went over this again and the changes look good. I think what we need to work on is to prevent that we join a cluster that has any indices allocated that were created with 2.x. We can do that in the handshake routine I will look into this and create an issue |
s1monw
added a commit
to s1monw/elasticsearch
that referenced
this pull request
Nov 28, 2016
Today we can easily join a cluster that holds an index we don't support since we currently allow rolling upgrades from 5.x to 6.x. This commit adds additional safety that fails cluster state validation if there is an open index with an incompatible index version created in the cluster. Realtes to elastic#21670
jpountz
added a commit
that referenced
this pull request
Nov 29, 2016
`timestamp` and `ttl` already emit deprecation warnings when they are set in the mappings, but not when they are used in index requests. We should fix it so that users are not caught by surprise when we stop accepting these parameters. Relates #21670
jpountz
added a commit
that referenced
this pull request
Nov 29, 2016
`timestamp` and `ttl` already emit deprecation warnings when they are set in the mappings, but not when they are used in index requests. We should fix it so that users are not caught by surprise when we stop accepting these parameters. Relates #21670
jpountz
force-pushed
the
remove_mapping_bwc
branch
from
4f0d92e to
a066aa3
Compare
For the record, I also had to remove the geo-hash cell and geo-distance range
queries to make the code compile. These queries already throw an exception in
all cases with 5.x indices, so that does not hurt any more.
I also had to rename all 2.x bwc indices from `index-${version}` to
`unsupported-${version}` to make `OldIndexBackwardCompatibilityIT`
happy.
jpountz
force-pushed
the
remove_mapping_bwc
branch
from
a066aa3 to
b53acfe
Compare
s1monw
added a commit
that referenced
this pull request
Dec 1, 2016
…rade & open index (#21830) Today we can easily join a cluster that holds an index we don't support since we currently allow rolling upgrades from 5.x to 6.x. Along the same lines we don't check if we can support an index based on the nodes in the cluster when we open, restore or metadata-upgrade and index. This commit adds additional safety that fails cluster state validation, open, restore and /or upgrade if there is an open index with an incompatible index version created in the cluster. Realtes to #21670
s1monw
added a commit
to s1monw/elasticsearch
that referenced
this pull request
Dec 1, 2016
…rade & open index (elastic#21830) Today we can easily join a cluster that holds an index we don't support since we currently allow rolling upgrades from 5.x to 6.x. Along the same lines we don't check if we can support an index based on the nodes in the cluster when we open, restore or metadata-upgrade and index. This commit adds additional safety that fails cluster state validation, open, restore and /or upgrade if there is an open index with an incompatible index version created in the cluster. Realtes to elastic#21670
s1monw
added a commit
that referenced
this pull request
Dec 2, 2016
…rade & open index (#21830) (#21914) Today we can easily join a cluster that holds an index we don't support since we currently allow rolling upgrades from 5.x to 6.x. Along the same lines we don't check if we can support an index based on the nodes in the cluster when we open, restore or metadata-upgrade and index. This commit adds additional safety that fails cluster state validation, open, restore and /or upgrade if there is an open index with an incompatible index version created in the cluster. Realtes to #21670
bleskes
added a commit
that referenced
this pull request
Dec 19, 2016
Sequence BWC logic consists of two elements: 1) Wire level BWC using stream versions. 2) A changed to the global checkpoint maintenance semantics. For the sequence number infra to work with a mixed version clusters, we have to consider situation where the primary is on an old node and replicas are on new ones (i.e., the replicas will receive operations without seq#) and also the reverse (i.e., the primary sends operations to a replica but the replica can't process the seq# and respond with local checkpoint). An new primary with an old replica is a rare because we do not allow a replica to recover from a new primary. However, it can occur if the old primary failed and a new replica was promoted or during primary relocation where the source primary is treated as a replica until the master starts the target. 1) Old Primary & New Replica - this case is easy as is taken care of by the wire level BWC. All incoming requests will have their seq# set to `UNASSIGNED_SEQ_NO`, which doesn't confuse the local checkpoint logic (keeping it at `NO_OPS_PERFORMED`) 2) New Primary & Old replica - this one is trickier as the global checkpoint service currently takes all in sync replicas into consideration for the global checkpoint calculation. In order to deal with old replicas, we change the semantics to say all *new node* in sync replicas. That means the replicas on old nodes don't count for the global checkpointing. In this state the seq# infra is not fully operational (you can't search on it, because copies may miss it) but it is maintained on shards that can support it. The old replicas will have to go through a file based recovery at some point and will get the seq# information at that point. There is still an edge case where a new primary fails and an old replica takes over. I'lll discuss this one with @ywelsch as I prefer to avoid it completely. This PR also re-enables the BWC tests which were disabled. As such it had to fix any BWC issue that had crept in. Most notably an issue with the removal of the `timestamp` field in #21670. The commit also includes a fix for the default value of the seq number field in replicated write requests (it was 0 but should be -2), that surface some other minor bugs which are fixed as well. Last - I added some debugging tools like more sane node names and forcing replication request to implement a `toString`
ywelsch
added a commit
that referenced
this pull request
May 24, 2017
This is a relict from the TTL functionality that has been removed in #21670
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For the record, I also had to remove the geo-hash cell and geo-distance range
queries to make the code compile. These queries already throw an exception in
all cases with 5.x indices, so that does not hurt any more.
I also had to rename all 2.x bwc indices from
index-${version}tounsupported-${version}to makeOldIndexBackwardCompatibilityIThappy.