Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to a Lucene 8 snapshot #33310

Merged
merged 96 commits into from
Sep 6, 2018
Merged

Upgrade to a Lucene 8 snapshot #33310

merged 96 commits into from
Sep 6, 2018

Conversation

jimczi
Copy link
Contributor

@jimczi jimczi commented Aug 31, 2018

The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:

  • I muted some tests in FieldSubsetReaderTests, they are fixed in another pr.
  • Tests that can produce negative scores have been adapted but we need to forbid them completely:
    Forbid negative query scores #33309

Closes #32899

jimczi and others added 30 commits August 9, 2018 10:06
The merge policy that was used could lead to unpredictive merges due to the
randomization of `setDeletesPctAllowed`.

Closes #32457
We do not support passphrases on the secure settings storage (the
keystore). Yet, we added support for this in the API layer. This commit
removes this support so that we are not limited in our future options,
or have to make a breaking change.
* HLRC: adding machine learning delete job

* Fixing whitespace

* Moving docs and tests around

* Unifying ml asciidoc file naming convention
This commit disables the automatic `refresh_interval` in order to ensure
that index readers cannot differ between the normal and scroll search.
This issue is related to the 7.5 Lucene upgrade which contains a change that
makes single segment merge more likely to occur (max deletes percentage).

Closes #32682
This removes def from the classes map in PainlessLookup and instead always special 
cases it. This prevents potential calls against the def type that shouldn't be made and 
forces all cases of def throughout Painless code to be special cased.
The docs here incorrectly state that it is okay for a heap dump file to
exist when heap dump path is configured to a fixed filename. This is
incorrect, the JVM will fail to write the heap dump if a heap dump file
already exists at the specified location (see the DumpWriter constructor
DumpWriter::DumpWriter(const char* path) in the JVM source).
[ML] Re-enabling BWC tests

Re-enable BWC tests for ML now that #32816 has been backported to 6.x
The Kibana settings docs that these watches rely on can sometimes
contain no xpack settings. When this is the case, we will end up with a
null pointer exception in the script. We need to guard against in these
scripts so this commit does that.
This commit removes the put privilege API in favor of having a single API to
create and update privileges. If we see the need to have an API like this in
the future we can always add it back.
Add tests for build-tools to make sure example plugins build stand-alone using it.

This will catch issues such as referencing files from the buildSrc directly, breaking external uses of build-tools.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

Copy link
Contributor

@jpountz jpountz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jimczi jimczi merged commit 7ad71f9 into master Sep 6, 2018
@jimczi jimczi deleted the upgrade/lucene-8 branch September 6, 2018 12:42
romseygeek added a commit that referenced this pull request Sep 21, 2020
The `standard` tokenfilter was removed by #33310, and should have been
unuseable in any indexes created since 7.0. However, a cacheing bug fixed
by #51092 meant that it was still possible in certain circumstances to create
indexes referencing the standard filter in versions up to 7.5.2. Our checks
in AnalysisModule still refer to 7.0.0, however, meaning that a cluster that
contains one of these rogue indexes cannot be upgraded.

This commit adjusts the AnalysisModule checks so that we only refuse to
build a mapping referring to standard filter if the index created version is
7.6 or later.

Fixes #62644
romseygeek added a commit that referenced this pull request Sep 21, 2020
The `standard` tokenfilter was removed by #33310, and should have been
unuseable in any indexes created since 7.0. However, a cacheing bug fixed
by #51092 meant that it was still possible in certain circumstances to create
indexes referencing the standard filter in versions up to 7.5.2. Our checks
in AnalysisModule still refer to 7.0.0, however, meaning that a cluster that
contains one of these rogue indexes cannot be upgraded.

This commit adjusts the AnalysisModule checks so that we only refuse to
build a mapping referring to standard filter if the index created version is
7.6 or later.

Fixes #62644
romseygeek added a commit that referenced this pull request Sep 21, 2020
The `standard` tokenfilter was removed by #33310, and should have been
unuseable in any indexes created since 7.0. However, a cacheing bug fixed
by #51092 meant that it was still possible in certain circumstances to create
indexes referencing the standard filter in versions up to 7.5.2. Our checks
in AnalysisModule still refer to 7.0.0, however, meaning that a cluster that
contains one of these rogue indexes cannot be upgraded.

This commit adjusts the AnalysisModule checks so that we only refuse to
build a mapping referring to standard filter if the index created version is
7.6 or later.

Fixes #62644
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lucene 8 upgrade checklist