Bump Elasticsearch version to 9.0.0 #112570

mark-vieira · 2024-09-05T23:32:01Z

This is the initial work to bump the Elasticsearch version to 9.0 in preparation for our next major version release.

What this pull request includes

The stack version has been bumped to 9.0.0.
Backward compatibility testing logic has been updated to account for the new major version and branching changes. That is, once this PR is merged to main, the current main will be branched to 8.x. Back compat testing will now checkout the 8.x branch for testing that snapshot version.
The minimum wire compatibility version has been updated to 8.16.0. This is now determined dynamically based on known versions so no further changes should be necessary when 8.x is bumped to 8.17.
Rest compatibility tests are running using tests sourced from the 8.x branch. These are effectively identical to those in main so all existing rest compat overrides and transforms have been removed.
Our rest compatibility plugins have been simplified as well. Task names no longer include the previous major version as there will only ever be one of them. This will make future major version bumps slightly easier.
BWC test versions have been updated to remove all 7.x versions since there is now no level of backward compatibility with that version of Elasticsearch.
The minimum index version has been bumped to V_8_0_0 and all references in tests to index versions earlier to this have either been removed, or the test muted. This should mean that we can start wiping out production references to these versions, or replace them with placeholders, without affecting tests.
Any tests that were explicitly testing upgrades or migrations from 7.x index versions have been removed or muted. If it was unclear whether the test was strictly related to 7.x (since it likely was using a 7.x version constant) it was muted and annotated with @UpdateForV9 for later investigation.
The EQL correctness tests have been muted since they rely on restoring a snapshot of test data. This snapshot unfortunately is from Elasticsearch 7.x, so cannot be restored in a 9.0 cluster. A new 9.0-compatible snapshot will need to be created for these tests to be re-enabled. See EsEQLCorrectnessIT fails on Elasticsearch 9.0 #112572.

What this pull request does not include

Production code references to pre-8.0 index versions have not yet been removed. This will be done in a follow up.
The Lucene version remains 9.x. The upgrade to Lucene 10 is a separate effort.
The minimum transport version remains V_7_17_0. This will be bumped in a follow up.
The REST API version remains V_8. A constant has been added for V_9 but is not used. Also, the bootstrap check to ensure the current REST API version matches the current major version has been disabled. This will need to be re-enabled once the REST API version is incremented.
No functional changes. For all intents and purposes this behaves exactly like 8.x, with the exception of the removal of support for pre-8.0 indexes. Anything deprecated for removal in 9.0 still remains. This will be done in a follow up.
~~Native ML code is still version 8.16.0. We do not yet have 9.0 snapshot builds of ml-cpp, so this will need to be updated in a follow up.~~ (done in this PR)

# Conflicts: # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy

# Conflicts: # .buildkite/pipelines/intake.yml # .buildkite/pipelines/periodic.yml # .ci/snapshotBwcVersions

jakelandis · 2024-09-09T15:07:25Z

@elasticmachine update branch

brianseeders · 2024-09-09T16:40:24Z

@elasticmachine merge upstream

jakelandis

LGTM

thecoop · 2024-09-10T09:03:15Z

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/BwcVersions.java

@@ -286,7 +266,17 @@ public List<Version> getUnreleasedWireCompatible() {
    }

    public Version getMinimumWireCompatibleVersion() {
-        return MINIMUM_WIRE_COMPATIBLE_VERSION;
+        // Determine minimum wire compatible version from list of known versions.
+        // Current BWC policy states the minimum wire compatible version is the last minor release or the previous major version.


typo or -> of

thecoop · 2024-09-10T09:05:43Z

.../elasticsearch/gradle/internal/test/rest/compat/compat/AbstractYamlRestCompatTestPlugin.java

+            .getUnreleased()
+            .stream()
+            .filter(v -> v.getMajor() == currentMajor - 1)
+            .min(Comparator.reverseOrder())


max(Comparator.naturalOrder()) is more understandable

thecoop · 2024-09-10T09:06:37Z

build-tools-internal/src/test/groovy/org/elasticsearch/gradle/internal/BwcVersionsSpec.groovy

            (v('8.1.0')): new UnreleasedVersionInfo(v('8.1.0'), 'main', ':distribution')
        ]
        bwc.wireCompatible == [v('7.17.0'), v('8.0.0'), v('8.1.0')]
-        bwc.indexCompatible == osFiltered([v('7.14.0'), v('7.14.1'), v('7.14.2'), v('7.15.0'), v('7.15.1'), v('7.15.2'), v('7.16.0'), v('7.16.1'), v('7.16.2'), v('7.17.0'), v('8.0.0'), v('8.1.0')])
+        bwc.indexCompatible == [v('7.14.0'), v('7.14.1'), v('7.14.2'), v('7.15.0'), v('7.15.1'), v('7.15.2'), v('7.16.0'), v('7.16.1'), v('7.16.2'), v('7.17.0'), v('8.0.0'), v('8.1.0')]


I'm not sure what's going on here, but I assume it's deliberate that indexCompatible /minWireCompatible is still referring to v7 versions

brianseeders · 2024-09-10T17:28:47Z

@elasticmachine merge upstream

jakelandis · 2024-09-10T19:09:24Z

@elasticmachine merge upstream

…sync with main

jakelandis · 2024-09-11T12:51:34Z

@elasticmachine merge upstream

elasticmachine · 2024-09-11T12:51:37Z

merge conflict between base and head

…tion-ironbank-ubi * upstream/main: (302 commits) Deduplicate BucketOrder when deserializing (elastic#112707) Introduce test utils for ingest pipelines (elastic#112733) [Test] Account for auto-repairing for shard gen file (elastic#112778) Do not throw in task enqueued by CancellableRunner (elastic#112780) Mute org.elasticsearch.script.StatsSummaryTests testEqualsAndHashCode elastic#112439 Mute org.elasticsearch.repositories.blobstore.testkit.integrity.RepositoryVerifyIntegrityIT testTransportException elastic#112779 Use a dedicated test executor in MockTransportService (elastic#112748) Estimate segment field usages (elastic#112760) (Doc+) Inference Pipeline ignores Mapping Analyzers (elastic#112522) Fix verifyVersions task (elastic#112765) (Doc+) Terminating Exit Codes (elastic#112530) (Doc+) CAT Nodes default columns (elastic#112715) [DOCS] Augment installation warnings (elastic#112756) Mute org.elasticsearch.repositories.blobstore.testkit.integrity.RepositoryVerifyIntegrityIT testCorruption elastic#112769 Bump Elasticsearch to a minimum of JDK 21 (elastic#112252) ESQL: Compute support for filtering ungrouped aggs (elastic#112717) Bump Elasticsearch version to 9.0.0 (elastic#112570) add CDR related data streams to kibana_system priviliges (elastic#112655) Support widening of numeric types in union-types (elastic#112610) Introduce data stream options and failure store configuration classes (elastic#109515) ...

mark-vieira added 27 commits September 3, 2024 15:56

Initial version bump to Elasticsearch 9.0

4de21a2

Reintroduce concept of a major-1.x branch to BwcVersions

da4131a

Update rest compat plugin tests

57f4549

Remove testing of 7.x here for now

ec32cec

Use minimum wire compatible version for CCS testing

d865159

Remove overriden rest compatibility tests

23a67bc

Fix more build-tools tests

934f805

Skip the DRA step for now

849dbf6

Filter known index versions earlier than minimum compatible version

7f4d253

Revert minimum index version for now, we can follow up

96f129f

Fix full cluster restart tests against older elasticsearch versions

4224077

Let's shake out all the tests

0c6f6a7

Just use API version 8 for now

4f97013

Disable rest api version assertion

2902a7e

More test fixes

df7a1a9

Merge branch 'main' into es-9.0-bump

68c03c2

# Conflicts: # build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy

Remove rest compat testing overrides

5cee3be

Remove compat test config

abefa06

Mute some tests that need to be updated for 9.0

bfcd057

Bump minimum compatible index version

097e52f

Update pipelines

ee3d58b

Merge branch 'main' into es-9.0-bump

f14b7d0

# Conflicts: # .buildkite/pipelines/intake.yml # .buildkite/pipelines/periodic.yml # .ci/snapshotBwcVersions

Bump versions for 8.15.1 release

c6fa8d3

Remove (or mute) all test references to pre 8.0 index versions

aa40d48

Cleanup

c567e4d

More test fixes

960ce1c

Remove temporary skip

05a5469

mark-vieira added :Delivery/Build Build or test infrastructure :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts test-windows Trigger CI checks on Windows labels Sep 5, 2024

mark-vieira added 3 commits September 6, 2024 13:45

Simply BwcVersions to remove filtering of unavailable aarch64 versions

500f49b

Spotless

0867135

Merge branch 'main' into es-9.0-bump

2df8ffa

Merge branch 'main' into es-9.0-bump

b399a2a

elasticmachine requested a review from a team as a code owner September 9, 2024 15:07

brianseeders added the test-update-serverless label Sep 9, 2024

Merge branch 'main' into es-9.0-bump

4396152

jakelandis approved these changes Sep 9, 2024

View reviewed changes

thecoop approved these changes Sep 10, 2024

View reviewed changes

thecoop reviewed Sep 10, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into es-9.0-bump

bf48b1d

Merge branch 'main' into es-9.0-bump

f263fe3

elasticmachine and others added 3 commits September 10, 2024 21:09

Merge branch 'main' into es-9.0-bump

2fa0cc2

remove reference to yamlRestTestV7CompatTransform brought in by last …

10ba96c

…sync with main

bump the ML version

9ea48ec

Merge remote-tracking branch 'upstream/main' into es-9.0-bump

bd39d7c

mark-vieira merged commit 4ce661c into main Sep 11, 2024
20 checks passed

mark-vieira deleted the es-9.0-bump branch September 11, 2024 16:40

davidkyle pushed a commit that referenced this pull request Sep 12, 2024

Bump Elasticsearch version to 9.0.0 (#112570)

1a05488

linghengqian mentioned this pull request Sep 24, 2024

Removes use of elasticsearch-rest-high-level-client to support the upcoming release of ElasticSearch 9.0.0 dromara/easy-es#132

Open

arteam mentioned this pull request Oct 30, 2024

[test] Dynamically pick up the upper bound snapshot index version #114703

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump Elasticsearch version to 9.0.0 #112570

Bump Elasticsearch version to 9.0.0 #112570

mark-vieira commented Sep 5, 2024 •

edited by jakelandis

Loading

jakelandis commented Sep 9, 2024

brianseeders commented Sep 9, 2024

jakelandis left a comment

thecoop Sep 10, 2024

thecoop Sep 10, 2024

thecoop Sep 10, 2024

brianseeders commented Sep 10, 2024

jakelandis commented Sep 10, 2024

jakelandis commented Sep 11, 2024

elasticmachine commented Sep 11, 2024

Bump Elasticsearch version to 9.0.0 #112570

Bump Elasticsearch version to 9.0.0 #112570

Conversation

mark-vieira commented Sep 5, 2024 • edited by jakelandis Loading

What this pull request includes

What this pull request does not include

jakelandis commented Sep 9, 2024

brianseeders commented Sep 9, 2024

jakelandis left a comment

Choose a reason for hiding this comment

thecoop Sep 10, 2024

Choose a reason for hiding this comment

thecoop Sep 10, 2024

Choose a reason for hiding this comment

thecoop Sep 10, 2024

Choose a reason for hiding this comment

brianseeders commented Sep 10, 2024

jakelandis commented Sep 10, 2024

jakelandis commented Sep 11, 2024

elasticmachine commented Sep 11, 2024

mark-vieira commented Sep 5, 2024 •

edited by jakelandis

Loading