Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BWC test failures for MultiVersionRepositoryAccessIT #50819

Closed
mark-vieira opened this issue Jan 9, 2020 · 17 comments · Fixed by #50848, #51120 or #51246
Closed

BWC test failures for MultiVersionRepositoryAccessIT #50819

mark-vieira opened this issue Jan 9, 2020 · 17 comments · Fixed by #50848, #51120 or #51246
Assignees
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI

Comments

@mark-vieira
Copy link
Contributor

mark-vieira commented Jan 9, 2020

Our most recent master BWC test run failed against versions 7.0.x and 7.1.x with the same test failures indicating we either made a BWC incompatible change in master or perhaps just the test needs updating.

The build run in question included this commit, which might potentially be related. @original-brownbear any thoughts?

Example build scan: https://gradle-enterprise.elastic.co/s/dpoop7fvn2rwq/tests/failed
Jenkins build with links to each BWC version: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc/471/

@mark-vieira mark-vieira added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Jan 9, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@original-brownbear original-brownbear self-assigned this Jan 9, 2020
@original-brownbear
Copy link
Member

@mark-vieira yea I know how to fix this (already did in 7.x) but I'm afraid I will only be able to do so in 10-12h. Will do it right away in my morning sorry for the noise

@mark-vieira
Copy link
Contributor Author

No rush. These builds only run every 8 hours so we are talking maybe 1 or 2 more failures so hardly much in the way of noise in the grand scheme of things. Thanks, Armin!

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 10, 2020
We need the same fix we did in `7.x` (elastic#50797) and only get snapshot status
for the current version or older. Otherwise these tests break for
e.g.`7.0.1` due to the same index metadata incompatibility.

Closes elastic#50819
original-brownbear added a commit that referenced this issue Jan 10, 2020
We need the same fix we did in `7.x` (#50797) and only get snapshot status
for the current version or older. Otherwise these tests break for
e.g.`7.0.1` due to the same index metadata incompatibility.

Closes #50819
@mark-vieira
Copy link
Contributor Author

@mark-vieira mark-vieira reopened this Jan 13, 2020
@original-brownbear
Copy link
Member

@mark-vieira ah sorry about that! Will fix it right away in my morning in 12h

@mark-vieira
Copy link
Contributor Author

Thanks, Armin!

@danielmitterdorfer
Copy link
Member

For the sake of completeness: We have two more build failures in:

(omitting build scans as they don't add any new information)

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 14, 2020
The HLRC doesn't like what its getting back from some older 6.x versions for the restore status
so I moved that request to the low level client.
Closes elastic#50819
@original-brownbear
Copy link
Member

Relatively trivial fix incoming in #51006 :)

@mark-vieira
Copy link
Contributor Author

Thanks again, Armin!

@original-brownbear
Copy link
Member

npnp closed via #51006

original-brownbear added a commit that referenced this issue Jan 15, 2020
* Fix MultiVersionRepository BwC Tests

The HLRC doesn't like what its getting back from some older 6.x versions for the restore status
so I moved that request to the low level client.
Closes #50819
@mark-vieira
Copy link
Contributor Author

@mark-vieira mark-vieira reopened this Jan 16, 2020
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 16, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes elastic#50819
@original-brownbear
Copy link
Member

Sorry about that, the failure we're seeing there could in fact also hit master fix incoming in #51120 ... tests failed to account for running on non-OSS clusters

original-brownbear added a commit that referenced this issue Jan 16, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes #50819
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 16, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes elastic#50819
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 16, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes elastic#50819
original-brownbear added a commit that referenced this issue Jan 16, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes #50819
original-brownbear added a commit that referenced this issue Jan 16, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes #50819
@tvernum
Copy link
Contributor

tvernum commented Jan 20, 2020

Sorry @original-brownbear it looks like it's still failing in 7.x (694b8ab) BWC against 6.4.3
https://gradle-enterprise.elastic.co/s/uczk6mc54bwts/tests/kza37bzrrhlvw-pwyqvbdz22w3g

@tvernum tvernum reopened this Jan 20, 2020
@tvernum
Copy link
Contributor

tvernum commented Jan 20, 2020

I haven't checked whether the logs / causes are identical, but it looks like it's broken from [6.3.0, 6.5.4]

@ywangd
Copy link
Member

ywangd commented Jan 20, 2020

Same type of failures also occur for 7.6 BWC tests against from 6.3.0 to 6.6.4

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/13/

@original-brownbear
Copy link
Member

Looks like there is yet another random failure left here unrelated to those fixed so far. I'm on it again.

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 21, 2020
If the rollup jobs index doesn't exist for some reason (like running against a 6.x cluster)
we should just assume the jobs have been cleaned up and move on.

Closes elastic#50819
@original-brownbear
Copy link
Member

Fix incoming in #51246

original-brownbear added a commit that referenced this issue Jan 22, 2020
* Fix Rest Tests Failing to Cleanup Rollup Jobs

If the rollup jobs index doesn't exist for some reason (like running against a 6.x cluster)
we should just assume the jobs have been cleaned up and move on.

Closes #50819
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 22, 2020
* Fix Rest Tests Failing to Cleanup Rollup Jobs

If the rollup jobs index doesn't exist for some reason (like running against a 6.x cluster)
we should just assume the jobs have been cleaned up and move on.

Closes elastic#50819
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 22, 2020
* Fix Rest Tests Failing to Cleanup Rollup Jobs

If the rollup jobs index doesn't exist for some reason (like running against a 6.x cluster)
we should just assume the jobs have been cleaned up and move on.

Closes elastic#50819
original-brownbear added a commit that referenced this issue Jan 22, 2020
* Fix Rest Tests Failing to Cleanup Rollup Jobs

If the rollup jobs index doesn't exist for some reason (like running against a 6.x cluster)
we should just assume the jobs have been cleaned up and move on.

Closes #50819
original-brownbear added a commit that referenced this issue Jan 22, 2020
* Fix Rest Tests Failing to Cleanup Rollup Jobs

If the rollup jobs index doesn't exist for some reason (like running against a 6.x cluster)
we should just assume the jobs have been cleaned up and move on.

Closes #50819
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
We need the same fix we did in `7.x` (elastic#50797) and only get snapshot status
for the current version or older. Otherwise these tests break for
e.g.`7.0.1` due to the same index metadata incompatibility.

Closes elastic#50819
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes elastic#50819
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment