Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] NodeShutdownShardsIT testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline failing #76689

Closed
ywangd opened this issue Aug 18, 2021 · 5 comments · Fixed by #77220 or #106502
Closed

[CI] NodeShutdownShardsIT testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline failing #76689

ywangd opened this issue Aug 18, 2021 · 5 comments · Fixed by #77220 or #106502
Assignees
@rjernst
Labels
:Core/Infra/Node Lifecycle Node startup, bootstrapping, and shutdown medium-risk An open issue or test failure that is a medium risk to future releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI

Comments

@ywangd
Copy link
Member

ywangd commented Aug 18, 2021

This failure happened on my docs PR. But the error message seems legit.

Build scan:
https://gradle-enterprise.elastic.co/s/fgw373m4igido/tests/:x-pack:plugin:shutdown:internalClusterTest/org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT/testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline

Reproduction line:
./gradlew ':x-pack:plugin:shutdown:internalClusterTest' --tests "org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT.testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline" -Dtests.seed=337514F1572DD3FC -Dtests.locale=ar-MA -Dtests.timezone=Australia/Victoria -Druntime.java=11 -Dtests.fips.enabled=true

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT&tests.test=testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline

Failure excerpt:

java.lang.AssertionError: 
Expected: <COMPLETE>
     but: was <NOT_STARTED>

  at __randomizedtesting.SeedInfo.seed([337514F1572DD3FC:F5DB7C92118651C6]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT.testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline(NodeShutdownShardsIT.java:106)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:566)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:834)
@ywangd ywangd added :Core/Infra/Node Lifecycle Node startup, bootstrapping, and shutdown >test-failure Triaged test failures from CI labels Aug 18, 2021
@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Aug 18, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@ywangd ywangd mentioned this issue Aug 18, 2021
Merged
@imotov imotov mentioned this issue Aug 19, 2021
Merged
@rjernst rjernst mentioned this issue Aug 20, 2021
Closed
@davidkyle davidkyle mentioned this issue Aug 23, 2021
Merged
@davidkyle
Copy link
Member

Muted on master in abad722

@gwbrown gwbrown self-assigned this Sep 1, 2021
@gwbrown gwbrown mentioned this issue Sep 2, 2021
Merged
@ywangd
Copy link
Member Author

ywangd commented Oct 12, 2021

Reopenning as it still fails on master intake: https://gradle-enterprise.elastic.co/s/lq42hbge2queu/tests/:x-pack:plugin:shutdown:internalClusterTest/org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT/testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline?top-execution=1

Error message is the same. Not reproducible locally.

@ywangd ywangd reopened this Oct 12, 2021
@ywelsch
Copy link
Contributor

ywelsch commented Oct 20, 2021

Continues to fail: https://gradle-enterprise.elastic.co/s/rr7e4yovmgjca/tests/:x-pack:plugin:shutdown:internalClusterTest/org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT/testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline?top-execution=1

remuted in 4179247

@martijnvg
Copy link
Member

This failed on the 7.17 branch: https://gradle-enterprise.elastic.co/s/yovevakxaakh4/tests/:x-pack:plugin:shutdown:internalClusterTest/org.elasticsearch.xpack.shutdown.NodeShutdownShardsIT/testShardStatusStaysCompleteAfterNodeLeavesIfRegisteredWhileNodeOffline?top-execution=1

In total this failed twice in the last 3 months.
I will mute this test on the 7.17 branch as well.

@stu-elastic stu-elastic added the medium-risk An open issue or test failure that is a medium risk to future releases label Oct 17, 2023
@rjernst rjernst assigned rjernst and unassigned gwbrown Mar 19, 2024
rjernst added a commit to rjernst/elasticsearch that referenced this issue Mar 19, 2024
@rjernst
Mark node seen task as successful
489c4ed 
The task for updating cluster state with nodes seen by shutdown was
previously switched to use batched tasks. However, the task is never
marked as complete, which leads to the tasks piling up. This commit
marks the task as complete and re-enables a test that appears to succeed
now.

closes elastic#76689
@rjernst rjernst mentioned this issue Mar 19, 2024
Merged
rjernst added a commit that referenced this issue Mar 19, 2024
@rjernst
Mark node seen task as successful (#106502)
83bc94a 
The task for updating cluster state with nodes seen by shutdown was
previously switched to use batched tasks. However, the task is never
marked as complete, which leads to the tasks piling up. This commit
marks the task as complete and re-enables a test that appears to succeed
now.

closes #76689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rjernst rjernst

Labels
:Core/Infra/Node Lifecycle Node startup, bootstrapping, and shutdown medium-risk An open issue or test failure that is a medium risk to future releases Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
8 participants
@rjernst @martijnvg @gwbrown @ywangd @davidkyle @ywelsch @elasticmachine @stu-elastic