Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] MlNodeShutdownIT testJobsVacateShuttingDownNode failing #77297

Closed
valeriy42 opened this issue Sep 6, 2021 · 1 comment · Fixed by #77304
Closed

[CI] MlNodeShutdownIT testJobsVacateShuttingDownNode failing #77297

valeriy42 opened this issue Sep 6, 2021 · 1 comment · Fixed by #77304
Labels
Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@valeriy42
Copy link
Contributor

The test failed 65 times in the past 3 days. Linux and Windows seem to be affected.

Build scan:
https://gradle-enterprise.elastic.co/s/zwaikfu7qp5x6/tests/:x-pack:plugin:ml:internalClusterTest/org.elasticsearch.xpack.ml.integration.MlNodeShutdownIT/testJobsVacateShuttingDownNode

Reproduction line:
./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.MlNodeShutdownIT.testJobsVacateShuttingDownNode" -Dtests.seed=16EFF3E0D6B3C497 -Dtests.locale=nl -Dtests.timezone=Asia/Dubai -Druntime.java=11 -Dtests.fips.enabled=true

Applicable branches:
master

Reproduces locally?:
Yes

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.ml.integration.MlNodeShutdownIT&tests.test=testJobsVacateShuttingDownNode

Failure excerpt:

org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: target node name is only valid for REPLACE type shutdowns, but was given type [RESTART] and target node name [yUitdkAztgOgIDLCExNQ];

  at __randomizedtesting.SeedInfo.seed([16EFF3E0D6B3C497:4F9332AA9E279F12]:0)
  at org.elasticsearch.xpack.shutdown.PutShutdownNodeAction$Request.validate(PutShutdownNodeAction.java:139)
  at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:42)
  at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:163)
  at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:100)
  at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:80)
  at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:375)
  at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54)
  at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:375)
  at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:364)
  at org.elasticsearch.xpack.ml.integration.MlNodeShutdownIT.testJobsVacateShuttingDownNode(MlNodeShutdownIT.java:79)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:566)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:834)

@valeriy42 valeriy42 added Team:ML Meta label for the ML team >test-failure Triaged test failures from CI labels Sep 6, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Sep 6, 2021
There was a small typo in elastic#77281 that made the test fail 2/9ths
of the time.

Fixes elastic#77297
droberts195 added a commit that referenced this issue Sep 6, 2021
There was a small typo in #77281 that made the test fail 2/9ths
of the time.

Fixes #77297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants