Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] AutodetectMemoryLimitIT testManyDistinctOverFields failing #105347

Closed
DaveCTurner opened this issue Feb 9, 2024 · 2 comments · Fixed by #105727
Closed

[CI] AutodetectMemoryLimitIT testManyDistinctOverFields failing #105347

DaveCTurner opened this issue Feb 9, 2024 · 2 comments · Fixed by #105727
Labels
blocker :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

Has failed like this a couple of times in the last 90d

Build scan:
https://gradle-enterprise.elastic.co/s/vowhqrir6w5zo/tests/:x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest/org.elasticsearch.xpack.ml.integration.AutodetectMemoryLimitIT/testManyDistinctOverFields

Reproduction line:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest' --tests "org.elasticsearch.xpack.ml.integration.AutodetectMemoryLimitIT.testManyDistinctOverFields" -Dtests.seed=857E02DD582003FE -Dtests.locale=id-ID -Dtests.timezone=Indian/Reunion -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.xpack.ml.integration.AutodetectMemoryLimitIT#testManyDistinctOverFields

Failure excerpt:

java.lang.AssertionError: 
Expected: a value less than <120000000L>
     but: <120173944L> was greater than <120000000L>

  at __randomizedtesting.SeedInfo.seed([857E02DD582003FE:30A629233E9A127B]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2119)
  at org.elasticsearch.xpack.ml.integration.AutodetectMemoryLimitIT.testManyDistinctOverFields(AutodetectMemoryLimitIT.java:228)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)
@DaveCTurner DaveCTurner added :ml Machine learning >test-failure Triaged test failures from CI labels Feb 9, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine elasticsearchmachine added blocker Team:ML Meta label for the ML team labels Feb 9, 2024
DaveCTurner added a commit that referenced this issue Feb 9, 2024
@DaveCTurner
AwaitsFix for #105347
c9b5f7b
@droberts195
Copy link
Contributor

The first failure was on 3rd November last year.

The PR that most likely caused this is elastic/ml-cpp#2585, which was merged on 11th October. So there's a fair gap there before the first failure, but given how sporadic the failures were it's not too hard to believe.

The failure margin over the expected upper bound is tiny, so I'll just increase it a bit.

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Feb 22, 2024
@droberts195
[ML] Fix AutodetectMemoryLimitIT.testManyDistinctOverFields
9c6685b 
It seems that the changes of elastic/ml-cpp#2585
combined with the randomness of the test could cause it to fail
very occasionally, and by a tiny percentage over the expected
upper bound. This change reenables the test by very slightly
increasing the upper bound.

Fixes elastic#105347
@droberts195 droberts195 mentioned this issue Feb 22, 2024
Merged
droberts195 added a commit that referenced this issue Feb 22, 2024
@droberts195
[ML] Fix AutodetectMemoryLimitIT.testManyDistinctOverFields (#105727)
268ba12 
It seems that the changes of elastic/ml-cpp#2585
combined with the randomness of the test could cause it to fail
very occasionally, and by a tiny percentage over the expected
upper bound. This change reenables the test by very slightly
increasing the upper bound.

Fixes #105347
droberts195 added a commit to droberts195/elasticsearch that referenced this issue Feb 22, 2024
@droberts195
[ML] Fix AutodetectMemoryLimitIT.testManyDistinctOverFields (elastic#…
1d93db3 
…105727)

It seems that the changes of elastic/ml-cpp#2585
combined with the randomness of the test could cause it to fail
very occasionally, and by a tiny percentage over the expected
upper bound. This change reenables the test by very slightly
increasing the upper bound.

Fixes elastic#105347
elasticsearchmachine pushed a commit that referenced this issue Feb 22, 2024
@droberts195
[ML] Fix AutodetectMemoryLimitIT.testManyDistinctOverFields (#105727) (…
8744ebb 
…#105734)

It seems that the changes of elastic/ml-cpp#2585
combined with the randomness of the test could cause it to fail
very occasionally, and by a tiny percentage over the expected
upper bound. This change reenables the test by very slightly
increasing the upper bound.

Fixes #105347
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocker :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Fix AutodetectMemoryLimitIT.testManyDistinctOverFields droberts195/elasticsearch
3 participants
@DaveCTurner @droberts195 @elasticsearchmachine