Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] PrimaryReplicaSyncerTests failures with #32030

Closed
cbuescher opened this issue Jul 13, 2018 · 3 comments · Fixed by #32045
Closed

[CI] PrimaryReplicaSyncerTests failures with #32030

cbuescher opened this issue Jul 13, 2018 · 3 comments · Fixed by #32045
Assignees
@dnhatn
Labels
:Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. >test-failure Triaged test failures from CI v7.0.0-beta1

Comments

@cbuescher
Copy link
Member

Failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=amazon/2554/console

java.lang.RuntimeException: file handle leaks: [FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/amazon/server/build/testrun/test/J2/temp/org.elasticsearch.index.shard.PrimaryReplicaSyncerTests_49C4DD0A60DD4DED-001/tempDir-004/indices/_na_/0/translog/translog-1.tlog), FileChannel(/var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/amazon/server/build/testrun/test/J2/temp/org.elasticsearch.index.shard.PrimaryReplicaSyncerTests_49C4DD0A60DD4DED-001/tempDir-004/indices/_na_/0/translog/translog-2.tlog)]
	at __randomizedtesting.SeedInfo.seed([49C4DD0A60DD4DED]:0)
	at org.apache.lucene.mockfile.LeakFS.onClose(LeakFS.java:63)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:77)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:78)
	at org.apache.lucene.mockfile.FilterFileSystem.close(FilterFileSystem.java:78)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:228)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception
	at org.apache.lucene.mockfile.LeakFS.onOpen(LeakFS.java:46)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:81)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:197)
	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:166)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:202)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.elasticsearch.index.translog.Translog.openReader(Translog.java:279)
	at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:225)
	at org.elasticsearch.index.translog.Translog.<init>(Translog.java:177)
	at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:424)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:183)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:156)
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:2124)
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:2106)
	at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1318)
	at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1273)
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:415)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:301)
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1540)
	at org.elasticsearch.index.shard.IndexShardTestCase.recoverShardFromStore(IndexShardTestCase.java:386)
	at org.elasticsearch.index.shard.IndexShardTestCase.newStartedShard(IndexShardTestCase.java:359)
	at org.elasticsearch.index.shard.PrimaryReplicaSyncerTests.testSyncerOnClosingShard(PrimaryReplicaSyncerTests.java:124)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	... 8 more

Unfortunately doesn't locally REPRODUCE WITH:

./gradlew :server:test \
  -Dtests.seed=49C4DD0A60DD4DED \
  -Dtests.class=org.elasticsearch.index.shard.PrimaryReplicaSyncerTests \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Zulu
@cbuescher cbuescher added >test-failure Triaged test failures from CI v7.0.0 :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. labels Jul 13, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@cbuescher
Copy link
Member Author

Similar looking failures:

Jul 9:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu&&virtual/2546/console

Jun 23 (6.x. branch):
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+periodic/2240/console

@dnhatn dnhatn self-assigned this Jul 13, 2018
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue Jul 13, 2018
@dnhatn
Ensure to release translog snapshot in primary-replica resync
2cd685a 
We create a translog snapshot inside the resync method, and that
snapshot will be closed by the resync listener. However, if the resync
method throws an exception before the resync listener is initialized,
the translog snapshot won't be released.

Closes elastic#32030
@dnhatn dnhatn mentioned this issue Jul 13, 2018
Merged
@dnhatn
Copy link
Member

dnhatn commented Jul 13, 2018

I opened #32045

dnhatn added a commit that referenced this issue Jul 17, 2018
@dnhatn
Ensure to release translog snapshot in primary-replica resync (#32045)
ef81c1d 
Previously we create a translog snapshot inside the resync method, 
and that snapshot will be closed by the resync listener. However, if 
the resync method throws an exception before the resync listener 
is initialized, the translog snapshot won't be released.

Closes #32030
dnhatn added a commit that referenced this issue Jul 17, 2018
@dnhatn
Ensure to release translog snapshot in primary-replica resync (#32045)
d223845 
Previously we create a translog snapshot inside the resync method, 
and that snapshot will be closed by the resync listener. However, if 
the resync method throws an exception before the resync listener 
is initialized, the translog snapshot won't be released.

Closes #32030
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
:Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. >test-failure Triaged test failures from CI v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure to release translog snapshot in primary-replica resync dnhatn/elasticsearch
4 participants
@colings86 @cbuescher @dnhatn @elasticmachine