Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trino-raptor-legacy tests are flaky, e.g. sometimes fail with OutOfMemoryError #12726

Closed
findepi opened this issue Jun 7, 2022 · 7 comments · Fixed by #12777
Closed

trino-raptor-legacy tests are flaky, e.g. sometimes fail with OutOfMemoryError #12726

findepi opened this issue Jun 7, 2022 · 7 comments · Fixed by #12777
Labels
bug Something isn't working

Comments

@findepi
Copy link
Member

findepi commented Jun 7, 2022

No description provided.

@findepi findepi added the bug Something isn't working label Jun 7, 2022
@findepi
Copy link
Member Author

findepi commented Jun 7, 2022

https://github.com/trinodb/trino/runs/6773580556?check_suite_focus=true

Error:  java.lang.OutOfMemoryError: Java heap space
Error:  Terminating due to java.lang.OutOfMemoryError: Java heap space
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
Warning:  Corrupted STDOUT by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire-reports/2022-06-07T13-56-41_617-jvmRun1.dumpstream
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19:57 min
[INFO] Finished at: 2022-06-07T14:16:33Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project trino-raptor-legacy: There are test failures.
Error: 
Error:  Please refer to /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire-reports for the individual test results.
Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/trino/trino/plugin/trino-raptor-legacy && /opt/hostedtoolcache/Java_Zulu_jdk/11.0.15-10/x64/bin/java -Dfile.encoding=UTF-8 -Xmx3g -Xms3g -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire/surefirebooter17473602102114892421.jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire 2022-06-07T13-56-41_617-jvmRun1 surefire888842529460435438tmp surefire_03751112285414833365tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 3
Error:  Crashed tests:
Error:  io.trino.plugin.raptor.legacy.TestRaptorBucketedConnectorTest
Error:  io.trino.plugin.raptor.legacy.TestRaptorConnectorTest
Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/trino/trino/plugin/trino-raptor-legacy && /opt/hostedtoolcache/Java_Zulu_jdk/11.0.15-10/x64/bin/java -Dfile.encoding=UTF-8 -Xmx3g -Xms3g -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire/surefirebooter17473602102114892421.jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire 2022-06-07T13-56-41_617-jvmRun1 surefire888842529460435438tmp surefire_03751112285414833365tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 3
Error:  Crashed tests:
Error:  io.trino.plugin.raptor.legacy.TestRaptorBucketedConnectorTest
Error:  io.trino.plugin.raptor.legacy.TestRaptorConnectorTest
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Error:  	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
Error:  	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
Error:  	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
Error:  	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:972)
Error:  	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
Error:  	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Error:  	at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:47)
Error:  	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:156)
Error:  	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:72)
Error:  -> [Help 1]

@findepi findepi mentioned this issue Jun 7, 2022
Merged
@findepi
Copy link
Member Author

findepi commented Jun 8, 2022

https://github.com/trinodb/trino/runs/6783236500?check_suite_focus=true

Error:  java.lang.OutOfMemoryError: Java heap space
Error:  Terminating due to java.lang.OutOfMemoryError: Java heap space
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
Warning:  Corrupted STDOUT by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire-reports/2022-06-07T22-56-19_016-jvmRun1.dumpstream
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19:40 min
[INFO] Finished at: 2022-06-07T23:15:55Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project trino-raptor-legacy: There are test failures.
Error: 
Error:  Please refer to /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire-reports for the individual test results.
Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/trino/trino/plugin/trino-raptor-legacy && /opt/hostedtoolcache/Java_Zulu_jdk/11.0.15-10/x64/bin/java -Dfile.encoding=UTF-8 -Xmx3g -Xms3g -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire/surefirebooter5439167013146119987.jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire 2022-06-07T22-56-19_016-jvmRun1 surefire10605864679852646269tmp surefire_01264388357155[8364](https://github.com/trinodb/trino/runs/6783236500?check_suite_focus=true#step:8:8365)398tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 3
Error:  Crashed tests:
Error:  io.trino.plugin.raptor.legacy.TestRaptorConnectorTest
Error:  io.trino.plugin.raptor.legacy.TestRaptorMySqlConnectorTest
Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/trino/trino/plugin/trino-raptor-legacy && /opt/hostedtoolcache/Java_Zulu_jdk/11.0.15-10/x64/bin/java -Dfile.encoding=UTF-8 -Xmx3g -Xms3g -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire/surefirebooter5439167013146119987.jar /home/runner/work/trino/trino/plugin/trino-raptor-legacy/target/surefire 2022-06-07T22-56-19_016-jvmRun1 surefire10605864679852646269tmp surefire_012643883571558364398tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 3
Error:  Crashed tests:
Error:  io.trino.plugin.raptor.legacy.TestRaptorConnectorTest
Error:  io.trino.plugin.raptor.legacy.TestRaptorMySqlConnectorTest

@findepi findepi mentioned this issue Jun 8, 2022
Closed
@findepi
Copy link
Member Author

findepi commented Jun 8, 2022

in #12730 (comment) it seems the module tests fail with roughly 40% probability (not necessarily with OOM though).

this is a serious problem from CI perspective.

cc @trinodb/maintainers @nineinchnick

@losipiuk
Copy link
Member

losipiuk commented Jun 8, 2022

Side question - is it worthy to invest development time into raptor plugin? Do we know if it is used by anyone - maybe we should just remove it?

@losipiuk losipiuk mentioned this issue Jun 8, 2022
Merged
@findepi findepi changed the title trino-raptor-legacy tests sometimes fail with OutOfMemoryError trino-raptor-legacy tests are flaky, e.g. sometimes fail with OutOfMemoryError Jun 9, 2022
@findepi
Copy link
Member Author

findepi commented Jun 9, 2022

https://github.com/trinodb/trino/runs/6801844249?check_suite_focus=true

@ksobolew ksobolew mentioned this issue Jun 9, 2022
Merged
@electrum
Copy link
Member

electrum commented Jun 9, 2022

@losipiuk Raptor is very useful for testing as it exercises the engine in ways that other connectors do not. Also, since it's standalone, it's easy to work with in integration tests. I'm utilizing it when working on the new MERGE code, for example.

@electrum electrum mentioned this issue Jun 9, 2022
Merged
@findepi
Copy link
Member Author

findepi commented Jun 10, 2022

https://github.com/trinodb/trino/runs/6832015130?check_suite_focus=true

@findepi findepi mentioned this issue Jun 10, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Set DB_CLOSE_DELAY for Raptor H2 electrum/trino
3 participants
@electrum @findepi @losipiuk