Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky: product tests fail with MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException #5427

Closed
kokosing opened this issue Oct 6, 2020 · 8 comments · Fixed by #15879
Closed

Flaky: product tests fail with MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException #5427

kokosing opened this issue Oct 6, 2020 · 8 comments · Fixed by #15879
Labels
bug Something isn't working test

Comments

@kokosing
Copy link
Member

kokosing commented Oct 6, 2020 

2020-10-06T09:45:09.4244712Z tests               | 2020-10-06 15:30:09 INFO: FAILURE     /    io.prestosql.tests.hive.TestReadUniontype.testReadUniontype (Groups: smoke) took 17.3 seconds
2020-10-06T09:45:09.4312579Z tests               | 2020-10-06 15:30:09 SEVERE: Failure cause:
2020-10-06T09:45:09.4315564Z tests               | io.prestosql.tempto.query.QueryExecutionException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException
2020-10-06T09:45:09.4318877Z tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:114)
2020-10-06T09:45:09.4321086Z tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:82)
2020-10-06T09:45:09.4323410Z tests               | 	at io.prestosql.tests.hive.TestReadUniontype.testReadUniontype(TestReadUniontype.java:99)
2020-10-06T09:45:09.4325582Z tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-10-06T09:45:09.4327713Z tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-10-06T09:45:09.4330450Z tests               | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-10-06T09:45:09.4332376Z tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2020-10-06T09:45:09.4334220Z tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2020-10-06T09:45:09.4337433Z tests               | 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
2020-10-06T09:45:09.4338721Z tests               | 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
2020-10-06T09:45:09.4344102Z tests               | 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
2020-10-06T09:45:09.4345835Z tests               | 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
2020-10-06T09:45:09.4347530Z tests               | 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
2020-10-06T09:45:09.4349163Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-10-06T09:45:09.4351083Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-10-06T09:45:09.4352196Z tests               | 	at java.base/java.lang.Thread.run(Thread.java:834)
2020-10-06T09:45:09.4354072Z tests               | Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException
2020-10-06T09:45:09.4356283Z tests               | 	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
2020-10-06T09:45:09.4358248Z tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:122)
2020-10-06T09:45:09.4360460Z tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:107)
2020-10-06T09:45:09.4361746Z tests               | 	... 15 more
2020-10-06T09:45:09.4362000Z tests               | 
2020-10-06T09:45:09.8620299Z tests

form pt (config-default, suite-1)
@kokosing kokosing added bug Something isn't working test labels Oct 6, 2020
@kokosing
Copy link
Member Author

kokosing commented Oct 6, 2020

It looks like the issue is on Hive/Hadoop side.

It happened with: #5421

@wendigo Do capture logs? How can I access them?

@kokosing kokosing mentioned this issue Oct 6, 2020
Merged
@wendigo
Copy link
Contributor

wendigo commented Oct 6, 2020 •
edited
Loading

@kokosing you can't. This requires changes in the workflow. First step is here: #5258

To upload an artifact correct github token must be added to secrets

@kokosing kokosing changed the title TestReadUniontype.testReadUniontype is flaky Flaky: product tests fail with MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException Oct 6, 2020
@kokosing
Copy link
Member Author

kokosing commented Oct 6, 2020

Another example:

| 2020-10-06 13:19:24 INFO: Testing with bucketingType=BUCKETED_V1, value='prestosql rocks', insertWithPresto=false, expectedFileNamePossibilites=[000001_0]
tests               | 2020-10-06 13:19:25 INFO: FAILURE     /    io.prestosql.tests.hive.TestHiveBucketedTables.testBucketingVersion (Groups: ) took 49.2 seconds
tests               | 2020-10-06 13:19:25 SEVERE: Failure cause:
tests               | io.prestosql.tempto.query.QueryExecutionException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:114)
tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:82)
tests               | 	at io.prestosql.tests.hive.TestHiveBucketedTables.testBucketingVersion(TestHiveBucketedTables.java:304)
tests               | 	at io.prestosql.tests.hive.TestHiveBucketedTables.testBucketingVersion(TestHiveBucketedTables.java:280)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
tests               | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
tests               | 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
tests               | 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
tests               | 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
tests               | 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:834)
tests               | Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:122)
tests               | 	at io.prestosql.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:107)
tests               | 	... 16 more

From: #5330

@rclaude rclaude mentioned this issue Oct 12, 2020
Merged
@ebyhr
Copy link
Member

ebyhr commented Oct 14, 2022

https://github.com/trinodb/trino/actions/runs/3247307809/jobs/5327306564

@ebyhr ebyhr mentioned this issue Oct 14, 2022
Closed
@findepi
Copy link
Member

findepi commented Dec 23, 2022

recently on master (https://github.com/trinodb/trino/runs/10276873480)

TestReadUniontype > testUnionTypeSchemaEvolution(0: AVRO) [groups: smoke]
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: java.nio.channels.ClosedByInterruptException

@findinpath
Copy link
Contributor

https://github.com/trinodb/trino/actions/runs/4005673824/jobs/6876545578

2023-01-25 19:24:48 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveBucketedTables.testBucketingVersion (Groups: ) took 1.1 seconds
tests               | 2023-01-25 19:24:48 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching reduce.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)

@findinpath findinpath mentioned this issue Jan 25, 2023
Merged
@findepi
Copy link
Member

findepi commented Jan 27, 2023

https://github.com/trinodb/trino/actions/runs/4022709157/jobs/6913463514

tests               | 2023-01-27 18:39:22 WARNING: not retrying; stacktrace 'io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching reduce.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.hive.TestHiveBucketedTables.populateHivePartitionedTable(TestHiveBucketedTables.java:490)
tests               | 	at io.trino.tests.product.hive.TestHiveBucketedTables.testSelectAfterMultipleInsertsForPartitionedTable(TestHiveBucketedTables.java:196)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
tests               | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
tests               | 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
tests               | 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
tests               | 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
tests               | 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:833)
tests               | Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching reduce.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:128)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 	... 16 more
tests               | 	Suppressed: java.lang.Exception: Query: INSERT INTO TABLE tempto_mut_bucketed_partitioned_nation_wgvo8xew PARTITION (part_key = 'insert_2') SELECT * FROM nation
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		... 17 more
tests               | ' does not match pattern '(could only be replicated to 0 nodes instead of minReplication|could only be written to 0 of the 1 minReplication)'
tests               | 2023-01-27 18:39:22 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveBucketedTables.testSelectAfterMultipleInsertsForPartitionedTable (Groups: ) took 27.9 seconds
tests               | 2023-01-27 18:39:22 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching reduce.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.hive.TestHiveBucketedTables.populateHivePartitionedTable(TestHiveBucketedTables.java:490)
tests               | 	at io.trino.tests.product.hive.TestHiveBucketedTables.testSelectAfterMultipleInsertsForPartitionedTable(TestHiveBucketedTables.java:196)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
tests               | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
tests               | 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
tests               | 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
tests               | 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
tests               | 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:833)
tests               | Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching reduce.xml: java.nio.channels.ClosedByInterruptException
tests               | 	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:128)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 	... 16 more
tests               | 	Suppressed: java.lang.Exception: Query: INSERT INTO TABLE tempto_mut_bucketed_partitioned_nation_wgvo8xew PARTITION (part_key = 'insert_2') SELECT * FROM nation
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		... 17 more
tests               | 
tests               | 2023-01-27 18:39:27 INFO: [451 of 483] io.trino.tests.product.hive.TestHiveBucketedTables.testSelectAfterMultipleInsertsForSortedTable (Groups: )

This was referenced Jan 27, 2023
Merged
Merged
@findepi
Copy link
Member

findepi commented Jan 27, 2023

I propose that we retry on such failures: #15879

This was referenced Sep 5, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working test
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Retry product tests on MapRedTask ClosedByInterruptException failures findepi/trino
5 participants
@wendigo @findepi @ebyhr @kokosing @findinpath