Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ACID subdirectory corruption in product test #16315

Closed
electrum opened this issue Mar 1, 2023 · 24 comments · Fixed by #20840
Closed

ACID subdirectory corruption in product test #16315

electrum opened this issue Mar 1, 2023 · 24 comments · Fixed by #20840

Comments

@electrum
Copy link
Member

electrum commented Mar 1, 2023

This read failure indicates some sort of corruption on write. It looks like the delta delete directory is written inside another delta delete directory. Maybe something is appending the delta directory to the path twice.

2023-02-27 15:47:55 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned (Groups: hive_transactional) took 1 minutes and 19 seconds
2023-02-27 15:47:55 SEVERE: Failure cause:
io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20230227_100255_00801_ghngm): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_lgl1350uyr/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:178)
	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:115)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.sql.SQLException: Query failed (#20230227_100255_00801_ghngm): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_lgl1350uyr/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:295)
	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:255)
	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1855)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:180)
	... 5 more
	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_true_NONE_lgl1350uyr ORDER BY col
		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:178)
		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:115)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:568)
		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
		... 5 more
Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_lgl1350uyr/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:679)
	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:541)
	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:391)
	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:305)
	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
	at io.trino.$gen.Trino_408_88_gc352739____20230227_094111_2.run(Unknown Source)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
@findepi
Copy link
Member

findepi commented Jun 16, 2023

https://github.com/trinodb/trino/pull/17903/checks?check_run_id=14274033199

TestHiveTransactionalTable > testReadFullAcidPartitioned [groups: hive_transactional]
java.sql.SQLException: Query failed (#20230614_202601_00961_xtnb4): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_srmpxox19l/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000

@findepi
Copy link
Member

findepi commented Jun 21, 2023

https://github.com/trinodb/trino/actions/runs/5332425529/jobs/9662369164?pr=17980

2023-06-21T11:02:39.3016196Z tests               | 2023-06-21 16:47:39 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned (Groups: hive_transactional) took 2 minutes and 4 seconds
2023-06-21T11:02:39.3039952Z tests               | 2023-06-21 16:47:39 SEVERE: Failure cause:
2023-06-21T11:02:39.3041425Z tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20230621_110239_00855_svej4): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_adbe2huj9l/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
2023-06-21T11:02:39.3042489Z tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
2023-06-21T11:02:39.3043169Z tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
2023-06-21T11:02:39.3043954Z tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
2023-06-21T11:02:39.3044646Z tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
2023-06-21T11:02:39.3045272Z tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
2023-06-21T11:02:39.3045955Z tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
2023-06-21T11:02:39.3046854Z tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
2023-06-21T11:02:39.3047413Z tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
2023-06-21T11:02:39.3047957Z tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
2023-06-21T11:02:39.3048536Z tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
2023-06-21T11:02:39.3049262Z tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:178)
2023-06-21T11:02:39.3050148Z tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:115)
2023-06-21T11:02:39.3050910Z tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-06-21T11:02:39.3051574Z tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-06-21T11:02:39.3052307Z tests               | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-06-21T11:02:39.3052926Z tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-06-21T11:02:39.3053519Z tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2023-06-21T11:02:39.3054167Z tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
2023-06-21T11:02:39.3054761Z tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
2023-06-21T11:02:39.3055350Z tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
2023-06-21T11:02:39.3055907Z tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2023-06-21T11:02:39.3056528Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-06-21T11:02:39.3057173Z tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-06-21T11:02:39.3057692Z tests               | 	at java.base/java.lang.Thread.run(Thread.java:833)
2023-06-21T11:02:39.3058800Z tests               | Caused by: java.sql.SQLException: Query failed (#20230621_110239_00855_svej4): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_adbe2huj9l/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
2023-06-21T11:02:39.3059761Z tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1943)
2023-06-21T11:02:39.3060405Z tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:295)
2023-06-21T11:02:39.3061108Z tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:255)
2023-06-21T11:02:39.3061770Z tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
2023-06-21T11:02:39.3062455Z tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
2023-06-21T11:02:39.3063079Z tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1855)
2023-06-21T11:02:39.3063791Z tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
2023-06-21T11:02:39.3064554Z tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
2023-06-21T11:02:39.3065286Z tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
2023-06-21T11:02:39.3065999Z tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
2023-06-21T11:02:39.3066611Z tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
2023-06-21T11:02:39.3067171Z tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:180)
2023-06-21T11:02:39.3067583Z tests               | 	... 5 more
2023-06-21T11:02:39.3068085Z tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_true_NONE_adbe2huj9l ORDER BY col
2023-06-21T11:02:39.3068770Z tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
2023-06-21T11:02:39.3069403Z tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
2023-06-21T11:02:39.3070028Z tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
2023-06-21T11:02:39.3070670Z tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
2023-06-21T11:02:39.3071242Z tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
2023-06-21T11:02:39.3071748Z tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
2023-06-21T11:02:39.3072320Z tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
2023-06-21T11:02:39.3072935Z tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
2023-06-21T11:02:39.3073503Z tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
2023-06-21T11:02:39.3074023Z tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
2023-06-21T11:02:39.3074596Z tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
2023-06-21T11:02:39.3075328Z tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:178)
2023-06-21T11:02:39.3076196Z tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:115)
2023-06-21T11:02:39.3076976Z tests               | 		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-06-21T11:02:39.3077642Z tests               | 		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-06-21T11:02:39.3078463Z tests               | 		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-06-21T11:02:39.3079079Z tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-06-21T11:02:39.3079775Z tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2023-06-21T11:02:39.3080418Z tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
2023-06-21T11:02:39.3081027Z tests               | 		at org.testng.internal.InvokeMethodR

@findinpath
Copy link
Contributor

https://github.com/trinodb/trino/actions/runs/5609451430/jobs/10264052043

 2023-07-20 17:43:11 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20230720_115811_01662_i5tzn): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_s1dnccizyo/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:114)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
tests               | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
tests               | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:833)

@findepi
Copy link
Member

findepi commented Jul 21, 2023

relates to #8920

@ebyhr
Copy link
Member

ebyhr commented Nov 14, 2023

@findinpath
Copy link
Contributor

https://github.com/trinodb/trino/actions/runs/6862530540/job/18661057508?pr=19731

2023-11-14 17:13:54 INFO: Test io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid took 1.19m
tests               | 2023-11-14 17:13:54 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 11 seconds
tests               | 2023-11-14 17:13:54 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231114_112853_00545_rpj79): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_br7xsjg0yo/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231114_112853_00545_rpj79): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_br7xsjg0yo/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more

@findepi
Copy link
Member

findepi commented Nov 15, 2023

https://github.com/trinodb/trino/actions/runs/6879919718/job/18713534527?pr=19756

tests               | 2023-11-15 22:26:36 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 20 seconds
tests               | 2023-11-15 22:26:36 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231115_164136_00544_4cve8): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_c9gpu07hcs/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231115_164136_00544_4cve8): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_c9gpu07hcs/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_false_NONE_c9gpu07hcs ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_c9gpu07hcs/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_68_g9324cb5e____20231115_162911_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@findepi
Copy link
Member

findepi commented Nov 15, 2023

#8920 seems to describe potential mechanics of the problem

@findepi
Copy link
Member

findepi commented Nov 16, 2023

https://github.com/trinodb/trino/actions/runs/6883239760/job/18724030554?pr=19757

| 2023-11-16 04:21:05 INFO: Test io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid took 1.24m
tests               | 2023-11-16 04:21:05 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 14 seconds
tests               | 2023-11-16 04:21:05 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231115_223605_00544_ittxi): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_jvwjquv3nb/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231115_223605_00544_ittxi): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_jvwjquv3nb/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_false_NONE_jvwjquv3nb ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_jvwjquv3nb/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_77_gb9f7d6c____20231115_222348_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@findepi
Copy link
Member

findepi commented Nov 16, 2023

also previously reported as #4927

@ebyhr
Copy link
Member

ebyhr commented Nov 17, 2023

@findepi
Copy link
Member

findepi commented Nov 21, 2023

https://github.com/trinodb/trino/actions/runs/6940970228/job/18881753183?pr=19841

ests               | 2023-11-21 15:12:17 INFO: not retrying; @Flaky annotation not present
tests               | 2023-11-21 15:12:17 INFO: Test io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid took 1.26m
tests               | 2023-11-21 15:12:17 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 15 seconds
tests               | 2023-11-21 15:12:17 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231121_092717_00854_hixaz): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_28m4oqiixg/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231121_092717_00854_hixaz): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_28m4oqiixg/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_false_NONE_28m4oqiixg ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_28m4oqiixg/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_173_gf62f4ad____20231121_091246_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@findepi
Copy link
Member

findepi commented Nov 21, 2023

https://github.com/trinodb/trino/actions/runs/6940970228/job/18881754940?pr=19841

tests               | 2023-11-21 15:11:16 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 14 seconds
tests               | 2023-11-21 15:11:16 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231121_092616_00545_kngvw): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_fretsw3vgf/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231121_092616_00545_kngvw): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_fretsw3vgf/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_false_NONE_fretsw3vgf ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_fretsw3vgf/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_173_gf62f4ad____20231121_091326_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@findepi
Copy link
Member

findepi commented Nov 22, 2023

https://github.com/trinodb/trino/actions/runs/6946693990/job/18899680760?pr=19853

tests               | 2023-11-21 23:16:47 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 17 seconds
tests               | 2023-11-21 23:16:47 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231121_173147_00854_u6z2p): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_5xbes98hic/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231121_173147_00854_u6z2p): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_5xbes98hic/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_false_NONE_5xbes98hic ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_5xbes98hic/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_179_gb0674f9____20231121_171638_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@findepi
Copy link
Member

findepi commented Nov 23, 2023

https://github.com/trinodb/trino/actions/runs/6959734449/job/18938511888?pr=19863

tests               | 2023-11-22 22:18:09 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid (Groups: hive_transactional) took 1 minutes and 13 seconds
tests               | 2023-11-22 22:18:09 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231122_163309_00545_rgecb): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_0bk6kuuxt0/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231122_163309_00545_rgecb): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_0bk6kuuxt0/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_false_NONE_0bk6kuuxt0 ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcid(TestHiveTransactionalTable.java:101)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_false_none_0bk6kuuxt0/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_199_g6300c37____20231122_162108_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@findepi
Copy link
Member

findepi commented Nov 29, 2023

https://github.com/trinodb/trino/actions/runs/7034076498/job/19142232176?pr=19953

tests               | 2023-11-29 21:05:52 INFO: FAILURE     /    io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned (Groups: hive_transactional) took 1 minutes and 14 seconds
tests               | 2023-11-29 21:05:52 SEVERE: Failure cause:
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231129_152052_00899_d62m5): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_3xviug8gwi/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)
tests               | 	at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 	at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 	at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:114)
tests               | 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)
tests               | Caused by: java.sql.SQLException: Query failed (#20231129_152052_00899_d62m5): Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_3xviug8gwi/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1937)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:294)
tests               | 	at io.trino.jdbc.TrinoResultSet$ResultsPageIterator.computeNext(TrinoResultSet.java:254)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
tests               | 	at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
tests               | 	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1949)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
tests               | 	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
tests               | 	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
tests               | 	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
tests               | 	at io.trino.jdbc.TrinoResultSet$AsyncIterator.lambda$new$1(TrinoResultSet.java:179)
tests               | 	... 5 more
tests               | 	Suppressed: java.lang.Exception: Query: SELECT col, fcol FROM test_read_full_acid_true_NONE_3xviug8gwi ORDER BY col
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQueryNoParams(JdbcQueryExecutor.java:136)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:112)
tests               | 		at io.trino.tempto.query.JdbcQueryExecutor.executeQuery(JdbcQueryExecutor.java:84)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.lambda$executeQuery$0(QueryExecutors.java:54)
tests               | 		at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
tests               | 		at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
tests               | 		at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
tests               | 		at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
tests               | 		at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
tests               | 		at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
tests               | 		at io.trino.tests.product.utils.QueryExecutors$1.executeQuery(QueryExecutors.java:54)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.doTestReadFullAcid(TestHiveTransactionalTable.java:177)
tests               | 		at io.trino.tests.product.hive.TestHiveTransactionalTable.testReadFullAcidPartitioned(TestHiveTransactionalTable.java:114)
tests               | 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
tests               | 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
tests               | 		at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
tests               | 		at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
tests               | 		at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
tests               | 		... 5 more
tests               | Caused by: io.trino.spi.TrinoException: Found file in sub-directory of ACID directory: hdfs://hadoop-master:9000/user/hive/warehouse/test_read_full_acid_true_none_3xviug8gwi/part_col=2/delete_delta_0000002_0000005/delete_delta_0000002_0000005/bucket_00000
tests               | 	at io.trino.plugin.hive.util.AcidTables.getAcidState(AcidTables.java:136)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.getTransactionalSplits(BackgroundHiveSplitLoader.java:539)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:481)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:366)
tests               | 	at io.trino.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:280)
tests               | 	at io.trino.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
tests               | 	at io.trino.$gen.Trino_433_340_g9dcaa59____20231129_150207_2.run(Unknown Source)
tests               | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:79)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
tests               | 	at java.base/java.lang.Thread.run(Thread.java:1583)

findepi added a commit to findepi/trino that referenced this issue Nov 29, 2023
Some test were marked as flaky and had retries enabled, but many others
are equally affected.

This also updates links from
trinodb#4927 to
trinodb#16315.
wendigo pushed a commit that referenced this issue Nov 29, 2023
Some test were marked as flaky and had retries enabled, but many others
are equally affected.

This also updates links from
#4927 to
#16315.
shreyas-dview pushed a commit to dview-io/trino that referenced this issue Dec 12, 2023
Some test were marked as flaky and had retries enabled, but many others
are equally affected.

This also updates links from
trinodb#4927 to
trinodb#16315.
@findepi
Copy link
Member

findepi commented Dec 15, 2023

@findepi
Copy link
Member

findepi commented Dec 21, 2023

@findepi
Copy link
Member

findepi commented Jan 11, 2024

@findepi
Copy link
Member

findepi commented Jan 12, 2024

@findepi
Copy link
Member

findepi commented Jan 31, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

5 participants