testStatsBasedRepartitionDataOnInsert flaky #11140

losipiuk · 2022-02-22T08:34:47Z

[ERROR] io.trino.plugin.iceberg.TestIcebergOrcConnectorTest.testStatsBasedRepartitionDataOnInsert  Time elapsed: 1.576 s  <<< FAILURE!
java.lang.AssertionError: 
[Rows for query [SELECT count(*) FROM "repartition_SELECTDISTINCTorderkeycustkeyorderstatusFROMtpchtinyordersinsert_orderstatus_1ns2vmr9lb$files"]] 
Expecting:
  <(3)>
to contain exactly in any order:
  <[(9)]>
elements not found:
  <(9)>
and elements not expected:
  <(3)>

	at io.trino.sql.query.QueryAssertions$QueryAssert.lambda$matches$2(QueryAssertions.java:346)
	at io.trino.sql.query.QueryAssertions$QueryAssert.matches(QueryAssertions.java:333)
	at io.trino.sql.query.QueryAssertions$QueryAssert.matches(QueryAssertions.java:328)
	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testRepartitionData(BaseIcebergConnectorTest.java:2874)
	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.lambda$testStatsBasedRepartitionData$27(BaseIcebergConnectorTest.java:2831)
	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:74)
	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:66)
	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:61)
	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testStatsBasedRepartitionData(BaseIcebergConnectorTest.java:2830)
	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testStatsBasedRepartitionDataOnInsert(BaseIcebergConnectorTest.java:2809)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

https://github.com/trinodb/trino/runs/5280874831?check_suite_focus=true

The text was updated successfully, but these errors were encountered:

fengguangyuan · 2022-02-23T11:02:19Z

Is this an occasional failure? I could not reproduce it on Master branch. :(

ebyhr · 2022-02-23T11:54:08Z

@fengguangyuan This is an occasional failure.

fengguangyuan · 2022-03-09T09:18:32Z

After a lot spare time on this issue, I think the root cause of the failure is being clear.

Basic test info:
    Table:
         name:          tpch.tiny.orders
         files:         3
         part key:      orderstatus
         part values:   ['F', 'O', 'P']
    Cluster:
         workers: 3

    Expected files count:  9

Conclusion

Imbalanced distribution of data pages on 3 output buffer slots in TaskOutputOperator leads to the unexpected number of written files, therefore the number of files could be in range [3, 9].

Details

First of all, we know that INSERT INTO statement will produce a physical plan, which is roughly described as:
scan stage(scan -> hash -> output) -> exchange stage(exchange -> exchange sink) -> write stage

At the scan stage, the last operator TaskOutputOperator is responsible for putting data pages into it's internal buffer, an instance of ArbitraryOutputBuffer which contains 3 ClientBuffers, and then waiting to be consumed by the upstream tasks in parallel, here these tasks are from exchange stage.

ArbitraryOutputBuffer has 3 output client buffers, implying 3 parallelism, because there are three simulated worker nodes created by the test framework.

Ideally, TaskOutputOperator should keep even distribution of all the data pages on 3 buffer channels, for example if there were 9 pages, each buffer should own 3.

But actually, considering all the data pages are not received by ArbitraryOutputBuffer once, ArbitraryOutputBuffer using an improved round-robin strategy to locate buffers for each enqueue. The code below describes the strategy and is what I used to test:

public class ArbitraryOutputBuffer
        implements OutputBuffer
{
    @GuardedBy("this")
    private OutputBuffers outputBuffers = createInitialEmptyOutputBuffers(ARBITRARY);

    // all the pages should be put into this buffer at first, then move to `buffers`
    private final MasterBuffer masterBuffer;

    // the size is determined by the upstream task, here it's 3 corresponding to 3 worker nodes
    @GuardedBy("this")
    private final ConcurrentMap<OutputBufferId, ClientBuffer> buffers = new ConcurrentHashMap<>();

    //  The index of the first client buffer that should be polled
    private final AtomicInteger nextClientBufferIndex = new AtomicInteger(0);

    @Override
    public void enqueue(List<Slice> pages)
    {
        ...
        /* Test code start */
        long beforeCnt = masterBuffer.getBufferedPages();
        /* Test code end */

        // add pages to the buffer (this will increase the reference count by one)
        masterBuffer.addPages(serializedPageReferences);
        
        /* Test code start */
        System.out.printf("ThreadId %s: page size: %s, before master buffer size %s, after master buffer size %s,\n",
                Thread.currentThread().getName(), bytesAdded, beforeCnt, masterBuffer.getBufferedPages());
        /* Test code end */

        // process any pending reads from the client buffers
        List<ClientBuffer> buffers = safeGetBuffersSnapshot();
        if (buffers.isEmpty()) {
            return;
        }
        // handle potential for racy update of next index and client buffers present
        int index = nextClientBufferIndex.get() % buffers.size();
        for (int i = 0; i < buffers.size(); i++) {
            // buffers.get(index).loadPagesIfNecessary(masterBuffer);
            
            /* Test code start */
            ClientBuffer clientBuffer = buffers.get(index);
            clientBuffer.loadPagesIfNecessary(masterBuffer);
            BufferInfo bufferInfo = clientBuffer.getInfo();
            System.out.printf("ThreadId %s: %s now buffered total %s pages on bufferId %s with destroyed %s!\n",
                    Thread.currentThread().getName() + "-" + Thread.currentThread().getId(),
                    this.getClass().getName(),
                    bufferInfo.getBufferedPages(),
                    bufferInfo.getBufferId(),
                    clientBuffer.isDestroyed());
            /* Test code end */

            index = (index + 1) % buffers.size();

            if (masterBuffer.isEmpty()) {
                // Resume from the next client buffer on the next iteration
                nextClientBufferIndex.set(index);
                break;
            }
        }
    }
}

Below logs are from task 20220308_114754_00170_7cd8a.2.1.0 at scan stage, we could see that buffer-1 received the most pages, buffer-2 received fewer pages, while buffer-0 received 0. With no doubts, all the tasks consumed pages from buffer-0 will produce nothing, that is the exchange task reading buffer-0 sink nothing, and finally the write task reading sinked data partition 0 wrote nothing.

This tells the reason why the test could occasionally failed on mismatched files count. In this case, total 6 files were produced by two workers, rather than expected 9.

ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: page size: 5501, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: page size: 5522, before master buffer size 2, after master buffer size 3,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: page size: 2540, before master buffer size 3, after master buffer size 4,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-3-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: page size: 5501, before master buffer size 4, after master buffer size 5,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: page size: 5501, before master buffer size 1, after master buffer size 2,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: page size: 5522, before master buffer size 6, after master buffer size 7,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: page size: 5501, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: page size: 2540, before master buffer size 7, after master buffer size 8,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: page size: 5522, before master buffer size 5, after master buffer size 6,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-0-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 8 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: page size: 5522, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: page size: 2540, before master buffer size 1, after master buffer size 2,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-1-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 8 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 8 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: page size: 2540, before master buffer size 2, after master buffer size 3,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-2-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 8 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: page size: 5501, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 3 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: page size: 5522, before master buffer size 1, after master buffer size 2,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: page size: 5501, before master buffer size 2, after master buffer size 3,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 2 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: page size: 5522, before master buffer size 3, after master buffer size 4,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 2 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: page size: 2540, before master buffer size 4, after master buffer size 5,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-6-66: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: page size: 5501, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: page size: 5501, before master buffer size 1, after master buffer size 2,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: page size: 2540, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-4-63: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: page size: 5522, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: page size: 2540, before master buffer size 1, after master buffer size 2,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-7-64: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: page size: 5522, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 2 pages on bufferId 2 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: page size: 2540, before master buffer size 0, after master buffer size 1,
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 0 pages on bufferId 0 with destroyed false!
ThreadId 20220308_114754_00170_7cd8a.2.1.0-5-65: io.trino.execution.buffer.ArbitraryOutputBuffer now buffered total 1 pages on bufferId 1 with destroyed false!

Discussion

Imbalanced distribution of pages by ArbitraryOutputBuffer is acceptable?
Imbalanced distribution will decrease the overall performance?
It's better to use a more heuristic method to determine the output buffer id, such as the client buffer size, the upstream workers QPS or ... ?

I hope the above explanations would not make you puzzled. :)

fengguangyuan · 2022-03-09T09:18:50Z

@ebyhr @losipiuk

findepi · 2022-03-09T11:09:10Z

This tells the reason why the test could occasionally failed on mismatched files count. In this case, total 6 files were produced by two workers, rather than expected 9.

Correct. That's why the test uses assertEventually.
Apparently that's sometimes not enough.

findepi · 2022-05-30T13:32:00Z

https://github.com/trinodb/trino/runs/6651751956?check_suite_focus=true

findepi · 2022-06-20T08:12:32Z

https://github.com/trinodb/trino/runs/6899758149?check_suite_focus=true

findepi · 2022-06-28T11:32:06Z

https://github.com/trinodb/trino/runs/7089990501?check_suite_focus=true

Error:    TestIcebergParquetConnectorTest>BaseIcebergConnectorTest.testStatsBasedRepartitionDataOnInsert:3173->BaseIcebergConnectorTest.testStatsBasedRepartitionData:3195->BaseIcebergConnectorTest.lambda$testStatsBasedRepartitionData$15:3196->BaseIcebergConnectorTest.testRepartitionData:3239 [Rows for query [SELECT count(*) FROM "repartition_SELECTDISTINCTorderkeycustkeyorderstatusFROMtpchtinyordersinsert_orderstatus_ybf5539lvf$files"]] 
Expecting:
  <(6)>
to contain exactly in any order:
  <[(9)]>
elements not found:
  <(9)>
and elements not expected:
  <(6)>

findepi · 2022-09-27T17:12:26Z

https://github.com/trinodb/trino/actions/runs/3135264357/jobs/5091989947

findinpath · 2022-10-01T07:15:06Z

https://github.com/trinodb/trino/actions/runs/3160325124/jobs/5146309904

findinpath · 2022-10-02T07:20:10Z

https://github.com/trinodb/trino/actions/runs/3163547761/jobs/5151527253

findepi · 2023-05-08T19:17:07Z

@wendigo any comment why this is reopened? thanks

wendigo · 2023-05-08T19:36:20Z

I've linked a new occurance

findepi · 2023-05-08T19:38:02Z

i didn't see any comment here, hence the question.

wendigo · 2023-05-08T19:50:20Z

@findepi https://github.com/trinodb/trino/actions/runs/4914724739/jobs/8778902522

wendigo · 2023-05-08T19:50:33Z

@findepi https://github.com/trinodb/trino/actions/runs/4914724739/jobs/8778902522

wendigo · 2023-05-08T19:51:24Z

@findepi https://github.com/trinodb/trino/actions/runs/4914724739/jobs/8778902522

findepi · 2023-05-09T14:56:44Z

https://github.com/trinodb/trino/actions/runs/4925370678/jobs/8800359681?pr=17383

2023-05-09T14:45:50.7113079Z [ERROR] Tests run: 1556, Failures: 1, Errors: 0, Skipped: 31, Time elapsed: 1,855.731 s <<< FAILURE! - in TestSuite
2023-05-09T14:45:50.7114300Z [ERROR] io.trino.plugin.iceberg.TestIcebergMinioOrcConnectorTest.testStatsBasedRepartitionDataOnInsert  Time elapsed: 348.707 s  <<< FAILURE!
2023-05-09T14:45:50.7115313Z org.assertj.core.error.MultipleAssertionsError: 
2023-05-09T14:45:50.7115625Z 
2023-05-09T14:45:50.7115803Z The following assertion failed:
2023-05-09T14:45:50.7116544Z 1) [Rows for query [SELECT count(*) FROM "repartition_SELECTDISTINCTorderkeycustkeyorderstatusFROMtpchtinyordersinsert_orderstatus_qsytnlfkk0$files"]] 
2023-05-09T14:45:50.7117203Z Expecting actual:
2023-05-09T14:45:50.7117537Z   (6)
2023-05-09T14:45:50.7117890Z to contain exactly in any order:
2023-05-09T14:45:50.7118241Z   [(9)]
2023-05-09T14:45:50.7118563Z elements not found:
2023-05-09T14:45:50.7118895Z   (9)
2023-05-09T14:45:50.7119231Z and elements not expected:
2023-05-09T14:45:50.7119577Z   (6)
2023-05-09T14:45:50.7119751Z 
2023-05-09T14:45:50.7119758Z 
2023-05-09T14:45:50.7120192Z 	at org.assertj.core.error.AssertionErrorCreator.multipleAssertionsError(AssertionErrorCreator.java:106)
2023-05-09T14:45:50.7121100Z 	at org.assertj.core.api.AbstractAssert.multipleAssertionsError(AbstractAssert.java:979)
2023-05-09T14:45:50.7121844Z 	at org.assertj.core.api.AbstractAssert.satisfiesForProxy(AbstractAssert.java:875)
2023-05-09T14:45:50.7122512Z 	at org.assertj.core.api.AbstractAssert.satisfies(AbstractAssert.java:862)
2023-05-09T14:45:50.7123271Z 	at io.trino.sql.query.QueryAssertions$QueryAssert.matches(QueryAssertions.java:383)
2023-05-09T14:45:50.7124013Z 	at io.trino.sql.query.QueryAssertions$QueryAssert.matches(QueryAssertions.java:377)
2023-05-09T14:45:50.7124766Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testRepartitionData(BaseIcebergConnectorTest.java:4637)
2023-05-09T14:45:50.7125663Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.lambda$testStatsBasedRepartitionData$23(BaseIcebergConnectorTest.java:4594)
2023-05-09T14:45:50.7126418Z 	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:43)
2023-05-09T14:45:50.7127046Z 	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:34)
2023-05-09T14:45:50.7127954Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testStatsBasedRepartitionData(BaseIcebergConnectorTest.java:4593)
2023-05-09T14:45:50.7128970Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testStatsBasedRepartitionDataOnInsert(BaseIcebergConnectorTest.java:4574)
2023-05-09T14:45:50.7129911Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-05-09T14:45:50.7130667Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-05-09T14:45:50.7131479Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-05-09T14:45:50.7132156Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-05-09T14:45:50.7132955Z 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2023-05-09T14:45:50.7133638Z 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
2023-05-09T14:45:50.7134292Z 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
2023-05-09T14:45:50.7134905Z 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
2023-05-09T14:45:50.7135568Z 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
2023-05-09T14:45:50.7136221Z 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
2023-05-09T14:45:50.7136986Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-05-09T14:45:50.7137808Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-05-09T14:45:50.7138371Z 	at java.base/java.lang.Thread.run(Thread.java:833)

krvikash · 2023-05-09T19:27:08Z

https://github.com/trinodb/trino/actions/runs/4926424212/jobs/8805991023?pr=17409

2023-05-09T19:07:40.0167702Z [ERROR] Tests run: 1558, Failures: 1, Errors: 0, Skipped: 31, Time elapsed: 2,078.153 s <<< FAILURE! - in TestSuite
2023-05-09T19:07:40.0169427Z [ERROR] io.trino.plugin.iceberg.TestIcebergAvroConnectorTest.testStatsBasedRepartitionDataOnInsert  Time elapsed: 0.019 s  <<< FAILURE!
2023-05-09T19:07:40.0170346Z org.assertj.core.error.MultipleAssertionsError: 
2023-05-09T19:07:40.0170834Z 
2023-05-09T19:07:40.0171079Z The following assertion failed:
2023-05-09T19:07:40.0172307Z 1) [Rows for query [SELECT count(*) FROM "repartition_SELECTDISTINCTorderkeycustkeyorderstatusFROMtpchtinyordersinsert_orderstatus_bal0f49dd6$files"]] 
2023-05-09T19:07:40.0172970Z Expecting actual:
2023-05-09T19:07:40.0173280Z   (3)
2023-05-09T19:07:40.0173636Z to contain exactly in any order:
2023-05-09T19:07:40.0173993Z   [(9)]
2023-05-09T19:07:40.0174318Z elements not found:
2023-05-09T19:07:40.0174639Z   (9)
2023-05-09T19:07:40.0174977Z and elements not expected:
2023-05-09T19:07:40.0175299Z   (3)
2023-05-09T19:07:40.0175516Z 
2023-05-09T19:07:40.0175615Z 
2023-05-09T19:07:40.0176128Z 	at org.assertj.core.error.AssertionErrorCreator.multipleAssertionsError(AssertionErrorCreator.java:106)
2023-05-09T19:07:40.0177046Z 	at org.assertj.core.api.AbstractAssert.multipleAssertionsError(AbstractAssert.java:979)
2023-05-09T19:07:40.0177932Z 	at org.assertj.core.api.AbstractAssert.satisfiesForProxy(AbstractAssert.java:875)
2023-05-09T19:07:40.0178770Z 	at org.assertj.core.api.AbstractAssert.satisfies(AbstractAssert.java:862)
2023-05-09T19:07:40.0179463Z 	at io.trino.sql.query.QueryAssertions$QueryAssert.matches(QueryAssertions.java:383)
2023-05-09T19:07:40.0180141Z 	at io.trino.sql.query.QueryAssertions$QueryAssert.matches(QueryAssertions.java:377)
2023-05-09T19:07:40.0181071Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testRepartitionData(BaseIcebergConnectorTest.java:4637)
2023-05-09T19:07:40.0182040Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.lambda$testStatsBasedRepartitionData$23(BaseIcebergConnectorTest.java:4594)
2023-05-09T19:07:40.0182850Z 	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:43)
2023-05-09T19:07:40.0183527Z 	at io.trino.testing.assertions.Assert.assertEventually(Assert.java:34)
2023-05-09T19:07:40.0184390Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testStatsBasedRepartitionData(BaseIcebergConnectorTest.java:4593)
2023-05-09T19:07:40.0185950Z 	at io.trino.plugin.iceberg.BaseIcebergConnectorTest.testStatsBasedRepartitionDataOnInsert(BaseIcebergConnectorTest.java:4574)
2023-05-09T19:07:40.0186889Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-05-09T19:07:40.0187674Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-05-09T19:07:40.0188664Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-05-09T19:07:40.0189393Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-05-09T19:07:40.0190087Z 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
2023-05-09T19:07:40.0190805Z 	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
2023-05-09T19:07:40.0191408Z 	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
2023-05-09T19:07:40.0192711Z 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
2023-05-09T19:07:40.0198369Z 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
2023-05-09T19:07:40.0199050Z 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
2023-05-09T19:07:40.0199697Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-05-09T19:07:40.0200413Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-05-09T19:07:40.0200926Z 	at java.base/java.lang.Thread.run(Thread.java:833)

findepi · 2023-05-10T07:38:49Z

https://github.com/trinodb/trino/actions/runs/4927265886/jobs/8807331960?pr=17383

2023-05-09T22:35:32.9270291Z [ERROR] Tests run: 1556, Failures: 1, Errors: 0, Skipped: 31, Time elapsed: 2,123.722 s <<< FAILURE! - in TestSuite
2023-05-09T22:35:32.9273524Z [ERROR] io.trino.plugin.iceberg.TestIcebergMinioOrcConnectorTest.testStatsBasedRepartitionDataOnInsert  Time elapsed: 14.599 s  <<< FAILURE!
2023-05-09T22:35:32.9274503Z org.assertj.core.error.MultipleAssertionsError: 
2023-05-09T22:35:32.9274866Z 
2023-05-09T22:35:32.9275082Z The following assertion failed:
2023-05-09T22:35:32.9275910Z 1) [Rows for query [SELECT count(*) FROM "repartition_SELECTDISTINCTorderkeycustkeyorderstatusFROMtpchtinyordersinsert_orderstatus_j4vhyeg40j$files"]] 
2023-05-09T22:35:32.9276660Z Expecting actual:
2023-05-09T22:35:32.9277177Z   (6)
2023-05-09T22:35:32.9277617Z to contain exactly in any order:
2023-05-09T22:35:32.9278052Z   [(9)]
2023-05-09T22:35:32.9278456Z elements not found:
2023-05-09T22:35:32.9278862Z   (9)
2023-05-09T22:35:32.9279294Z and elements not expected:
2023-05-09T22:35:32.9279688Z   (6)

losipiuk added the test label Feb 22, 2022

losipiuk mentioned this issue Feb 22, 2022

Released memory retained by ExchangeSource #11097

Merged

hashhar mentioned this issue Jun 16, 2022

Fix ignoring trailing content in JsonUtils #12862

Merged

findepi mentioned this issue Jun 28, 2022

Fix method names of casts to/from varchar #13010

Merged

This was referenced Sep 27, 2022

Migrate Hive stats definition to use standard aggregations #14294

Merged

TestIcebergParquetConnectorTest.testStatsBasedRepartitionDataOnCtas flaky #14368

Closed

findepi added the bug Something isn't working label Sep 29, 2022

ebyhr self-assigned this Sep 29, 2022

findinpath mentioned this issue Oct 2, 2022

Perform range optimization for BETWEEN predicate on date_trunc and temporal casts #14390

Closed

ebyhr mentioned this issue Oct 3, 2022

Increase timeout for testStatsBasedRepartitionData in Iceberg #14397

Merged

3 tasks

ebyhr closed this as completed in #14397 Oct 3, 2022

findepi mentioned this issue Oct 3, 2022

Ensure Glue configuration isn't bound in Iceberg when Glue not in use #14403

Merged

wendigo reopened this May 8, 2023

wendigo mentioned this issue May 8, 2023

Update various dependencies (both libraries and plugins) #17384

Merged

findepi mentioned this issue May 9, 2023

Handle missing MV StorageDescriptor in Glue #17383

Merged

findepi mentioned this issue May 10, 2023

Use bigger data set to address testStatsBasedRepartitionData flakiness #17431

Merged

findepi closed this as completed in #17431 May 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testStatsBasedRepartitionDataOnInsert flaky #11140

testStatsBasedRepartitionDataOnInsert flaky #11140

losipiuk commented Feb 22, 2022 •

edited by findepi

Loading

fengguangyuan commented Feb 23, 2022 •

edited

Loading

ebyhr commented Feb 23, 2022

fengguangyuan commented Mar 9, 2022

fengguangyuan commented Mar 9, 2022

findepi commented Mar 9, 2022

findepi commented May 30, 2022

findepi commented Jun 20, 2022

findepi commented Jun 28, 2022

findepi commented Sep 27, 2022

findinpath commented Oct 1, 2022 •

edited

Loading

findinpath commented Oct 2, 2022

findepi commented May 8, 2023

wendigo commented May 8, 2023

findepi commented May 8, 2023

wendigo commented May 8, 2023

wendigo commented May 8, 2023

wendigo commented May 8, 2023

findepi commented May 9, 2023

krvikash commented May 9, 2023

findepi commented May 10, 2023

testStatsBasedRepartitionDataOnInsert flaky #11140

testStatsBasedRepartitionDataOnInsert flaky #11140

Comments

losipiuk commented Feb 22, 2022 • edited by findepi Loading

fengguangyuan commented Feb 23, 2022 • edited Loading

ebyhr commented Feb 23, 2022

fengguangyuan commented Mar 9, 2022

Conclusion

Details

Discussion

fengguangyuan commented Mar 9, 2022

findepi commented Mar 9, 2022

findepi commented May 30, 2022

findepi commented Jun 20, 2022

findepi commented Jun 28, 2022

findepi commented Sep 27, 2022

findinpath commented Oct 1, 2022 • edited Loading

findinpath commented Oct 2, 2022

findepi commented May 8, 2023

wendigo commented May 8, 2023

findepi commented May 8, 2023

wendigo commented May 8, 2023

wendigo commented May 8, 2023

wendigo commented May 8, 2023

findepi commented May 9, 2023

krvikash commented May 9, 2023

findepi commented May 10, 2023

losipiuk commented Feb 22, 2022 •

edited by findepi

Loading

fengguangyuan commented Feb 23, 2022 •

edited

Loading

findinpath commented Oct 1, 2022 •

edited

Loading