Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revisit all TODO in AbstractKuduConnectorTest #11815

Closed
ebyhr opened this issue Apr 6, 2022 · 6 comments · Fixed by #12995
Closed

Revisit all TODO in AbstractKuduConnectorTest #11815

ebyhr opened this issue Apr 6, 2022 · 6 comments · Fixed by #12995
Assignees
Labels
good first issue Good for newcomers test

Comments

@ebyhr
Copy link
Member

ebyhr commented Apr 6, 2022

There're more than 10 skipped tests in Kudu with below comment, but it would be better to add tests in connector specific way. Otherwise, it's hard to detect regression likes #11814. After supporting default partitions, we can just remove overridden tests.

        // TODO Support these test once kudu connector can create tables with default partitions
        throw new SkipException("TODO");
@ebyhr ebyhr added good first issue Good for newcomers test labels Apr 6, 2022
@chen-ni
Copy link
Contributor

chen-ni commented Jun 14, 2022

Is anybody working on this? If not can I take it?

@ebyhr
Copy link
Member Author

ebyhr commented Jun 14, 2022

@chen-ni No one is working on this as far as I know. Please go for it.

@chen-ni
Copy link
Contributor

chen-ni commented Jun 15, 2022

@ebyhr OK, thanks!

@chen-ni
Copy link
Contributor

chen-ni commented Jun 15, 2022

@ebyhr Please correct me if I'm wrong:

  • AbstractKuduConnectorTest extends BaseConnectorTest, so by default it will run all the tests defined in BaseConnectorTest. However, some tests require support for default partitioning, but Kudu does not support it yet. If we run those tests, they fail;
  • So we ignored those tests by overriding them with empty ones. As a consequence, we failed to catch bugs that those tests could have caught;
  • What I need to do is, instead of ignoring those tests, override them with tests that specify partitioning. They should have the same testing scope as in BaseConnectorTest;
  • After the Kudu connector has support for default partitioning, the overriding tests I'm going to write will be removed. We will be able to execute all the tests from BaseConnectorTest;

@chen-ni
Copy link
Contributor

chen-ni commented Jun 21, 2022

@ebyhr I'm working on the testInsert() test now and I'm very confused.

  • My first question is, what happens to default partitioning in Kudu connector when we create table with CREATE ... AS ...?

On one hand, we can indeed create table without specifying default partitions in this way, like: https://github.com/trinodb/trino/blob/master/plugin/trino-kudu/src/test/java/io/trino/plugin/kudu/AbstractKuduConnectorTest.java#L260

// AbstractKuduConnectorTest.java
@Test
@Override
public void testExplainAnalyzeWithDeleteWithSubquery()
{
    String tableName = "test_delete_" + randomTableSuffix();

    // note that default partitions are not specified here
    assertUpdate("CREATE TABLE " + tableName + " AS SELECT * FROM nation", 25);
    assertExplainAnalyze("EXPLAIN ANALYZE DELETE FROM " + tableName + " WHERE regionkey IN (SELECT regionkey FROM region WHERE name LIKE 'A%' LIMIT 1)",
            "SemiJoin.*");
    assertUpdate("DROP TABLE " + tableName);
}

On the other hand, the testInsert test is skipped just because of default partitions: https://github.com/trinodb/trino/blob/master/plugin/trino-kudu/src/test/java/io/trino/plugin/kudu/AbstractKuduConnectorTest.java#L330

// AbstractKuduConnectorTest.java
@Override
public void testInsert()
{
    // TODO Support these test once kudu connector can create tables with default partitions
    throw new SkipException("TODO");
}

But the testInsert test in the parent class is actually creating table in the same CREATE ... AS ... fashion: https://github.com/trinodb/trino/blob/master/testing/trino-testing/src/main/java/io/trino/testing/BaseConnectorTest.java#L2404

// BaseConnectorTest.java
@Test
public void testInsert()
{
   // ...

    String query = "SELECT phone, custkey, acctbal FROM customer";

    try (TestTable table = new TestTable(getQueryRunner()::execute, "test_insert_", "AS " + query + " WITH NO DATA")) {
    // ...
    }
}

So it doesn't make sense to me to skip the test because of "default partitions". I've also tried explicitly specifying default partitions while executing CREATE ... AS ... but failed. I didn't find any documentation about it either.

The test does fail, though. If I remove the overriding test and run the testInsert test for a Kudu connector, I got this error:

java.lang.AssertionError: Execution of 'actual' query failed: INSERT INTO test_insert_1gc20vgmo0 SELECT phone, custkey, acctbal FROM customer

	at org.testng.Assert.fail(Assert.java:83)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:150)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:106)
	at io.trino.testing.AbstractTestQueryFramework.assertUpdate(AbstractTestQueryFramework.java:303)
	at io.trino.testing.AbstractTestQueryFramework.assertUpdate(AbstractTestQueryFramework.java:298)
	at io.trino.testing.BaseConnectorTest.testInsert(BaseConnectorTest.java:2407)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at org.testng.TestRunner.privateRun(TestRunner.java:756)
	at org.testng.TestRunner.run(TestRunner.java:610)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
	at org.testng.TestNG.runSuites(TestNG.java:1133)
	at org.testng.TestNG.run(TestNG.java:1104)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
Caused by: java.lang.RuntimeException: java.lang.UnsupportedOperationException
	at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:122)
	at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:526)
	at io.trino.testing.QueryAssertions.assertQuery(QueryAssertions.java:147)
	... 28 more
	Suppressed: java.lang.Exception: SQL: INSERT INTO test_insert_1gc20vgmo0 SELECT phone, custkey, acctbal FROM customer
		at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:529)
		... 29 more
Caused by: java.lang.UnsupportedOperationException
	at io.trino.spi.block.Block.getSliceLength(Block.java:34)
	at io.trino.spi.type.VarcharType.getSlice(VarcharType.java:188)
	at io.trino.plugin.kudu.KuduPageSink.appendColumn(KuduPageSink.java:175)
	at io.trino.plugin.kudu.KuduPageSink.appendPage(KuduPageSink.java:123)
	at io.trino.operator.TableWriterOperator.addInput(TableWriterOperator.java:270)
	at io.trino.operator.Driver.processInternal(Driver.java:415)
	at io.trino.operator.Driver.lambda$process$10(Driver.java:313)
	at io.trino.operator.Driver.tryWithLock(Driver.java:698)
	at io.trino.operator.Driver.process(Driver.java:305)
	at io.trino.operator.Driver.processForDuration(Driver.java:276)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1092)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
	at io.trino.$gen.Trino_testversion____20220621_001652_1.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Which brings to my second question:

  • Is this a known issue? If so how to get a workaround? Should I implement the test without using CREATE ... AS ...?

@ebyhr
Copy link
Member Author

ebyhr commented Jun 21, 2022

what happens to default partitioning in Kudu connector when we create table with CREATE ... AS ...?

The connector creates a hidden row_uuid column as a partition column. SeeKuduMetadata#beginCreateTable for more details.

Is this a known issue? If so how to get a workaround? Should I implement the test without using CREATE ... AS ...?

It's a bug. #12915 will fix the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers test
Development

Successfully merging a pull request may close this issue.

2 participants