Add Delta Lake product tests #11565

findinpath · 2022-03-18T11:48:22Z

Description

Expose Delta Lake product tests

TODOs :

Figure out how to setup Databricks environment to be used the Delta Lake connector tests.
Open issues:

Databricks spark-jdbc42 is not available on Maven Central so it can't be directly referenced as a dependency in Maven for product tests (needed to connect to Databricks cluster)
Databricks Community Edition has a few limitations for which need to be found appropriate workarounds:
- no auto-restart when querying a terminated cluster - can be solved by creating a new cluster via Databricks Clusters API v2
- no Instance profiles functionality available on community clusters - this is a serious limitation because the Delta Lake connector tests create tables backed by AWS S3 buckets
- per Community account only one cluster can be created - can be solved by creating multiple accounts in order to test Databricks 9.1 LTS, 7.3 LTS

Is this change a fix, improvement, new feature, refactoring, or other?

Tests

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Delta Lake connector

How would you describe this change to a non-technical end user or system administrator?

This change contributes to ensuring accuracy of the functionality exposed by the Delta Lake connector.

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

...-product-tests/src/main/java/io/trino/tests/product/deltalake/TestOssDeltaLakeHdfsReads.java

alexjo2144 · 2022-04-28T20:06:17Z

Does Minio need it's own module, or can it go into one of the existing shared ones, like trino-testing?

findinpath · 2022-04-29T04:39:32Z

Does Minio need it's own module, or can it go into one of the existing shared ones, like trino-testing?

trino-testing is about testing infrastructure with a focus on test utilities.
Ideally would sit the MinioClient somewhere outside of Trino.
I haven't found (so far) a better solution for where to place this client class.

testing/trino-testing-minio/pom.xml

plugin/trino-delta-lake/pom.xml

findinpath · 2022-05-14T05:28:42Z

There's file rename you can take out testing/trino-product-tests-launcher/src/main/resources/docker/presto-product-tests/conf/environment/singlenode-delta-lake-oss/minio-files/data/presto-ci-test/dummy

@alexjo2144 Adressed the issue by creating the minio bucket directory on the fly while configuring MinIO via testcontainers.

findinpath · 2022-05-14T05:30:12Z

This branch has conflicts that must be resolved

I am currently intentionally not fixing the Git conflicts in order to keep the basis of the current PR which is used in a private downstream project.

alexjo2144 · 2022-05-16T16:28:19Z

...src/main/java/io/trino/tests/product/launcher/env/environment/EnvSinglenodeDeltaLakeOss.java

+            throw new UncheckedIOException(e);
+        }
+        builder.configureContainer(MINIO_CONTAINER_NAME, container ->
+                container.withCopyFileToContainer(forHostPath(minioBucketDirectory), "/data/" + S3_BUCKET_NAME));


This should go in the common/Minio file. It's not clear here what's special about the /data/ directory

isn't /data where MinIO docker container stores buckets?

/data is the argument given to minio server to point the directory where the buckets are physically stored.

The Docker container process is started with:

.withCommand("server", "--address", format("0.0.0.0:%d", MINIO_PORT), "/data")

I think that this customisation is specific for EnvSinglenodeDeltaLakeOss and not a common setting for MinioContainer.

We should at least have /data/ stored in a static class variable on Minio then

let's follow up on this

findinpath · 2022-05-16T21:12:58Z

I am currently intentionally not fixing the Git conflicts in order to keep the basis of the current PR which is used in a private downstream project.

Change of plan. Fixing the conflicts.

findepi · 2022-05-17T09:51:36Z

@findepi what do you think about taking out all the test resources once we have the Databricks compatibility env set up?

Seems like once we have that we can create all the table's we need on the fly

Some of the ones in the integration test suite do test specific edge cases, but the ones here are just copies of TPCH

Good question.
There are some advantages of static test resources

support tests that can be run directly from an IDE, by a contributor.
ensure we support tables created by an old version of Dabaricks runtime, not only the one we're currently testing against. While new data needs to be read only by modern software version, data created by old software version needs to remain readable.

Allow the Delta Lake product tests to make use of the Delta Lake testing resources.

findinpath · 2022-05-19T18:59:02Z

Rebased on master.

findepi · 2022-05-19T20:14:26Z

@alexjo2144 ptal

The Delta Lake product tests can be all executed with SuiteDeltaLake suite class. The following test product test environments are exposed: - single-node-delta-lake-oss: used to test the compatibility of the Trino Delta Lake connector with Apache Spark with Delta OSS - single-node-delta-lake-databricks: used to test the compatibility of the Trino Delta Lake connector with Delta Lake Databricks - single-node-delta-lake-kerberized-hdfs: used to test Delta Lake connector on top of kerberized Hadoop environment - single-node-minio-data-lake: lightweight environment that can be used to test the Lakehouse connectors with HMS & MinIO The aim of the Delta Lake product tests is to ensure compatibility with both implementations of Delta Lake: - Delta OSS - Databricks Delta These product tests were originally written for the Starburst Enterprise Delta Lake connector. Co-authored by various engineers at Starburst Data: Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Alex Jo <alex.jo@starburstdata.com> Co-authored-by: Łukasz Osipiuk <lukasz@osipiuk.net> Co-authored-by: Konrad Dziedzic <konraddziedzic@gmail.com> Co-authored-by: Adam J. Shook <shook@datacatessen.com> Co-authored-by: Mateusz Gajewski <mateusz.gajewski@gmail.com> Co-authored-by: Gaurav Sehgal <gaurav.sehgal8297@gmail.com> Co-authored-by: Raunaq Morarka <raunaqmorarka@gmail.com> Co-authored-by: Ashhar Hasan <ashhar.hasan@starburstdata.com> Co-authored-by: Michał Ślizak <michal.slizak+github@gmail.com> Co-authored-by: Grzegorz Kokosiński <grzegorz@starburstdata.com> Co-authored-by: Arkadiusz Czajkowski <arek@starburstdata.com> Co-authored-by: Jacob I. Komissar <jacob.komissar@starburstdata.com> Co-authored-by: Krzysztof Sobolewski <krzysztof.sobolewski@starburstdata.com> Co-authored-by: Krzysztof Skrzypczynski <krzysztof.skrzypczynski@starburstdata.com> Co-authored-by: Yuya Ebihara <yuya.ebihara@starburstdata.com> Co-authored-by: Praveen Krishna <praveenkrishna@tutanota.com> Co-authored-by: Karol Sobczak <napewnotrafi@gmail.com> Co-authored-by: Sasha Sheikin <myminitrue@gmail.com> Co-authored-by: Szymon Homa <szymon.homa@starburstdata.com>

cla-bot bot added the cla-signed label Mar 18, 2022

github-actions bot added the tests:hive label Mar 18, 2022

findinpath force-pushed the oss-delta-lake-product-tests branch from 94c2923 to 6c00390 Compare March 21, 2022 11:21

findinpath commented Mar 21, 2022

View reviewed changes

...-product-tests/src/main/java/io/trino/tests/product/deltalake/TestOssDeltaLakeHdfsReads.java Outdated Show resolved Hide resolved

findinpath commented Mar 21, 2022

View reviewed changes

...-product-tests/src/main/java/io/trino/tests/product/deltalake/TestOssDeltaLakeHdfsReads.java Outdated Show resolved Hide resolved

findinpath mentioned this pull request Mar 21, 2022

Support table redirections from Hive to Delta Lake #11550

Merged

findepi changed the title ~~Open source Delta Lake product tests~~ Add Delta Lake product tests Mar 21, 2022

findinpath force-pushed the oss-delta-lake-product-tests branch 2 times, most recently from d00b9a8 to 6e3e356 Compare March 23, 2022 13:52

findinpath marked this pull request as ready for review March 25, 2022 12:59

findinpath marked this pull request as draft March 25, 2022 12:59

findinpath force-pushed the oss-delta-lake-product-tests branch from 6e3e356 to 8781e2a Compare April 6, 2022 21:08

findinpath mentioned this pull request Apr 14, 2022

Ignore views on information_schema.columns queries #11946

Merged

findinpath force-pushed the oss-delta-lake-product-tests branch 6 times, most recently from 0740a29 to e77f81f Compare April 27, 2022 10:51

findinpath marked this pull request as ready for review April 27, 2022 14:50

findinpath requested review from findepi and alexjo2144 April 27, 2022 14:56

findinpath force-pushed the oss-delta-lake-product-tests branch 2 times, most recently from 303427c to 06bfb9a Compare April 27, 2022 19:48

findinpath force-pushed the oss-delta-lake-product-tests branch from 06bfb9a to bd9e5b5 Compare April 29, 2022 14:15

findepi reviewed May 5, 2022

View reviewed changes

testing/trino-testing-minio/pom.xml Outdated Show resolved Hide resolved

testing/trino-testing-minio/pom.xml Outdated Show resolved Hide resolved

plugin/trino-delta-lake/pom.xml Outdated Show resolved Hide resolved

findinpath force-pushed the oss-delta-lake-product-tests branch 2 times, most recently from 310fb13 to 5ba4bd2 Compare May 9, 2022 14:53

alexjo2144 reviewed May 16, 2022

View reviewed changes

findinpath force-pushed the oss-delta-lake-product-tests branch 2 times, most recently from f4b70bd to 97b8ca5 Compare May 16, 2022 21:08

findinpath force-pushed the oss-delta-lake-product-tests branch 5 times, most recently from 5a74dc8 to 3d0e6e0 Compare May 17, 2022 09:31

findepi requested a review from alexjo2144 May 17, 2022 09:52

findinpath force-pushed the oss-delta-lake-product-tests branch from 3d0e6e0 to 684e998 Compare May 17, 2022 11:20

findinpath added 4 commits May 19, 2022 20:37

Extract MinIO client functionality to trino-testing-containers module

16ed6d4

Add MinIO environment extender

aec4492

Remove unused files

483fc43

Move testing resources to trino-testing-resources

9810123

Allow the Delta Lake product tests to make use of the Delta Lake testing resources.

findinpath force-pushed the oss-delta-lake-product-tests branch from 95d0a5a to 44591e1 Compare May 19, 2022 18:58

findinpath force-pushed the oss-delta-lake-product-tests branch from 79e7376 to 032704c Compare May 19, 2022 20:45

findinpath force-pushed the oss-delta-lake-product-tests branch from 2020ab3 to 4a81969 Compare May 20, 2022 09:24

findinpath added 2 commits May 20, 2022 14:38

empty

375d690

empty

d1e2a88

alexjo2144 approved these changes May 23, 2022

View reviewed changes

findepi merged commit 4338264 into trinodb:master May 24, 2022

github-actions bot added this to the 382 milestone May 24, 2022

mosabua mentioned this pull request May 24, 2022

Add Trino 382 release notes #12440

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Delta Lake product tests #11565

Add Delta Lake product tests #11565

findinpath commented Mar 18, 2022 •

edited

Loading

alexjo2144 commented Apr 28, 2022

findinpath commented Apr 29, 2022

findinpath commented May 14, 2022

findinpath commented May 14, 2022

alexjo2144 May 16, 2022 •

edited

Loading

findepi May 17, 2022

findinpath May 17, 2022

alexjo2144 May 23, 2022

findepi May 24, 2022

findinpath commented May 16, 2022

findepi commented May 17, 2022

findinpath commented May 19, 2022

findepi commented May 19, 2022

Add Delta Lake product tests #11565

Add Delta Lake product tests #11565

Conversation

findinpath commented Mar 18, 2022 • edited Loading

Description

Related issues, pull requests, and links

Documentation

Release notes

alexjo2144 commented Apr 28, 2022

findinpath commented Apr 29, 2022

findinpath commented May 14, 2022

findinpath commented May 14, 2022

alexjo2144 May 16, 2022 • edited Loading

Choose a reason for hiding this comment

findepi May 17, 2022

Choose a reason for hiding this comment

findinpath May 17, 2022

Choose a reason for hiding this comment

alexjo2144 May 23, 2022

Choose a reason for hiding this comment

findepi May 24, 2022

Choose a reason for hiding this comment

findinpath commented May 16, 2022

findepi commented May 17, 2022

findinpath commented May 19, 2022

findepi commented May 19, 2022

findinpath commented Mar 18, 2022 •

edited

Loading

alexjo2144 May 16, 2022 •

edited

Loading