Support table redirections from Hive to Delta Lake #11550

findinpath · 2022-03-17T18:21:33Z

The Hive connector can make use of the hive.delta-lake-catalog-name
configuration property for enable table redirects towards the
Delta Lake tables.

Description

Is this change a fix, improvement, new feature, refactoring, or other?

Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Hive connector

How would you describe this change to a non-technical end user or system administrator?

This functionality allows redirecting behind the scenes the a table task used for reading / updating the table from the Hive connector to the Delta Lake connector in case that the table to which the SQL command points to is a Delta Lake table.

Related issues, pull requests, and links

Depends on #11565

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

findepi · 2022-03-21T12:47:42Z

the CI is red.

btw is this the reason this is a Draft?

findinpath · 2022-03-21T12:59:40Z

btw is this the reason this is a Draft?

I intend to add the product tests for ensuring that the functionality works as expected before setting the PR ready for review.

findinpath · 2022-03-21T13:30:55Z

the CI is red.

There were missing some config coverage tests for Hive before.

findepi · 2022-03-21T21:04:08Z

Per offline discussion, this is blocked by #11565 so that we can add tests.

findinpath · 2022-03-23T12:48:35Z

TODO: Add TestHiveAndDatabricksRedirect and add to trino/testing/trino-product-tests-launcher/src/main/resources/docker/presto-product-tests/conf/environment/singlenode-delta-lake-databricks/hive.properties the following line:

hive.non-managed-table-writes-enabled=true

findepi · 2022-03-25T12:14:39Z

Tests require #11565

Dearkano · 2022-04-06T17:57:27Z

Hi, I'm wondering if there's any timeline for this feature's release? Thank you!

findinpath · 2022-04-19T15:37:57Z

Rebased on master and added integration tests for the redirection functionality.

plugin/trino-delta-lake/pom.xml

...ino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeSharedMetastoreTest.java

...ino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedGlueMetastore.java

findinpath · 2022-04-20T19:28:00Z

@findepi the compilation of the trino-delta-lake project is now failing.
I have no better option at hand than the exclusion of the aws-sdk-core depdency from the analyze-only goal https://maven.apache.org/plugins/maven-dependency-plugin/examples/exclude-dependencies-from-dependency-analysis.html

findepi · 2022-04-22T07:30:06Z

https://github.com/trinodb/trino/runs/6110999392?check_suite_focus=true

Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project trino-delta-lake: There are test failures.
Error:  
Error:  Please refer to /home/runner/work/trino/trino/plugin/trino-delta-lake/target/surefire-reports for the individual test results.
Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/trino/trino/plugin/trino-delta-lake && /opt/hostedtoolcache/Java_Zulu_jdk/11.0.15-10/x64/bin/java -Dfile.encoding=UTF-8 -Xmx3g -Xms3g -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:-OmitStackTraceInFastThrow -jar /home/runner/work/trino/trino/plugin/trino-delta-lake/target/surefire/surefirebooter5870515897782731080.jar /home/runner/work/trino/trino/plugin/trino-delta-lake/target/surefire 2022-04-21T13-15-20_232-jvmRun1 surefire15439557697247218053tmp surefire_011486807429141444913tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 137
Error:  Crashed tests:
Error:  io.trino.plugin.deltalake.TestDeltaLakeOptimizedWriterConnectorSmokeTest

cc @findinpath @hashhar

findinpath · 2022-04-25T07:50:16Z

plugin/trino-delta-lake/pom.xml

@@ -426,6 +432,18 @@
                            </ignoredResourcePatterns>
                        </configuration>
                    </plugin>
+
+                    <plugin>
+                        <artifactId>maven-dependency-plugin</artifactId>


created https://issues.apache.org/jira/browse/MDEP-802

findinpath · 2022-04-26T04:19:54Z

Rebased on master in an attempt to avoid build runner related issue related to the Github runner while trying to build trino-delta-lake project.

alexjo2144

Couple small things

plugin/trino-delta-lake/pom.xml

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveConfig.java

alexjo2144 · 2022-04-28T20:02:13Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveSessionProperties.java

@@ -116,6 +116,7 @@
    public static final String SIZE_BASED_SPLIT_WEIGHTS_ENABLED = "size_based_split_weights_enabled";
    public static final String MINIMUM_ASSIGNED_SPLIT_WEIGHT = "minimum_assigned_split_weight";
    public static final String NON_TRANSACTIONAL_OPTIMIZE_ENABLED = "non_transactional_optimize_enabled";
+    public static final String DELTA_LAKE_CATALOG_NAME = "delta_lake_catalog_name";


Same here, put the redirect catalog name fields together

The Hive connector can make use of the `hive.delta-lake-catalog-name` configuration property for enable table redirects towards the Delta Lake tables.

findinpath · 2022-04-29T14:03:27Z

Rebased on master to make use of 49381ba

cla-bot bot added the cla-signed label Mar 17, 2022

github-actions bot added the tests:hive label Mar 17, 2022

findinpath force-pushed the add-delta-to-hive-redirects branch from 5a062f9 to c245456 Compare March 21, 2022 13:30

findinpath marked this pull request as ready for review March 21, 2022 13:30

findinpath requested review from findepi and alexjo2144 March 21, 2022 13:31

findepi marked this pull request as draft March 21, 2022 21:04

findinpath marked this pull request as ready for review March 22, 2022 05:17

findepi marked this pull request as draft March 25, 2022 12:14

findinpath marked this pull request as ready for review March 25, 2022 13:00

findepi marked this pull request as draft April 1, 2022 15:23

findepi marked this pull request as ready for review April 6, 2022 20:39

findepi approved these changes Apr 6, 2022

View reviewed changes

findinpath force-pushed the add-delta-to-hive-redirects branch from e7f1dce to e5a20dd Compare April 11, 2022 14:25

findinpath force-pushed the add-delta-to-hive-redirects branch from e5a20dd to 6c5ae97 Compare April 19, 2022 15:37

findinpath requested a review from findepi April 19, 2022 15:38

findinpath commented Apr 19, 2022

View reviewed changes

plugin/trino-delta-lake/pom.xml Show resolved Hide resolved

findinpath force-pushed the add-delta-to-hive-redirects branch 2 times, most recently from bd5f4be to 6776fd2 Compare April 19, 2022 15:43

findinpath commented Apr 19, 2022

View reviewed changes

...ino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeSharedMetastoreTest.java Outdated Show resolved Hide resolved

findinpath commented Apr 19, 2022

View reviewed changes

...ino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedGlueMetastore.java Outdated Show resolved Hide resolved

findinpath force-pushed the add-delta-to-hive-redirects branch from 6776fd2 to af723a6 Compare April 20, 2022 05:11

This comment was marked as duplicate.

Sign in to view

findepi force-pushed the add-delta-to-hive-redirects branch from dcaca0a to 22f227b Compare April 21, 2022 10:41

findinpath commented Apr 25, 2022

View reviewed changes

findinpath force-pushed the add-delta-to-hive-redirects branch 2 times, most recently from d5c9463 to 7e20a59 Compare April 26, 2022 04:14

findinpath force-pushed the add-delta-to-hive-redirects branch 10 times, most recently from 6e980e8 to 21ea76e Compare April 28, 2022 18:25

findinpath requested review from findepi and alexjo2144 April 28, 2022 19:39

alexjo2144 reviewed Apr 28, 2022

View reviewed changes

findinpath force-pushed the add-delta-to-hive-redirects branch from 21ea76e to b37089a Compare April 29, 2022 04:36

Support table redirections from Hive to Delta Lake

cafe666

The Hive connector can make use of the `hive.delta-lake-catalog-name` configuration property for enable table redirects towards the Delta Lake tables.

findinpath force-pushed the add-delta-to-hive-redirects branch from b37089a to cafe666 Compare April 29, 2022 14:03

empty

26a414c

findepi merged commit 74b324c into trinodb:master May 5, 2022

findepi mentioned this pull request May 5, 2022

Release notes for 380 #12182

Closed

github-actions bot added this to the 380 milestone May 5, 2022

mosabua mentioned this pull request May 5, 2022

Add Trino 380 release notes #12184

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support table redirections from Hive to Delta Lake #11550

Support table redirections from Hive to Delta Lake #11550

findinpath commented Mar 17, 2022 •

edited

Loading

findepi commented Mar 21, 2022

findinpath commented Mar 21, 2022

findinpath commented Mar 21, 2022

findepi commented Mar 21, 2022

findinpath commented Mar 23, 2022 •

edited

Loading

findepi commented Mar 25, 2022

Dearkano commented Apr 6, 2022

findinpath commented Apr 19, 2022

findinpath commented Apr 20, 2022

This comment was marked as duplicate.

findepi commented Apr 22, 2022

findinpath Apr 25, 2022

findinpath commented Apr 26, 2022

alexjo2144 left a comment

alexjo2144 Apr 28, 2022

findinpath commented Apr 29, 2022

Support table redirections from Hive to Delta Lake #11550

Support table redirections from Hive to Delta Lake #11550

Conversation

findinpath commented Mar 17, 2022 • edited Loading

Description

Related issues, pull requests, and links

Documentation

Release notes

findepi commented Mar 21, 2022

findinpath commented Mar 21, 2022

findinpath commented Mar 21, 2022

findepi commented Mar 21, 2022

findinpath commented Mar 23, 2022 • edited Loading

findepi commented Mar 25, 2022

Dearkano commented Apr 6, 2022

findinpath commented Apr 19, 2022

findinpath commented Apr 20, 2022

This comment was marked as duplicate.

findepi commented Apr 22, 2022

findinpath Apr 25, 2022

Choose a reason for hiding this comment

findinpath commented Apr 26, 2022

alexjo2144 left a comment

Choose a reason for hiding this comment

alexjo2144 Apr 28, 2022

Choose a reason for hiding this comment

findinpath commented Apr 29, 2022

findinpath commented Mar 17, 2022 •

edited

Loading

findinpath commented Mar 23, 2022 •

edited

Loading