Support views in Delta lake Connector #11763

mdesmet · 2022-04-03T23:09:22Z

Description

Support views in Delta lake connector

Is this change a fix, improvement, new feature, refactoring, or other?

Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Delta lake connector

How would you describe this change to a non-technical end user or system administrator?

Related issues, pull requests, and links

Fixes Delta connector does support View creation #11609 , fixes Delta Lake connector shows hive views as tables #15231

Release notes

(x) Release notes entries required with the following suggested text:

# Delta Lake
* Add support for views. ({issue}`11763`)

findepi · 2022-04-04T09:30:14Z

@mdesmet the PR seems to cover AC changes that are unrelated to views.
I guess it wasn't intentional, please make sure they go into separate PRs.

findepi · 2022-04-04T09:30:44Z

cc @alexjo2144 @findinpath @homar

mdesmet · 2022-04-04T10:17:49Z

@mdesmet the PR seems to cover AC changes that are unrelated to views. I guess it wasn't intentional, please make sure they go into separate PRs.

Some view tests within BaseConnectorTest failed because of existing AC setup, I looked at how it's done in iceberg and based myself on that but haven't really tested all AC use cases and the implications of this change. I will setup a separate PR.

Also for clarity. in the original bug report Spark views were mentioned. I have not investigated that for now. I think If we want to support that we might need a translation layer similar like what Linkedin did for the HIve views (Coral). My focus has only been to support creation of views as in Iceberg/Hive.

findinpath · 2022-04-04T10:21:53Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeBasic.java

@@ -112,7 +112,7 @@ public void testNoColumnStats()
        assertQuery("SELECT c_str FROM no_column_stats WHERE c_int = 42", "VALUES 'foo'");
    }

-    @Test
+    @Test(enabled = false)


This is related to supporting all access modes. See #11782

I have added delta.security = system and that causes the metadata to be retrieved in MetadataManager (before the access control was set to SYSTEM),

https://github.com/mdesmet/trino/blob/1c1fb3021648803d5138948e13b93c93ec2cf93d/core/trino-main/src/main/java/io/trino/metadata/MetadataManager.java#L2257-L2264

please reenable the test.

then let's think how to make it pass. maybe we don't need delta.security = system here.
or maybe we need to merge the other PR first.
IDK yet, but let's have the test remind us (ie be failing until we can fix it)

This change is related to the other PR. I will remove that commit here, but as I said before it seems that more tests are failing because of the AC setup in the BaseConnectorTest.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSecurityModule.java

findinpath · 2022-04-04T10:30:45Z

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java

@@ -118,7 +157,9 @@ public HiveMetastoreBackedDeltaLakeMetastore(
    {
        Optional<Table> candidate = delegate.getTable(databaseName, tableName);
        candidate.ifPresent(table -> {
-            if (!TABLE_PROVIDER_VALUE.equalsIgnoreCase(table.getParameters().get(TABLE_PROVIDER_PROPERTY))) {
+            if (((table.getTableType().equals(EXTERNAL_TABLE.name()) || table.getTableType().equals(MANAGED_TABLE.name())) &&


one check per line pls (for readability sake)

Might be better to split these checks into two static util methods so that this line reads !isDeltaDataTable(table) && !isDeltaView(table)

findinpath · 2022-04-04T10:32:59Z

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java

+    }
+
+    private static boolean isView(String tableType, Map<String, String> tableParameters)
+


remove empty line

findepi · 2022-04-04T12:45:21Z

Delta, Hive and Iceberg can use the same catalog (eg Glue or HMS).
A view created by one connector should be accessible by all others.
Let's have a test for that.

findepi · 2022-04-04T12:47:38Z

Some view tests within BaseConnectorTest failed because of existing AC setup, I looked at how it's done in iceberg and based myself on that but haven't really tested all AC use cases and the implications of this change. I will setup a separate PR.

Understood. Let's separate and iterate from there

Also for clarity. in the original bug report Spark views were mentioned. I have not investigated that for now. I think If we want to support that we might need a translation layer similar like what Linkedin did for the HIve views (Coral). My focus has only been to support creation of views as in Iceberg/Hive.

This is totally different story. I don't know what's the level of support for Spark SQL in Coral. So far we were reading only HiveQL-based views with Coral, and that was really challenging to get right-ish. We can invest into SparkSQL-based views for Delta Lake, but that's certainly a project on its own. Let's have Trino Views supported first, just as we did for Iceberg.

mdesmet · 2022-04-05T20:17:18Z

So basically the issue is with following check in MetadataManager. The isCatalogManagedSecurity method returns false as we setup SYSTEM as the default AC mode in Delta.

    @Override
    public Optional<ViewDefinition> getView(Session session, QualifiedObjectName viewName)
    {
        Optional<ConnectorViewDefinition> connectorView = getViewInternal(session, viewName);
        if (connectorView.isEmpty() || connectorView.get().isRunAsInvoker() || isCatalogManagedSecurity(session, viewName.getCatalogName())) {
            return connectorView.map(view -> new ViewDefinition(viewName, view));
        }

        Identity runAsIdentity = systemSecurityMetadata.getViewRunAsIdentity(session, viewName.asCatalogSchemaTableName())
                .or(() -> connectorView.get().getOwner().map(Identity::ofUser))
                .orElseThrow(() -> new TrinoException(NOT_SUPPORTED, "Catalog does not support run-as DEFINER views: " + viewName));
        return Optional.of(new ViewDefinition(viewName, connectorView.get(), runAsIdentity));
    }

Then in the DeltaLakeAccessControlFactory we setup isUsingSystemSecurity to return true, However in Hive and Iceberg we don't store the owner of the view in that case. The same Iceberg BaseConnectorTest also fails on the same error if i set iceberg.security = system.

That's why i had implemented the Access Control modes first in order to be able to leverage the existing BaseConnector tests as the system AC doesn't support DEFINER views.

homar · 2022-04-07T08:17:51Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

+                getView(session, name).ifPresent(view -> views.put(name, view));
+            }
+            catch (TrinoException e) {
+                if (e.getErrorCode().equals(HIVE_VIEW_TRANSLATION_ERROR.toErrorCode())) {


Wouldn't it be simpler to have a Set with these 3 error codes(you can put comments in the place where set is created) and here only one if, something like:
if(errorCoded.contains(e.getErrorCode()) { //do nothing } else { throw e}

HIVE_VIEW_TRANSLATION_ERROR should never be thrown here anyway, since we're not translating views.

findinpath · 2022-09-06T14:13:43Z

@mdesmet please rebase on top of the changes from #11782

findinpath · 2022-10-10T05:08:40Z

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java


 public class HiveMetastoreBackedDeltaLakeMetastore
        implements DeltaLakeMetastore
 {
    public static final String TABLE_PROVIDER_PROPERTY = "spark.sql.sources.provider";
    public static final String TABLE_PROVIDER_VALUE = "DELTA";

+    // Be compatible with views defined by the Hive connector, which can be useful under certain conditions.


, which can be useful under certain conditions. -> this part of the comment does add a bit of mistery to the statement, but has no real gain in understanding for the reader.

findinpath · 2022-10-10T05:14:32Z

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java

+    private List<String> listDatabases(Optional<String> database)
+    {
+        if (database.isPresent()) {
+            // TODO: should this decision logic go into DeltaLakeMetadata or here


Is this a question for the review?

findinpath · 2022-10-10T05:16:46Z

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java

@@ -188,6 +224,106 @@ public void renameTable(ConnectorSession session, SchemaTableName from, SchemaTa
        delegate.renameTable(from.getSchemaName(), from.getTableName(), to.getSchemaName(), to.getTableName());
    }

+    @Override
+    public void createView(ConnectorSession session, SchemaTableName schemaViewName, ConnectorViewDefinition definition, boolean replace)


The logic from this method seems to be 1 to 1 copied from io.trino.plugin.iceberg.catalog.hms.TrinoHiveCatalog#createView from trino-iceberg.

This applies also for the other methods of the class newly introduced in this PR.

Can you think of a way to avoid such kind of code duplication?

The code in Delta and Iceberg is indeed similar but different from the Hive connector as the Hive connector acts upon a SemiTransactionalhiveMetastore with slightly different method calls. In Iceberg and Delta we don't have the SemiTransactionalhiveMetastore.

To share the code we would need to create a helper class in the Hive connector (the easier part) with all views related operations. This helper class would need access to the HiveMetastore implementation (which will relay to either actual Hive Metastore or GlueHiveMetastore (Glue API calls) due to Guice wiring). This works well for Delta. However on Iceberg the implementation is different. It has two factories to create either a TrinoCatalog for Hive metastore, which has access to the HiveMetastore implementation, either to The Glue catalog which directly calls the Glue API.

We could fix the latter by also using Hive's GlueHiveMetastore in the Icebergs TrinoGlueCatalogin order for the helper class to be used in both cases. There are however also differences between theTrinoGlueCatalogandGlueHiveMetastore` that will need to be investigated and properly tested.

Code reuse is possible but at definitely at a much higher cost also it would probably be better handled in a separate PR. The question arises what we do first: provide the views feature and do the internal refactoring after or first refactor and delay the views feature?

I'd go for trying out (as discussed offline) for grouping together the code which would be otherwise repetitive after landing your PR. This seems rather cheap to do and shouldn't add an increased complexity in the code of Delta and Iceberg connectors.

Regarding TrinoGlueCatalog I'd suggest to leave it as it is (at least for now).

The main purpose of my comment was DRY (Don't repeat yourself) and not a more complex refactoring (which in any case would not be in the scope of this PR).

I have followed your advice and added a ViewsMetastoreHelper class, potentially it could also be used for Hive but then we would have to use callbacks. I didn't do that for now.

findinpath · 2022-10-10T05:22:46Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

@@ -170,6 +170,9 @@ protected QueryRunner createQueryRunner()
    protected boolean hasBehavior(TestingConnectorBehavior connectorBehavior)
    {
        switch (connectorBehavior) {
+            case SUPPORTS_CREATE_VIEW:


As a follow-up to this PR, could you please create an issue (with good-first-issue label) to add support for comments on Delta views ?

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastore.java

findinpath · 2022-10-10T05:39:46Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

@@ -170,6 +170,9 @@ protected QueryRunner createQueryRunner()
    protected boolean hasBehavior(TestingConnectorBehavior connectorBehavior)
    {


The PR is missing a test where Trino views for Delta gets created on a Glue backed metastore.
Consider either such a test either in trino-delta-lake.

findinpath · 2022-11-07T05:22:52Z

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java

@@ -76,7 +76,6 @@
 {
    public static final String TABLE_PROVIDER_PROPERTY = "spark.sql.sources.provider";
    public static final String TABLE_PROVIDER_VALUE = "DELTA";
-


unrelated change - please restore.

findinpath · 2022-11-07T05:25:09Z

.../src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeViewsGlueMetastore.java

+        String view = "test_glue_view_" + randomTableSuffix();
+        try {
+            assertUpdate(format("CREATE VIEW %s AS SELECT 1 AS val ", view), 1);
+            assertQuery("SELECT val FROM " + view, "VALUES (1)");


nit: The paranthesis in the expression VALUES (1) are not necessary.

findinpath · 2022-11-07T05:25:55Z

.../src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeViewsGlueMetastore.java

+public class TestDeltaLakeViewsGlueMetastore
+        extends AbstractTestQueryFramework
+{
+    protected static final String SCHEMA = "test_delta_lake_glue_views_" + randomTableSuffix();


Why protected? I think private will do just fine.

findinpath · 2022-11-07T05:30:21Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ViewsMetastoreHelper.java

+        return Optional.of(definition);
+    }
+
+    public static boolean isView(String tableType, Map<String, String> tableParameters)


This method can stay private

findinpath · 2022-11-07T05:32:15Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ViewsMetastoreHelper.java

+        }
+    }
+
+    public List<SchemaTableName> listViews(ConnectorSession session, Optional<String> database)


ConnectorSession session parameter seems unused.

findinpath · 2022-11-07T05:35:54Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ViewsMetastoreHelper.java

+                .map(table -> new SchemaTableName(schema, table));
+    }
+
+    public Optional<ConnectorViewDefinition> getView(ConnectorSession session, SchemaTableName viewName)


ConnectorSession session parameter seems to be unused.
Make sure that also the other methods calling this method drop it as well : dropView , getViews.

findinpath · 2022-11-07T05:46:07Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ViewsMetastoreHelper.java

+                        view.getOwner()));
+    }
+
+    private Map<String, String> createViewProperties(ConnectorSession session)


Can we use createViewProperties(ConnectorSession session, String trinoVersion, String connectorName) call instead (to avoid code duplication) ?

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/thrift/ThriftHiveMetastore.java

findinpath · 2022-11-07T05:52:58Z

.../src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeViewsGlueMetastore.java

+        }
+    }
+
+    @AfterClass


@AfterClass(alwaysRun = true)

findinpath · 2022-11-07T05:53:57Z

.../src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeViewsGlueMetastore.java

+    protected static final String SCHEMA = "test_delta_lake_glue_views_" + randomTableSuffix();
+    protected static final String CATALOG_NAME = "test_delta_lake_glue_views";
+
+    private File schemaLocation;


Can be converted to local variable.

mdesmet · 2022-11-23T20:17:00Z

Where's the test for #11763 (comment)?

Delta, Hive and Iceberg can use the same catalog (eg Glue or HMS).
A view created by one connector should be accessible by all others.

@ebyhr : The only way to test that all three together is through a product test. It seems we currently don't have any configuration that has both Hive, Iceberg and Delta. Is that what we want or do you see another way to test it?

findinpath · 2022-11-24T04:55:07Z

Delta, Hive and Iceberg can use the same catalog (eg Glue or HMS).
A view created by one connector should be accessible by all others.

I think you can use BaseSharedMetastoreTest from trino-iceberg as PoC in trino-delta-lake project where you could showcase the above mentioned functionality between delta-lake and hive connector.
It is enough to showcase for the two of those mentioned and not for all.

Side note:

#11763 (comment)

@mdesmet the product test enviroment singlenode-delta-lake-oss has both hive and delta with table redirection support already configured.

...elta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeSharedMetastoreViewsTest.java

...elta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedFileMetastoreViews.java

...elta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedGlueMetastoreViews.java

...-lake/src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeGlueMetastore.java

.../src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeViewsGlueMetastore.java

mdesmet · 2022-11-29T13:08:03Z

@ebyhr : PTAL

ebyhr · 2022-11-30T00:47:24Z

/test-with-secrets sha=0cfc5761b4c02ec5137f9dfb751137f637bf42d0

...elta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeSharedMetastoreViewsTest.java

...elta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedGlueMetastoreViews.java

.../src/test/java/io/trino/plugin/deltalake/metastore/glue/TestDeltaLakeViewsGlueMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/TrinoViewHiveMetastore.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/TrinoViewUtil.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastore.java

ebyhr · 2022-12-05T00:02:00Z

/test-with-secrets sha=cd69957b50ee03ac83f57a86fa77017c47d65c4f

https://github.com/trinodb/trino/actions/runs/3615689927

...elta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeSharedMetastoreViewsTest.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java

ebyhr · 2022-12-08T01:49:49Z

/test-with-secrets sha=94226a3573dde2ac87ba7cadb5cf607174f4bf8f

https://github.com/trinodb/trino/actions/runs/3644481861

findepi · 2023-08-07T15:56:39Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/TrinoViewUtil.java

+
+        if (!isPrestoView(tableParameters)) {
+            // Hive views are not compatible
+            throw new HiveViewNotSupportedException(viewName);


What are the situations where isView above return true (the method is defined as isHiveOrPrestoView(tableType) && PRESTO_VIEW_COMMENT.equals(tableParameters.get(TABLE_COMMENT))) but isPrestoView will return false?

intuitively this line is not reachable?

(i am changing this in #18570)

cla-bot bot added the cla-signed label Apr 3, 2022

findinpath reviewed Apr 4, 2022

View reviewed changes

mdesmet force-pushed the feature/delta_views branch from 02a785a to 2737614 Compare April 5, 2022 19:16

homar reviewed Apr 7, 2022

View reviewed changes

homar approved these changes Apr 7, 2022

View reviewed changes

mdesmet mentioned this pull request Apr 8, 2022

Support all Trino Security Access Control modes in Delta lake Connector #11782

Merged

mdesmet force-pushed the feature/delta_views branch from 2737614 to 1701125 Compare September 9, 2022 06:35

github-actions bot added the tests:hive label Sep 9, 2022

mdesmet force-pushed the feature/delta_views branch 2 times, most recently from 2708f08 to 284cbe7 Compare September 11, 2022 22:18

mdesmet force-pushed the feature/delta_views branch 3 times, most recently from 955b6bf to d8c4da0 Compare October 9, 2022 20:50

findinpath reviewed Oct 10, 2022

View reviewed changes

mdesmet marked this pull request as draft October 28, 2022 18:50

mdesmet force-pushed the feature/delta_views branch 5 times, most recently from c99bf35 to a016c9c Compare October 30, 2022 19:10

mdesmet marked this pull request as ready for review October 30, 2022 19:13

mdesmet force-pushed the feature/delta_views branch from a016c9c to e00f037 Compare November 5, 2022 21:56

findinpath reviewed Nov 7, 2022

View reviewed changes

mdesmet force-pushed the feature/delta_views branch from b8fc01a to 5f8597e Compare November 23, 2022 23:15

mdesmet force-pushed the feature/delta_views branch 3 times, most recently from d335361 to e27141c Compare November 24, 2022 08:24

mdesmet self-assigned this Nov 24, 2022

ebyhr reviewed Nov 25, 2022

View reviewed changes

mdesmet force-pushed the feature/delta_views branch from e27141c to 2f14321 Compare November 25, 2022 20:13

findinpath mentioned this pull request Nov 29, 2022

Delta Lake connector shows hive views as tables #15231

Closed

mdesmet force-pushed the feature/delta_views branch 3 times, most recently from 8792f4c to 0cfc576 Compare November 29, 2022 10:41

ebyhr reviewed Dec 1, 2022

View reviewed changes

mdesmet force-pushed the feature/delta_views branch from 0cfc576 to cd69957 Compare December 3, 2022 11:44

Use randomNameSuffix instead of local method

95f56ff

ebyhr approved these changes Dec 7, 2022

View reviewed changes

mdesmet added 2 commits December 7, 2022 15:31

Extract Trino View functionality

b803b47

Implement Delta Lake views support

94226a3

mdesmet force-pushed the feature/delta_views branch from cd69957 to 94226a3 Compare December 7, 2022 14:33

ebyhr merged commit 6283fef into trinodb:master Dec 8, 2022

github-actions bot added this to the 404 milestone Dec 8, 2022

ebyhr mentioned this pull request Dec 8, 2022

Release notes for 405 #15058

Closed

colebow mentioned this pull request Dec 14, 2022

Add Trino 405 release notes #15139

Merged

findinpath mentioned this pull request Dec 30, 2022

Use views for the creating the temporary models in Delta tests starburstdata/dbt-trino#205

Merged

4 tasks

findepi reviewed Aug 7, 2023

View reviewed changes

		}

		private static boolean isView(String tableType, Map<String, String> tableParameters)

		@@ -170,6 +170,9 @@ protected QueryRunner createQueryRunner()
		protected boolean hasBehavior(TestingConnectorBehavior connectorBehavior)
		{

Support views in Delta lake Connector #11763

Support views in Delta lake Connector #11763

Conversation

mdesmet commented Apr 3, 2022 • edited by ebyhr Loading

Description

Related issues, pull requests, and links

Release notes

findepi commented Apr 4, 2022

findepi commented Apr 4, 2022

mdesmet commented Apr 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Apr 4, 2022

findepi commented Apr 4, 2022

mdesmet commented Apr 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Sep 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdesmet commented Nov 23, 2022

findinpath commented Nov 24, 2022 • edited Loading

mdesmet commented Nov 29, 2022

ebyhr commented Nov 30, 2022

ebyhr commented Dec 5, 2022 • edited Loading

ebyhr commented Dec 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdesmet commented Apr 3, 2022 •

edited by ebyhr

Loading

mdesmet commented Apr 5, 2022 •

edited

Loading

findinpath commented Nov 24, 2022 •

edited

Loading

ebyhr commented Dec 5, 2022 •

edited

Loading

ebyhr commented Dec 8, 2022 •

edited

Loading