Support JDBC catalog in Iceberg connector #11772

ebyhr · 2022-04-04T08:43:52Z

Description

Support JDBC catalog in Iceberg connector.
Fixes #9968

Release notes

(x) Release notes entries required with the following suggested text:

# Iceberg
* Add support for JDBC catalog. ({issue}`9968`)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/JdbcIcebergClient.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/TrinoJdbcCatalog.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/JdbcIcebergClient.java

...ino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/IcebergJdbcCatalogModule.java

...iceberg/src/test/java/io/trino/plugin/iceberg/catalog/jdbc/TestIcebergJdbcIcebergConfig.java

sungwy-backup · 2022-04-22T14:27:39Z

Very excited to see this getting traction. Just wanted to make a note here that we may expect to see a similar issue with #6850 if not accounted for.

The iceberg catalog on spark sql writes and preserves upper cases in schema and table names through its jdbc connection, and this could break on Trino iceberg-jdbc catalog if not accounted for. We've been testing out our own versions of jdbc catalog on our iceberg plugin, and we have come across this issue that we have yet to resolve.

Query
show tables from test
Result
exampletable1

In database, upper case is preserved as "exampleTable1"

Query
select * from test."exampleTable1" limit 5
Error
Table 'iceberg_jdbc.test.exampleTable1' does not exist

We do not see the same issue with lower cased table names

ebyhr · 2022-04-25T01:29:45Z

@sungwy The root cause relates to #17. We prefer small start when adding a new enhancement, so I won't handle case insensitive matching in this PR. We can improve such things later.

findepi · 2022-05-04T12:24:50Z

Do we have / want to have some compatibility tests with Spark?

meetsitaram · 2022-05-04T19:10:20Z

Can you provide some details on how to configure s3 access with jdbc catalog? For hive catalog, we add below config in iceberg.properties file.

hive.s3.endpoint=S3_ENDPOINT
hive.s3.aws-access-key=S3_ACCESS_KEY
hive.s3.aws-secret-key=S3_SECRET_KEY

Wondering if similar config is required with jdbc as well or some alternative exists that can access ~/.aws/config and ~/.aws/credentials files.

samredai · 2022-05-05T17:57:44Z

Can you provide some details on how to configure s3 access with jdbc catalog? For hive catalog, we add below config in iceberg.properties file.
hive.s3.endpoint=S3_ENDPOINT
hive.s3.aws-access-key=S3_ACCESS_KEY
hive.s3.aws-secret-key=S3_SECRET_KEY
Wondering if similar config is required with jdbc as well or some alternative exists that can access ~/.aws/config and ~/.aws/credentials files.

My understanding is that it would be the same config items you've listed (and testing with the IcebergQueryRunner seems to confirm this). The Iceberg catalog implementations just eventually pass these to the TrinoS3ConfigurationInitializer. Admittedly, the hive prefix for the parameter names feels odd when you're using any of the other catalog types (such as JDBC here) but IMO changing that probably requires too much to bundle with this PR.

meetsitaram · 2022-05-09T16:38:36Z

Can you provide some details on how to configure s3 access with jdbc catalog? For hive catalog, we add below config in iceberg.properties file.
hive.s3.endpoint=S3_ENDPOINT
hive.s3.aws-access-key=S3_ACCESS_KEY
hive.s3.aws-secret-key=S3_SECRET_KEY
Wondering if similar config is required with jdbc as well or some alternative exists that can access ~/.aws/config and ~/.aws/credentials files.
My understanding is that it would be the same config items you've listed (and testing with the IcebergQueryRunner seems to confirm this). The Iceberg catalog implementations just eventually pass these to the TrinoS3ConfigurationInitializer. Admittedly, the hive prefix for the parameter names feels odd when you're using any of the other catalog types (such as JDBC here) but IMO changing that probably requires too much to bundle with this PR.

the same configs are working with jdbc catalog as well. thx.

ebyhr · 2022-05-12T05:01:11Z

Added support for product tests with Spark.

...presto-product-tests/conf/environment/singlenode-spark-iceberg-jdbc-catalog/create-table.sql

sungwy · 2022-06-13T13:26:45Z

Hi @ebyhr : here's the recently merged PR in Apache Iceberg that resolves the Jdbc catalog reserved keyword conflict on NAMESPACE_PROPERTY_KEY for reference

apache/iceberg#5017

plugin/trino-iceberg/src/test/resources/jdbc/create-table.sql

findepi · 2022-07-20T13:17:30Z

plugin/trino-iceberg/src/test/resources/jdbc/create-table.sql

@@ -0,0 +1,20 @@
+/**
+  Table definition in Iceberg repository:
+  https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java


How do we make sure this file stays in sync with the referenced one?

or, maybe we can avoid having a copy here?

It looks like the namespace_properties_table changes along with others we were waiting for just got released on 0.14.0 a few days ago.

Maybe we could now reference the static variable JdbcUtil.CREATE_NAMESPACE_PROPERTIES_TABLE here going forward

It looks like the namespace_properties_table changes along with others

what are the changes?
does it mean we should test this catalog against Iceberg 13 and Iceberg 14 as well?

They changed column names in apache/iceberg#5017. The PR doesn't contain logic to handle the old table definition, so I suppose they just broke the backward compatibility.

does it mean we should test this catalog against Iceberg 13 and Iceberg 14 as well?

I think so. (or clarify the supported version in documentation)

They changed column names in apache/iceberg#5017. The PR doesn't contain logic to handle the old table definition, so I suppose they just broke the backward compatibility.

Concerning

According to this issue on Iceberg repo no backwards compatible was broken because the previous version of this was never released? apache/iceberg#4952

In fact I can confirm the Iceberg guys didn't release a breaking change, see the file in question at 0.13.2 https://github.com/apache/iceberg/blob/apache-iceberg-0.13.2/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java where this NAMESPACE_PROPERTY_KEY didn't exist, and now see same file at 0.14.0, where it does exist with the new value https://github.com/apache/iceberg/blob/apache-iceberg-0.14.0/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L83.

So sounds like it was never released with the old value

@nfcampos Your confirmation looks correct. Do you know if they have compatibility test in the repository?

Not that I know of, no

ebyhr · 2022-07-21T02:10:14Z

Just rebased on upstream to resolve logical conflicts.

martint · 2023-01-13T22:53:16Z

cc @mosabua @bitsondatadev for how to best provide those warnings in the docs.

bitsondatadev · 2023-01-13T22:54:35Z

@ebyhr, let's discuss some of the documentation phrasing on Slack.

ebyhr · 2023-01-17T04:13:54Z

Rebased on upstream to resolve conflicts.

ebyhr · 2023-01-17T09:34:37Z

Addressed comments.

findepi

Please give @alexjo2144 or @findinpath a chance to read too

alexjo2144

Are we respecting the jdbc.strict-mode table property?

Do we have follow up issues for Views and MVs?

docs/src/main/sphinx/connector/iceberg.rst

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/IcebergJdbcClient.java

alexjo2144 · 2023-01-17T19:07:18Z

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/IcebergJdbcConnectionFactory.java

+    public Connection openConnection()
+            throws SQLException
+    {
+        Connection connection = DriverManager.getConnection(connectionUrl);


Should we have some connection pool here to draw from?

I would skip supporting connection pools in this PR. We can add it later when needed.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/jdbc/TrinoJdbcCatalog.java

ebyhr · 2023-01-18T08:33:59Z

Are we respecting the jdbc.strict-mode table property?

We shouldn't create tables on non-existing namespaces regardless of the properties and it's tested by BaseConnectorTest#testCreateTableSchemaNotFound. The test doesn't ensure the concurrent modification though.

Do we have follow up issues for Views and MVs?

No because there's no plan to support them at this time.

ebyhr · 2023-01-18T08:51:04Z

(Rebased on upstream to resolve conflicts)

alexjo2144

Looks good to me

bitsondatadev

Just a docs review here.

Few small changes to the docs but they look good!

docs/src/main/sphinx/connector/iceberg.rst

bjackson-ep · 2023-02-09T19:38:45Z

Greetings @findepi and @bitsondatadev thank you for your diligent work on JDBC other connectors. Is there anything you need from the community to get this one across the line? I am happy to connect you with Dremio PMs or others that can provide clarifications or support. I see a lot of value in Trino and would love to see the query-where-it-lives grow to this catalog. We are believers in data as code and want to help evolve that space through technologies like Trino and Nessie. Thank you.

alexjo2144 · 2023-02-09T20:43:29Z

Is there anything you need from the community to get this one across the line?

The JDBC catalog was included in the 406 release a few weeks ago, so I would call it across the line. https://trino.io/docs/current/release/release-406.html#iceberg-connector

Let us know if you have a chance to try it out, feedback is always welcome.

bjackson-ep · 2023-02-09T22:50:43Z

#11701 Support for Nessie Catalogs is what I am talking about. There is a March 29, 2022 pull request in the queue for this. From: Alexander Jo ***@***.***> Sent: Thursday, February 9, 2023 2:44 PM To: trinodb/trino ***@***.***> Cc: Brandon Jackson ***@***.***>; Comment ***@***.***> Subject: Re: [trinodb/trino] Support JDBC catalog in Iceberg connector (PR #11772) External Email: This message originated from outside Empower Pharmacy. Please exercise caution before opening attachments, clicking links, replying, or providing information to the sender. Is there anything you need from the community to get this one across the line? The JDBC catalog was included in the 406 release a few weeks ago, so I would call it across the line. https://trino.io/docs/current/release/release-406.html#iceberg-connector Let us know if you have a chance to try it out, feedback is always welcome. — Reply to this email directly, view it on GitHub<#11772 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A2SNAU6KPC77BKN6WWHVJ5DWWVJH7ANCNFSM5SOYYGFA>. You are receiving this because you commented.Message ID: ***@***.******@***.***>>

penggewudi · 2023-06-25T03:30:24Z

@sungwy The root cause relates to #17. We prefer small start when adding a new enhancement, so I won't handle case insensitive matching in this PR. We can improve such things later.

Excuse me, I would like to ask how is the progress of this issue? We connect to Iceberg through the Postgre JDBC catalog, the following is the properties configuration of the configuration catalog.

connector.name=iceberg
hive.s3.endpoint=xxx
hive.s3.aws-access-key=xxx
hive.s3.aws-secret-key=xxx
hive.s3.path-style-access=true
hive.s3.ssl.enabled=false
iceberg.catalog.type=jdbc
iceberg.jdbc-catalog.catalog-name=jdbc
iceberg.jdbc-catalog.driver-class=org.postgresql.Driver
iceberg.jdbc-catalog.connection-url=jdbc:postgresql://xxx/jdbc
iceberg.jdbc-catalog.connection-user=xxx
iceberg.jdbc-catalog.connection-password=xxx
iceberg.jdbc-catalog.default-warehouse-dir=s3://xxx/

We use spark sql to write table named Test_Table into Iceberg, When querying with the following statement through Trino CLI, the error of Table 'xxx.test_table' does not exist will appear. I try to add parameters like iceberg.jdbc-catalog.case-insensitive-name-matching=true, it seems that Trino service It will report an error and cannot start.

SELECT * FROM Test_Table / SELECT * FROM test_table

In addition, when we write to Iceberg in the same way but the table named test_table, I can query and get the results I want. If I want to keep the consistency of writing and reading table names, how can I do it Woolen cloth?

cla-bot bot added the cla-signed label Apr 4, 2022

github-actions bot added the docs label Apr 4, 2022

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 3409dd0 to d2eb975 Compare April 5, 2022 06:28

erikerlandson mentioned this pull request Apr 5, 2022

Trino error message "failed to get status" os-climate/os_c_data_commons#154

Closed

hashhar mentioned this pull request Apr 8, 2022

Support for JDBC catalog in Iceberg connector #11883

Closed

meetsitaram reviewed Apr 11, 2022

View reviewed changes

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from d2eb975 to 5a6cd1f Compare May 2, 2022 08:00

ebyhr marked this pull request as ready for review May 2, 2022 08:37

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 5a6cd1f to 95ce171 Compare May 2, 2022 09:04

ebyhr mentioned this pull request May 6, 2022

Add PostgreSQL JDBC driver jar to Iceberg image trinodb/docker-images#125

Merged

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 95ce171 to 6012aeb Compare May 11, 2022 12:41

github-actions bot added the tests:hive label May 11, 2022

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch 2 times, most recently from 7a6f67d to 298e8c3 Compare May 12, 2022 00:42

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 298e8c3 to 1386ad5 Compare May 28, 2022 01:12

sungwy-backup reviewed Jun 2, 2022

View reviewed changes

...presto-product-tests/conf/environment/singlenode-spark-iceberg-jdbc-catalog/create-table.sql Outdated Show resolved Hide resolved

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 1386ad5 to 9c4cda8 Compare June 6, 2022 01:46

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 9c4cda8 to a14c771 Compare June 13, 2022 13:56

rice668 reviewed Jun 19, 2022

View reviewed changes

plugin/trino-iceberg/src/test/resources/jdbc/create-table.sql Outdated Show resolved Hide resolved

findepi reviewed Jul 20, 2022

View reviewed changes

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from a14c771 to 20f4e93 Compare July 21, 2022 02:09

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 94de1c3 to 8b536f3 Compare January 17, 2023 04:13

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from 8b536f3 to e203020 Compare January 17, 2023 09:34

findepi approved these changes Jan 17, 2023

View reviewed changes

findepi requested review from alexjo2144 and findinpath January 17, 2023 16:02

alexjo2144 reviewed Jan 17, 2023

View reviewed changes

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from e203020 to 5dd2a4a Compare January 18, 2023 08:09

bitsondatadev mentioned this pull request Jan 18, 2023

Support Nessie Catalog in Iceberg connector #11701

Merged

alexjo2144 approved these changes Jan 18, 2023

View reviewed changes

bitsondatadev approved these changes Jan 19, 2023

View reviewed changes

docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

docs/src/main/sphinx/connector/iceberg.rst Outdated Show resolved Hide resolved

Support JDBC catalog in Iceberg connector

ab3ad9b

ebyhr force-pushed the ebi/iceberg-jdbc-catalog branch from f0f86fa to ab3ad9b Compare January 19, 2023 03:29

ebyhr merged commit d868e91 into trinodb:master Jan 19, 2023

ebyhr deleted the ebi/iceberg-jdbc-catalog branch January 19, 2023 06:40

ebyhr mentioned this pull request Jan 19, 2023

Release notes for 406 #15563

Closed

github-actions bot added this to the 406 milestone Jan 19, 2023

colebow mentioned this pull request Jan 19, 2023

Add Trino 406 release notes #15625

Merged

sungwy mentioned this pull request Jan 24, 2023

Iceberg JDBC Connector: case-insensitive name matching support #15828

Open

This was referenced Feb 3, 2023

Document lack of view support in JDBC and REST catalogs. trinodb/tutorials#1

Closed

Document lack of view support in JDBC and REST catalogs #15971

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support JDBC catalog in Iceberg connector #11772

Support JDBC catalog in Iceberg connector #11772

ebyhr commented Apr 4, 2022 •

edited

Loading

sungwy-backup commented Apr 22, 2022

ebyhr commented Apr 25, 2022 •

edited

Loading

findepi commented May 4, 2022

meetsitaram commented May 4, 2022

samredai commented May 5, 2022

meetsitaram commented May 9, 2022

ebyhr commented May 12, 2022

sungwy commented Jun 13, 2022

findepi Jul 20, 2022

sungwy Jul 20, 2022

findepi Jul 20, 2022

ebyhr Jul 21, 2022

findepi Jul 21, 2022

nfcampos Jul 21, 2022

nfcampos Jul 21, 2022

ebyhr Jul 21, 2022

nfcampos Jul 21, 2022

ebyhr commented Jul 21, 2022

martint commented Jan 13, 2023

bitsondatadev commented Jan 13, 2023 •

edited

Loading

ebyhr commented Jan 17, 2023

ebyhr commented Jan 17, 2023

findepi left a comment

alexjo2144 left a comment

alexjo2144 Jan 17, 2023

ebyhr Jan 18, 2023

ebyhr commented Jan 18, 2023 •

edited

Loading

ebyhr commented Jan 18, 2023

alexjo2144 left a comment

bitsondatadev left a comment

bjackson-ep commented Feb 9, 2023

alexjo2144 commented Feb 9, 2023

bjackson-ep commented Feb 9, 2023 via email

penggewudi commented Jun 25, 2023 •

edited

Loading

Support JDBC catalog in Iceberg connector #11772

Support JDBC catalog in Iceberg connector #11772

Conversation

ebyhr commented Apr 4, 2022 • edited Loading

Description

Release notes

sungwy-backup commented Apr 22, 2022

ebyhr commented Apr 25, 2022 • edited Loading

findepi commented May 4, 2022

meetsitaram commented May 4, 2022

samredai commented May 5, 2022

meetsitaram commented May 9, 2022

ebyhr commented May 12, 2022

sungwy commented Jun 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr commented Jul 21, 2022

martint commented Jan 13, 2023

bitsondatadev commented Jan 13, 2023 • edited Loading

ebyhr commented Jan 17, 2023

ebyhr commented Jan 17, 2023

findepi left a comment

Choose a reason for hiding this comment

alexjo2144 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr commented Jan 18, 2023 • edited Loading

ebyhr commented Jan 18, 2023

alexjo2144 left a comment

Choose a reason for hiding this comment

bitsondatadev left a comment

Choose a reason for hiding this comment

bjackson-ep commented Feb 9, 2023

alexjo2144 commented Feb 9, 2023

bjackson-ep commented Feb 9, 2023 via email

penggewudi commented Jun 25, 2023 • edited Loading

ebyhr commented Apr 4, 2022 •

edited

Loading

ebyhr commented Apr 25, 2022 •

edited

Loading

bitsondatadev commented Jan 13, 2023 •

edited

Loading

ebyhr commented Jan 18, 2023 •

edited

Loading

penggewudi commented Jun 25, 2023 •

edited

Loading