Feature request: Support for jdbc catalog in Iceberg Connector

[racevedoo]

Would be great if trino supported jdbc catalog in iceberg connector

[csimplestring]

+1
If jdbc catalog for Iceberg is supported, then no need to use hive metastore, which makes it way easier to use.

[racevedoo]

+1 If jdbc catalog for Iceberg is supported, then no need to use hive metastore, which makes it way easier to use.

definitely!

[Arnonrgo]

Hi - has this been abandoned?

[jebnix]

@Arnonrgo No. @ebyhr has rebased #11772 to resolve conflicts. Hope we'll see this one get merged soon. Well anticipated

[ebyhr]

@issue-subscriber We lean toward to canceling JDBC catalog because REST catalog that enables accessing JDBC catalog was already merged recently. This is not a final decision. Please let us know if JDBC catalog support is still required. We're glad if you can share why a direct JDBC catalog without REST catalog is necessary in your use case.

[Arnonrgo]

@ebyhr The point is that the REST catalog (like the HIVE catalog for that matter) requires deploying an additional component. which means maintaining another docker image, patching for CVEs and other compliance issues, making sure it is up etc. - all that for updating a couple of tables. So basically using REST is too much of an overhead with zero-value

[bitsondatadev]

Thanks for that background @Arnonrgo. That is so far the biggest issue that we see with solely relying on the REST catalog over adopting multiple catalogs (JDBC, Nessie, etc..). To add more visibility into why we are even considering forgoing JDBC for now is where we focus our resources (maintainer and contributor review time). For instance, another issue people are bringing up is our lack of focus on the Trino Helm chart which we consider a priority. So every new feature we bring on has to be adopted with some scrutiny based on the larger needs of the community.

We don't want to make this decision lightly as we understand that this becomes extra overhead for users, but it's a balancing act. I don't believe you are alone in having the concerns about overhead so please, if anyone else shares Arnon's opinion please give his comment a 👍🏻 so we can get a count rather than a bunch of +1 and "same" comments. Thank you all for understanding while we deliberate. I will be keeping you all in the loop on this.

I'm also meeting with some folks from the Iceberg community today to see if we can get to the bottom of some technical concerns around JDBC in production that our maintainers have. Will fill you all in! Thank you for your patience!

[ebyhr]

@almogtavor @jebnix @kingbrown9 @lynch19 I found you react with emoji to my above comment. Please be more verbal. We would like to hear use-cases from the community. Thumbs-down doesn't explain anything about it.

[jebnix]

@ebyhr I think there is no reason to give up on this great PR and work that have already been invested on this super anticipated feature (top 5 most liked and commented issues of Trino), just because of temporary time management issues. I don’t want to get involved with how the Trino developers time get managed, of course. But regarding the overhead of the REST Catalog I have to agree. That’s a pretty serious overhead to give to teams that just want to set up and maintain Trino for the Iceberg usage. The REST Catalog is much better than Hive metastore, but that’s still redundant overhead comparing to JDBC. A JDBC catalog is necessary because of the simplicity and support it has. The REST catalog is quite new, and I can see any situation where people would stop asking for the JDBC catalog, since it’s so ideal. From what it looks like, there is about of 80-90% of the work done. I don’t see any reason for it to get dropped. I agree that the Helm stuff is super important, but I’m not at the decision making position anyway. Will the JDBC get used? A lot.

[bitsondatadev]

To update everyone here, based on @martint's comments here, we will be continuing to work on this feature to get it merged with the caveat that it will only be supported "as-is" with no guarantee from the Trino Software Foundation about future compatibility.

cc: @jebnix, @Arnonrgo, @kingbrown9, @lynch19, @almogtavor, and others 😃