Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Iceberg Glue and Hadoop catalog support #9646

Closed
wants to merge 1 commit into from

Conversation

jackye1995
Copy link
Member

@jackye1995 jackye1995 commented Oct 15, 2021

Draft based on #9584 and #9614 for adding Glue and Hadoop catalog support.
Refactoring step is specified in https://docs.google.com/document/d/1GCOzfMCofQ87ZLyi1_3yL4P3Jn3D2lKgztek9KfE1Mo/edit#heading=h.eei1d7r863wy

@cla-bot cla-bot bot added the cla-signed label Oct 15, 2021
@jackye1995 jackye1995 marked this pull request as draft October 15, 2021 02:52
@jackye1995
Copy link
Member Author

Marking as draft for now, I will also publish the Hadoop catalog part here.

I think after the 2 PRs linked above are merged, 1 more PR is needed for refactoring the catalog factory, then 2 more PRs for Glue and Hadoop sounds like the right division. Please let me know what you think.

@losipiuk @electrum @findepi

@jackye1995 jackye1995 changed the title Add Iceberg Glue catalog support Add Iceberg Glue and Hadoop catalog support Oct 15, 2021
@haldes
Copy link
Contributor

haldes commented Dec 2, 2021

Thanks, @jackye1995 for this PR. We are looking forward to this feature. Any high-level timelines will help. Thanks again.

@jackye1995
Copy link
Member Author

@haldes working on this right now, will publish in a few hours.

@jackye1995
Copy link
Member Author

published #10151 for anyone interested, that patch can be used if anyone want to directly use it while we are reviewing, I have ran all tests and Glue tests against it.

@RameshByndoor
Copy link
Member

@jackye1995 thanks, Hadoop catalog support is also getting along with this.? Any updates on that would be helpful.

@jackye1995
Copy link
Member Author

Hadoop catalog support is also getting along with this

I am noticing that people are starting to abuse the use of HadoopCatalog. HadoopCatalog was developed in Iceberg mostly for testing purpose, and is never recommended for production use because it has a lot of limitations. I am discussing with the community to see what is the best way go proceed forward, maybe it's not a good idea to continue adding support for it, let's see.

@cccs-tom
Copy link
Member

cccs-tom commented Dec 3, 2021

@jackye1995 I'm definitely biased in this, but we have a real use case that requires our data (including catalog / metastore) to be self-contained on our datalake and not depend on an external service of any kind. We chose Hadoop catalog because it seemed to be the simplest option, but if there are any options that we may have overlooked, I'd love to hear about them.

Having said that, I would also argue that I prefer having the Hadoop catalog option implemented, with documentation that states "not recommended for prod - use at your own risk". That at least gives us a jumping-off point to hopefully improve both Iceberg and Trino to a point where maybe Hadoop catalog becomes viable for prod. Just my 2¢...

@haldes
Copy link
Contributor

haldes commented Dec 5, 2021

@jackye1995 agree with @cccs-tom as we also have a use case in which we will need Hadoop catalog. Unfortunately, we are in a situation where we will not be able to access an external HMS and this feature will really help us.

@RameshByndoor
Copy link
Member

RameshByndoor commented Dec 7, 2021

Hadoop catalog support is also getting along with this

I am noticing that people are starting to abuse the use of HadoopCatalog. HadoopCatalog was developed in Iceberg mostly for testing purpose, and is never recommended for production use because it has a lot of limitations. I am discussing with the community to see what is the best way go proceed forward, maybe it's not a good idea to continue adding support for it, let's see.

@jackye1995 With majority of the functionalities being supported and community is already using it in hadoop catalog mode, There is no point restricting support only from trino connector. I would say Trino must be open to add hadoop(any iceberg supported ) catalog inherently and iceberg to callout limitations of using each catalog in it's document. would love to hear if any callouts which is very specific by adding it to Trino.

@jackye1995
Copy link
Member Author

for context, the main conversation happened in the ASF slack Iceberg channel, link: https://the-asf.slack.com/archives/CF01LKV9S/p1638208838030200

Conversation:

Suraj Chandran Nov 29th at 10:00 AM
So this might sound like a weird scenario, but we ended up with a scenario where we are writing all our iceberg data using Hive catalog but want to read it using both Hive as well as Hadoop catalog.
So the scenario is that we have been using all the Iceberg data with a HMS(hive metastore) in on-premise which works fine. Now we are exporting this data to cloud, but for whatever reason the HMS won't be accessible from cloud.
We already tried reading the iceberg data from Cloud using Hadoop catalog, which obviously doesn't work, because Hive catalog doesn't write version-hint.txt and the format is basically different.
Have others faced similar scenarios?
P.S: BTW, the reason we were using Hive catalog, was that Presto needs Hive Catalog to work. There is an open issue for supporting Hadoop catalog in Presto, but it will take its own due time. (edited) 
#5571 Add support for HDFS only iceberg tables
In order to keep list of tables and partitions Iceberg can be configured to use a Hive metastore however it can also be configured to store that information on files and thus not require running a Hive metastore.
This question was asked in a previous post.
@lxynov Is it planned to add support of hdfs only iceberg tables (like in spark https://iceberg.apache.org/spark/ &spark.sql.catalog.hadoop_prod.type = hadoop ) ?
Originally posted by @AbdullaevAPo in https://github.com/prestosql/presto/issue_comments/707585664
I've looked at the implementation of the PrestoSQL file IcebergMetadata. That class looks to be the interface which Presto uses to obtain the iceberg tables and schemas. Currently t… Show more
Labels
enhancement
Comments
1
<https://github.com/trinodb/trino|trinodb/trino>trinodb/trino | Oct 16th, 2020 | Added by GitHub
15 replies

Jack Ye  8 days ago
if you are talking about PrestoDB, we have added support for Hadoop using native mode. For Trino, I am in the progress of adding Hadoop catalog, that is my top priority in December, should be in very soon.
https://github.com/prestodb/presto/pull/16612
https://github.com/trinodb/trino/pull/9646
I would not suggest mixing the 2 catalog usages, there are ways to hack things around, but it’s just gonna make your life harder in long term.

#16612 Refactor Iceberg connector to support Iceberg native catalogs
<https://github.com/prestodb/presto|prestodb/presto>prestodb/presto | Aug 15th | Added by GitHub

#9646 Add Iceberg Glue and Hadoop catalog support
<https://github.com/trinodb/trino|trinodb/trino>trinodb/trino | Oct 14th | Added by GitHub

Jack Ye  8 days ago
btw, we have moved to a dedicated Apache Iceberg slack workspace: http://apache-iceberg.slack.com/

Jack Ye  8 days ago
https://join.slack.com/t/apache-iceberg/shared_invite/zt-tlv0zjz6-jGJEkHfb1~heMCJA3Uycrg

Suraj Chandran  8 days ago
Thanks much Jack!! This is great. This makes sense. We will follow the PR for Hadoop catalog support in presto

Suraj Chandran  8 days ago
And thanks for the new slack invite

Ryan Blue  7 days ago
I really would not recommend using the Hadoop catalog for anything but testing. It's great and light-weight for testing, but you want a real metastore for just about everything else

Ryan Blue  7 days ago
What about moving the Hive Metastore to the cloud and using it from both? That seems ideal to me

Ryan Blue  7 days ago
Another option is to build something around the JDBC Catalog where every time you sync a new table version to cloud storage, you also update the metadata pointer in the JDBC catalog.

Ryan Blue  7 days ago
You'd also want to disallow writes so they don't get out of sync, which would be easier with that approach than with the Hadoop catalog.

Jack Ye  7 days ago
 I really would not recommend using the Hadoop catalog for anything but testing
yes I think that’s what we have been advocating. However, I recently see an increasing demand for Hadoop catalog for production use, many people see it convenient to have a catalog fully in the storage system, and also some companies are advocating for that. I’ve heard feature requests also for a S3Catalog.

Ryan Blue  7 days ago
I have no idea how we'd make an S3 catalog work. We really should push back on this because it's a bad idea. People need better metadata in general, rather than getting rid of it.

Jack Ye  7 days ago
It’s not about getting rid of metadata, people are proposing to use a file system to implement all the catalog interfaces, including the storage of metadata maybe as a file. It is tempting to use Hadoop catalog for production, if we suggest people to use it for testing and then users see no blocker for production usage. Maybe what we should do is to prioritize the in-memory catalog, and use it for all the tests and examples.

findepi  7 days ago
I see how filesystem-only solution looks great to users. We’ve seen this in Trino with it’s file metastore implementation, which is for testing only, but still tried for production by some.
However, even though  it was a repeating story, i wouldn’t call it significant — it’s rather very small percentage of user-base. It’s sociological — some people refuse to listen when you tell them things not gonna work, and will insist on using them until they get burns…   It remains project’s responsibility to spend time on things that are viable for long-term, and filter out requests that are awesome at the face value only.
  I have no idea how we’d make an S3 catalog work.
This might be doable actually. At Starburst, we have experimental writer support for Delta Lake on S3, which requires similar compare-and-set semantics, leveraging strong consistency. Similar != same though, so it might not be applicable for Iceberg.
However, even if it becomes technically possible,  that doesn’t mean it’s generally recommendable. I would prefer people to use Glue, or generally a small number of catalogs that work great. Having plethora of options means that each of the option is not as well supported.

Jack Ye  7 days ago
interesting, S3 does not have mutual exclusion for file writing, I think that’s what is needed, how could strong consistency help?

Jack Ye  7 days ago
But yes I totally agree with the statement that even if it becomes technically possible, that doesn’t mean it’s generally recommendable.

I think as @findepi said, we cannot really stop people if there is a strong demand for Hadoop catalog, so adding the support and marking it as not recommended would likely be the best way forward.

@jackye1995
Copy link
Member Author

@RameshByndoor @haldes @cccs-tom had a discussion around this topic in the Iceberg community sync yesterday, I think we had some agreements:

(1) we will continue to add support for Hadoop catalog in Trino. We cannot stop people from adding it anyway. I was more thinking about if we can build support based on the FileMetastore available already in Trino, but I think there are still some key semantic differences between the two.
(2) we will try to make the tradeoffs more clear on Iceberg side in documentation to make sure people make the choice knowing all the implications.
(3) some locking related features will be developed to ensure mutual exclusion of file write if people want to use Hadoop catalog in cloud storage like S3

@findepi please let me know when we could make some progress in #10151, after that Hadoop catalog support will be a short PR that I already have locally.

@cccs-tom
Copy link
Member

That sounds amazing. Thanks again for all your hard work @jackye1995.

@haldes
Copy link
Contributor

haldes commented Dec 14, 2021

Thanks, @jackye1995

@findepi
Copy link
Member

findepi commented Dec 16, 2021

Superseded by #10151 + potential future PR with hadoop

@findepi findepi closed this Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

None yet

5 participants