Add support for BigQuery Metastore

[brokenjacobs]

Google now has a serverless iceberg custom catalog available called biglake metastore. Details here:
https://cloud.google.com/bigquery/docs/manage-open-source-metadata

They package a jar for hive/spark support. I was wondering if it would be possible for trino to support this catalog as well?

I've been trying to find a maven dependency to include this custom catalog and I have been unable to find it. I'm guessing this is why it's not implemented already?

[brokenjacobs]

@anoopj any chance of this getting implemented?

[anoopj]

@brokenjacobs We're working on a more feature-rich iteration of BigLake Metastore that is unified with BigQuery called BigQuery metastore. We expect this to be ready in the next few months and could be integrated into Trino. It will support any table format, not just Iceberg.

Please contact us at biglake-help@google.com with the project IDs if you would like to learn more or try out this feature.

[vlad-lyutenko]

@anoopj
Hi, I am working on adding support for BigLake metastore for Trino, PR will be ready soon.
So the main question is will it be a replacement of BigLake metastore and we should concentrate on this new BigQuery metastore and skip BigLake?

[anoopj]

will it be a replacement of BigLake metastore and we should concentrate on this new BigQuery metastore and skip BigLake?

Thank you for your contribution @vlad-lyutenko! Yes, BigQuery Metastore is the replacement for BigLake Metastore, so please skip the BigLake one.

[vlad-lyutenko]

Thank you for your contribution @vlad-lyutenko! Yes, BigQuery Metastore is the replacement for BigLake Metastore, so please skip the BigLake one.

Big thanks for response.

Maybe you have already some API jar like:

            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-biglake</artifactId>
            <version>0.34.0</version>

, which we use for BigLake integration now.
Or it just Rest API ?

[anoopj]

BigQuery Metastore follows the same API as BigQuery v2 REST API. From Java, it's just a Table with an ExternalCatalogTableOptions. Javadocs

We will be updating java-bigquery client library also to support this in future.

Please feel free to ask any questions in the email thread you opened to biglake-help@

[hesham-medhat]

This PR could definitely provide inspiration, from the main Iceberg repo: apache/iceberg#11039