Flush transaction log cache in Delta `flush_metadata_cache` procedure #16466

ebyhr · 2023-03-09T10:06:30Z

Description

Relates to #13737

Release notes

(x) Release notes are required, with the following suggested text:

# Delta Lake
* Flush internal caches of table snapshots and active data files with `flush_metadata_cache` procedure. ({issue}`16466`)

...duct-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeActiveFilesCache.java

...src/main/java/io/trino/plugin/deltalake/metastore/HiveMetastoreBackedDeltaLakeMetastore.java

...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/DecoratedHiveMetastoreModule.java

findinpath · 2023-03-09T13:51:22Z

...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/DecoratedHiveMetastoreModule.java

@@ -56,7 +68,9 @@ protected void setup(Binder binder)
        newExporter(binder).export(HiveMetastoreFactory.class)
                .as(generator -> generator.generatedNameOf(CachingHiveMetastore.class));

-        newSetBinder(binder, Procedure.class).addBinding().toProvider(FlushHiveMetastoreCacheProcedure.class).in(Scopes.SINGLETON);
+        if (installFlushMetadataCacheProcedure) {


Do we want this procedure installed for iceberg ?
tbh, I didn't know that it existed in iceberg already.

I'd rather have the method always available in the connector (register it on HiveProcedureModule) and throw an exception (as your implementation does for delta) in case that the caching metastore is not used.

Added another commit to avoid installing the procedure in Iceberg.

I'd rather have the method always available in the connector

What's the rationale or motivation of this? My opinion is basically the opposite (no need to install a procedure when it's unusable).

What's the rationale or motivation of this?

I was thinking more from a user perspective where a user may be confused that the procedure is not available (e.g. : while using AWS Glue metastore) for the hive/delta connector even though is advertised to be available in the documentation.

...ino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/FlushMetadataProcedure.java

...elta-lake/src/main/java/io/trino/plugin/deltalake/procedure/FlushMetadataCacheProcedure.java

...a-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeFlushMetadataCacheProcedure.java

...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/DecoratedHiveMetastoreModule.java

findepi · 2023-03-17T14:15:32Z

...in/trino-hive/src/main/java/io/trino/plugin/hive/metastore/DecoratedHiveMetastoreModule.java

@@ -42,6 +42,18 @@
 public class DecoratedHiveMetastoreModule
        extends AbstractConfigurationAwareModule
 {
+    private final boolean installFlushMetadataCacheProcedure;


the flush procedure should be installed whenever caching is supported

what about changing the parameter name and logic.
for example, we could skip binding CachingHiveMetastoreConfig for iceberg (and maybe some others)

the flush procedure should be installed whenever caching is supported

This makes it hard to exclude Hive's flush_metadata_cache from Delta Lake connector. I initially tried to exclude the procedure from DeltaLakeConnector#getProcedures, but Procedure class doesn't contain catalog or connector information (MethodHandle doesn't work nicely to compare objects).

findepi · 2023-03-17T14:16:45Z

...elta-lake/src/main/java/io/trino/plugin/deltalake/procedure/FlushMetadataCacheProcedure.java

+            TransactionLogAccess transactionLogAccess)
+    {
+        this.metastoreFactory = requireNonNull(metastoreFactory, "metastoreFactory is null");
+        this.cachingHiveMetastore = requireNonNull(cachingHiveMetastore, "cachingHiveMetastore is null");


would it make sense to try to delegate to Hive's flush procedure?

I tried to delegate to the connector at first, but the code was a little redundant. Let me merge as-is. I will take another look later.

findepi · 2023-03-17T14:17:39Z

...duct-tests/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeActiveFilesCache.java

+import static io.trino.tests.product.utils.QueryExecutors.onTrino;
+import static java.lang.String.format;
+
+public class TestDeltaLakeActiveFilesCache


do we need a product test?
maybe file: -based tests are enough?

FileHiveMetastore has the additional cache with listTablesCache #13115. It requires a refactoring to disable the cache (the duration is hard-coded now) or wait until the internal cache is expired.

Additionally, replace toUnmodifiableList with toImmutableList.

ebyhr · 2023-03-20T03:00:53Z

Rebased on master to resolve conflicts.

The procedure was unusable because Iceberg connector always disables caching metastore.

Co-Authored-By: Marius Grama <findinpath@gmail.com>

cla-bot bot added the cla-signed label Mar 9, 2023

ebyhr changed the title ~~Ebi/delta flush cache~~ Flush transaction log cache in Delta flush_metadata_cache procedure Mar 9, 2023

ebyhr force-pushed the ebi/delta-flush-cache branch from b5015ef to aa7008d Compare March 9, 2023 11:05

github-actions bot added delta-lake Delta Lake connector hive Hive connector tests:hive labels Mar 9, 2023

findinpath reviewed Mar 9, 2023

View reviewed changes

pajaks reviewed Mar 9, 2023

View reviewed changes

...ino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/FlushMetadataProcedure.java Outdated Show resolved Hide resolved

ebyhr force-pushed the ebi/delta-flush-cache branch from aa7008d to 8e50366 Compare March 10, 2023 00:41

github-actions bot added the iceberg Iceberg connector label Mar 10, 2023

ebyhr force-pushed the ebi/delta-flush-cache branch 2 times, most recently from 0e7d6ac to 266aec9 Compare March 13, 2023 23:57

ebyhr self-assigned this Mar 14, 2023

findinpath reviewed Mar 14, 2023

View reviewed changes

...ino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/FlushMetadataProcedure.java Outdated Show resolved Hide resolved

...ino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/FlushMetadataProcedure.java Outdated Show resolved Hide resolved

ebyhr force-pushed the ebi/delta-flush-cache branch from 266aec9 to 5a09dfb Compare March 14, 2023 10:20

findinpath reviewed Mar 14, 2023

View reviewed changes

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

github-actions bot added the docs label Mar 14, 2023

ebyhr force-pushed the ebi/delta-flush-cache branch from 5a09dfb to e08137e Compare March 15, 2023 00:12

findinpath approved these changes Mar 16, 2023

View reviewed changes

ebyhr requested a review from findepi March 17, 2023 04:42

findepi approved these changes Mar 17, 2023

View reviewed changes

ebyhr added 3 commits March 20, 2023 11:26

Remove unused argument from DeltaLakeMetastore.getTableLocation

4e5f874

Extract removeS3Directory into DeltaLakeTestUtils

f53ca0e

Additionally, replace toUnmodifiableList with toImmutableList.

Extract method in HiveMetastoreBackedDeltaLakeMetastore

fc0fd46

ebyhr force-pushed the ebi/delta-flush-cache branch from 7892030 to 118e67d Compare March 20, 2023 03:00

ebyhr force-pushed the ebi/delta-flush-cache branch 3 times, most recently from c6ae7f8 to 5bb3280 Compare March 22, 2023 05:07

ebyhr and others added 2 commits March 22, 2023 15:17

Don't install flush_metadata_cache procedure in Iceberg

7158624

The procedure was unusable because Iceberg connector always disables caching metastore.

Flush transaction log cache in Delta flush_metadata_cache procedure

3f88633

Co-Authored-By: Marius Grama <findinpath@gmail.com>

ebyhr force-pushed the ebi/delta-flush-cache branch from 5bb3280 to 3f88633 Compare March 22, 2023 06:17

ebyhr merged commit 1e5774d into trinodb:master Mar 22, 2023

ebyhr deleted the ebi/delta-flush-cache branch March 22, 2023 12:27

ebyhr mentioned this pull request Mar 22, 2023

Release notes for 411 #16453

Closed

github-actions bot added this to the 411 milestone Mar 22, 2023

findepi mentioned this pull request Mar 22, 2023

Refactor Delta's DROP TABLE support for corrupted tables #16651

Merged

colebow mentioned this pull request Mar 29, 2023

Add Trino 411 release notes #16552

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flush transaction log cache in Delta `flush_metadata_cache` procedure #16466

Flush transaction log cache in Delta `flush_metadata_cache` procedure #16466

ebyhr commented Mar 9, 2023 •

edited

Loading

findinpath Mar 9, 2023

findinpath Mar 9, 2023

ebyhr Mar 10, 2023 •

edited

Loading

findinpath Mar 10, 2023

findepi Mar 17, 2023

ebyhr Mar 20, 2023 •

edited

Loading

findepi Mar 17, 2023

ebyhr Mar 22, 2023

findepi Mar 17, 2023

ebyhr Mar 17, 2023

ebyhr commented Mar 20, 2023

Flush transaction log cache in Delta flush_metadata_cache procedure #16466

Flush transaction log cache in Delta flush_metadata_cache procedure #16466

Conversation

ebyhr commented Mar 9, 2023 • edited Loading

Description

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr Mar 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr Mar 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebyhr commented Mar 20, 2023

Flush transaction log cache in Delta `flush_metadata_cache` procedure #16466

Flush transaction log cache in Delta `flush_metadata_cache` procedure #16466

ebyhr commented Mar 9, 2023 •

edited

Loading

ebyhr Mar 10, 2023 •

edited

Loading

ebyhr Mar 20, 2023 •

edited

Loading