Register VACUUM operations in the delta log #24331

mdesmet · 2024-12-02T08:21:12Z

Description

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(*) Release notes are required, with the following suggested text:

## Section
* Added VACUUM START AND VACUUM END logging for Delta Lake `vacuum` procedure.

findinpath · 2024-12-02T18:09:16Z

In the description "Solves" -> "Fixes".

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

findinpath · 2024-12-02T18:20:35Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java

+        return vacuumLoggingEnabled;
+    }
+
+    @Config("delta.vacuum.logging.enabled")


Why do we need a specific configuration for this ?

I've landed on delta-io/delta@3ac120f
Maybe add this commit to description of the PR for reference.

BTW Trino does support concurrent writes on all the file systems. (Databricks has not out of the box support for AWS S3, while Trino does, which means there's a potential to cause eventually table corruption in a concurrent setting).

@ebyhr : Pleae comment. I think the reasoning is that in Spark this is configurable and disabled by default.

I suggested adding this property for compatibility with Spark and also a kill-switch.

The current implementation still has possibility of conflicts. For instance, the logging may fail if there is a concurrent operation because we don't have a retry code in vacuum logic.

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

ebyhr · 2024-12-03T05:37:36Z

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java

+        return vacuumLoggingEnabled;
+    }
+
+    @Config("delta.vacuum.logging.enabled")


I suggested adding this property for compatibility with Spark and also a kill-switch.

The current implementation still has possibility of conflicts. For instance, the logging may fail if there is a concurrent operation because we don't have a retry code in vacuum logic.

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

mdesmet · 2024-12-06T07:49:40Z

There is funny failure. Why the sizeOfDataToDelete is different when running in CI vs running locally. Shouldn't it be expected to be deteriministic?

-- failure 1 --
[Rows for query [SELECT version, operation, MAP_FILTER(operation_parameters, (k, v) -> k <> 'queryId'), operation_metrics FROM "test_vacuum4b6nq4t53e$history"]] 
Expecting actual:
  (0, CREATE TABLE AS SELECT, {}, {}), (1, MERGE, {}, {}), (2, VACUUM START, {}, {sizeOfDataToDelete=0, numFilesToDelete=0}), (3, VACUUM END, {status=COMPLETED}, {numDeletedFiles=0, numVacuumedDirectories=1}), (4, VACUUM START, {}, {sizeOfDataToDelete=5012, numFilesToDelete=5}), (5, VACUUM END, {status=COMPLETED}, {numDeletedFiles=5, numVacuumedDirectories=1})
to contain exactly in any order:
  [(0, CREATE TABLE AS SELECT, {}, {}),
    (1, MERGE, {}, {}),
    (2, VACUUM START, {}, {sizeOfDataToDelete=0, numFilesToDelete=0}),
    (3, VACUUM END, {status=COMPLETED}, {numDeletedFiles=0, numVacuumedDirectories=1}),
    (4, VACUUM START, {}, {sizeOfDataToDelete=5025, numFilesToDelete=5}),
    (5, VACUUM END, {status=COMPLETED}, {numDeletedFiles=5, numVacuumedDirectories=1})]
elements not found:
  (4, VACUUM START, {}, {sizeOfDataToDelete=5025, numFilesToDelete=5})
and elements not expected:
  (4, VACUUM START, {}, {sizeOfDataToDelete=5012, numFilesToDelete=5})
at QueryAssertions$ResultAssert.lambda$matches$0(QueryAssertions$ResultAssert.java:740)

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

mdesmet · 2024-12-09T08:49:07Z

As discussed with @ebyhr offline, made use of isGreaterThanOrEqualTo.

There is funny failure. Why the sizeOfDataToDelete is different when running in CI vs running locally. Shouldn't it be expected to be deteriministic?

ebyhr · 2024-12-10T03:43:07Z

Could you confirm CI failures?

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeSessionProperties.java

ebyhr · 2024-12-16T01:25:47Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

@@ -1746,6 +1750,34 @@ public void testVacuum()
            assertThat(getActiveFiles(tableName)).isEqualTo(updatedFiles);
            // old files should be cleaned up
            assertThat(getAllDataFilesFromTableDirectory(tableName)).isEqualTo(updatedFiles);
+            // operations should be logged
+            assertThat(query("SELECT version, operation, MAP_FILTER(operation_parameters, (k, v) -> k <> 'queryId') FROM \"" + tableName + "$history\"")).matches( """
+                    VALUES\s


Remove redundant \s. Same for other places.

This line still has redundant \s.

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConfig.java

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

ebyhr · 2024-12-19T09:52:12Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

+    }
+
+    @Test
+    public void testVacuumWithParquetTransactionLogging()


Parquet is used for both data files and checkpoints. Please rename to testVacuumTransactionLoggingOnCheckpoint or something.

ebyhr · 2024-12-19T09:52:27Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

+        testVacuumWithCheckPointInterval(true);
+    }
+
+    private void testVacuumWithCheckPointInterval(boolean isCheckPointInterval) throws InterruptedException {


Put throws InterruptedException and { on new lines.

ebyhr · 2024-12-19T09:52:41Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

@@ -1746,6 +1750,34 @@ public void testVacuum()
            assertThat(getActiveFiles(tableName)).isEqualTo(updatedFiles);
            // old files should be cleaned up
            assertThat(getAllDataFilesFromTableDirectory(tableName)).isEqualTo(updatedFiles);
+            // operations should be logged
+            assertThat(query("SELECT version, operation, MAP_FILTER(operation_parameters, (k, v) -> k <> 'queryId') FROM \"" + tableName + "$history\"")).matches( """
+                    VALUES\s


This line still has redundant \s.

ebyhr · 2024-12-19T09:56:32Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

@@ -1746,6 +1761,34 @@ public void testVacuum()
            assertThat(getActiveFiles(tableName)).isEqualTo(updatedFiles);
            // old files should be cleaned up
            assertThat(getAllDataFilesFromTableDirectory(tableName)).isEqualTo(updatedFiles);
+            // operations should be logged
+            assertThat(query("SELECT version, operation, MAP_FILTER(operation_parameters, (k, v) -> k <> 'queryId') FROM \"" + tableName + "$history\"")).matches( """


Please follow a text block format of #23958.

cla-bot bot added the cla-signed label Dec 2, 2024

github-actions bot added the delta-lake Delta Lake connector label Dec 2, 2024

mdesmet requested review from pajaks, ebyhr and findinpath December 2, 2024 08:21

mdesmet force-pushed the feat/register-vacuum-delta-log branch 2 times, most recently from aa39226 to 2008d86 Compare December 2, 2024 14:26

findinpath reviewed Dec 2, 2024

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java Outdated Show resolved Hide resolved

findinpath reviewed Dec 2, 2024

View reviewed changes

ebyhr reviewed Dec 3, 2024

View reviewed changes

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 2008d86 to 1ae966f Compare December 3, 2024 07:07

mdesmet commented Dec 3, 2024

View reviewed changes

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java Outdated Show resolved Hide resolved

mdesmet commented Dec 3, 2024

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java Outdated Show resolved Hide resolved

mdesmet marked this pull request as ready for review December 3, 2024 13:41

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 1ae966f to 868ae1a Compare December 3, 2024 13:45

ebyhr reviewed Dec 4, 2024

View reviewed changes

mdesmet force-pushed the feat/register-vacuum-delta-log branch 3 times, most recently from 722a27f to 44324e9 Compare December 4, 2024 08:47

mdesmet mentioned this pull request Dec 4, 2024

Add operationMetrics to CommitInfoEntry in Delta Lake #24368

Merged

mdesmet force-pushed the feat/register-vacuum-delta-log branch 3 times, most recently from c6fae4a to 6c315fa Compare December 5, 2024 10:03

mdesmet requested review from findinpath and ebyhr December 5, 2024 10:05

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 6c315fa to 2b0f6d5 Compare December 5, 2024 10:07

mdesmet commented Dec 5, 2024

View reviewed changes

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java Outdated Show resolved Hide resolved

pajaks approved these changes Dec 5, 2024

View reviewed changes

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 2b0f6d5 to 0806741 Compare December 6, 2024 06:46

pajaks reviewed Dec 6, 2024

View reviewed changes

pajaks self-requested a review December 6, 2024 08:28

mdesmet force-pushed the feat/register-vacuum-delta-log branch 2 times, most recently from 0dee91e to 31bc519 Compare December 9, 2024 08:48

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 31bc519 to f4cf893 Compare December 9, 2024 10:14

mdesmet force-pushed the feat/register-vacuum-delta-log branch from f4cf893 to e65ea00 Compare December 10, 2024 04:24

pajaks reviewed Dec 10, 2024

View reviewed changes

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/procedure/VacuumProcedure.java Outdated Show resolved Hide resolved

plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java Outdated Show resolved Hide resolved

mdesmet force-pushed the feat/register-vacuum-delta-log branch 3 times, most recently from ce141ba to e94c13a Compare December 15, 2024 03:35

ebyhr reviewed Dec 16, 2024

View reviewed changes

mdesmet force-pushed the feat/register-vacuum-delta-log branch from e94c13a to 68c1e7c Compare December 17, 2024 12:57

pajaks reviewed Dec 17, 2024

View reviewed changes

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java Outdated Show resolved Hide resolved

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 68c1e7c to 24955a0 Compare December 18, 2024 02:58

mdesmet requested review from pajaks and ebyhr December 18, 2024 05:10

ebyhr reviewed Dec 18, 2024

View reviewed changes

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java Outdated Show resolved Hide resolved

mdesmet added 2 commits December 19, 2024 11:03

Split vacuum work definition and execution

e0234c7

Register VACUUM operations in the delta log

aad54e9

mdesmet force-pushed the feat/register-vacuum-delta-log branch from 24955a0 to aad54e9 Compare December 19, 2024 06:50

pajaks approved these changes Dec 19, 2024

View reviewed changes

mdesmet requested a review from ebyhr December 19, 2024 09:30

ebyhr reviewed Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Register VACUUM operations in the delta log #24331

Register VACUUM operations in the delta log #24331

mdesmet commented Dec 2, 2024 •

edited

Loading

findinpath commented Dec 2, 2024

findinpath Dec 2, 2024 •

edited

Loading

mdesmet Dec 3, 2024

ebyhr Dec 3, 2024 •

edited

Loading

ebyhr Dec 3, 2024 •

edited

Loading

mdesmet commented Dec 6, 2024

mdesmet commented Dec 9, 2024

ebyhr commented Dec 10, 2024

ebyhr Dec 16, 2024

ebyhr Dec 19, 2024

ebyhr Dec 19, 2024

ebyhr Dec 19, 2024

ebyhr Dec 19, 2024

ebyhr Dec 19, 2024

Register VACUUM operations in the delta log #24331

Are you sure you want to change the base?

Register VACUUM operations in the delta log #24331

Conversation

mdesmet commented Dec 2, 2024 • edited Loading

Description

Additional context and related issues

Release notes

findinpath commented Dec 2, 2024

findinpath Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

mdesmet Dec 3, 2024

Choose a reason for hiding this comment

ebyhr Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

ebyhr Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

mdesmet commented Dec 6, 2024

mdesmet commented Dec 9, 2024

ebyhr commented Dec 10, 2024

ebyhr Dec 16, 2024

Choose a reason for hiding this comment

ebyhr Dec 19, 2024

Choose a reason for hiding this comment

ebyhr Dec 19, 2024

Choose a reason for hiding this comment

ebyhr Dec 19, 2024

Choose a reason for hiding this comment

ebyhr Dec 19, 2024

Choose a reason for hiding this comment

ebyhr Dec 19, 2024

Choose a reason for hiding this comment

mdesmet commented Dec 2, 2024 •

edited

Loading

findinpath Dec 2, 2024 •

edited

Loading

ebyhr Dec 3, 2024 •

edited

Loading

ebyhr Dec 3, 2024 •

edited

Loading