Support Iceberg OPTIMIZE with WHERE casting timestamp_tz column to a date #12918

findepi · 2022-06-21T12:25:27Z

Follow-up to #12795

Among other things, that PR added support for

-- regardless of session zone
ALTER TABLR iceberg_table EXECUTE optimize WHERE c_timestamp_tz > with_timezone(a_date_constant, 'UTC');

-- only when session zone is UTC
ALTER TABLR iceberg_table EXECUTE optimize WHERE c_timestamp_tz > a_date_constant

This PR adds support for

-- regardless of session zone
ALTER TABLR iceberg_table EXECUTE optimize WHERE CAST(c_timestamp_tz AS date) > a_date_constant

Further enhances #7905
Fixes #12362

...trino-main/src/main/java/io/trino/sql/planner/iterative/rule/PushPredicateIntoTableScan.java

losipiuk · 2022-06-21T14:28:02Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestConstraintExtractor.java

+     * {@link TimestampWithTimeZoneType}. Such unwrap would not be monotonic. Within Iceberg, we know
+     * that {@link TimestampWithTimeZoneType} is always in UTC zone (point in time, with no time zone information),


For future: I wonder if we can communicate a constraint on a type from connector to engine so standard optimizer can handle that.

Yes, we could do that as part of io.trino.spi.connector.ColumnMetadata.

Or, we could use a dedicate type: #2273
A type may be better because this limitation -- storing point in time only, without time zone -- isn't really specific to Iceberg.

findepi · 2022-06-21T14:38:53Z

...trino-main/src/main/java/io/trino/sql/planner/iterative/rule/PushPredicateIntoTableScan.java

-        Map<ColumnHandle, Domain> enforcedDomains = enforcedDomainsBuilder.buildOrThrow();
-        checkArgument(
-                enforcedDomains.size() + unenforcedDomains.size() == predicateDomains.size(),
-                "Enforced tuple domain cannot be determined. Connector returned an unenforced TupleDomain %s that contains columns not in predicate %s.",


cc @martint

findepi · 2022-06-21T14:38:57Z

...trino-main/src/main/java/io/trino/sql/planner/iterative/rule/PushPredicateIntoTableScan.java

@@ -437,9 +437,8 @@ public static TupleDomain<ColumnHandle> computeEnforced(TupleDomain<ColumnHandle
            if (unenforcedDomains.containsKey(predicateColumnHandle)) {
                Domain unenforcedDomain = unenforcedDomains.get(predicateColumnHandle);
                checkArgument(
-                        predicateDomain.equals(unenforcedDomain),
-                        "Enforced tuple domain cannot be determined. The connector is expected to enforce the respective domain entirely on none, some, or all of the column. " +


cc @martint

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/ConstraintExtractor.java

When `IcebergMetadata.applyFilter` is invoked with `Constraint` that has no useful summary (`TupleDomain`) and carries only expression or functional predicate, we can short-circuit the method execution.

Also improve wording.

`unwrapTimestampToDateCast` assumes the target type is `DATE` (and is invoked only when it is), so passing `targetType` is redundant.

…rced

The check dates back to the time when `TupleDomain` was the only information passed to the connector in the `ConnectorMetadata.applyFilter`. Its purpose was to ensure the connector does not erroneously return some new column constraints (`Domain` objects) in `ConstraintApplicationResult.remainingFilter`. With expression-based pushdown, the check is no longer valid. A connector may be able to translate `Constraint.expression` (or part thereof) into a `Domain` / `TupleDomain` and then enforce it, or return such simplified representation as a remaining `TupleDomain` (`ConstraintApplicationResult.remainingFilter`).

findepi · 2022-06-21T20:55:43Z

(just rebased to resolve a conflict with #12911)

erichwang · 2022-06-22T00:34:59Z

@findepi Just a quick question:

-- regardless of session zone
ALTER TABLR iceberg_table EXECUTE optimize WHERE CAST(c_timestamp_tz AS date) > a_date_constant

This wouldn't work for Iceberg time partitions since c_timestamp_tz could be in any time zone right? I thought we had discussed making it look like:

-- regardless of session zone
ALTER TABLR iceberg_table EXECUTE optimize WHERE CAST(c_timestamp_tz AT TIME ZONE 'UTC' AS date) > a_date_constant

To force the same timezone?

Side note: that being said, it looks like if we just do with_timezone(a_date_constant, 'UTC') in our query, that might already get us what we need, making this change more of a nice to have.

findepi · 2022-06-22T08:07:29Z

Just a quick question:
-- regardless of session zone
ALTER TABLR iceberg_table EXECUTE optimize WHERE CAST(c_timestamp_tz AS date) > a_date_constant
This wouldn't work for Iceberg time partitions since c_timestamp_tz could be in any time zone right?

in Iceberg, timestamp with time zone values are all in UTC. Iceberg stores point in time, but not the time zone.

Side note: that being said, it looks like if we just do with_timezone(a_date_constant, 'UTC') in our query, that might already get us what we need, making this change more of a nice to have.

Yep, this is already supported, but may be less intuitive to users.
CAST(c_timestamp_tz AS date) > a_date_constant looks simpler than
c_timestamp_tz > with_timezone(a_date_constant, 'UTC')

fortunately, it's or, not xor. People can choose whichever they prefer.

colebow · 2022-06-22T17:49:55Z

I think we should document this so users know how to leverage this...

erichwang · 2022-06-22T18:52:14Z

Yes, definitely prefer CAST(c_timestamp_tz AS date) > a_date_constant if possible. That's how I normally think about it.

findepi requested review from martint, homar, erichwang, losipiuk, ebyhr, alexjo2144, hashhar and findinpath June 21, 2022 12:25

cla-bot bot added the cla-signed label Jun 21, 2022

This was referenced Jun 21, 2022

Ability to OPTIMIZE a single time-based partition in Iceberg #12362

Closed

Unwrap cast of timestamp with normalized zone to date in comparison #12729

Open

losipiuk reviewed Jun 21, 2022

View reviewed changes

...trino-main/src/main/java/io/trino/sql/planner/iterative/rule/PushPredicateIntoTableScan.java Outdated Show resolved Hide resolved

losipiuk reviewed Jun 21, 2022

View reviewed changes

findepi force-pushed the findepi/iceberg-subsume-with-cast branch from 9be3f52 to 412e967 Compare June 21, 2022 14:38

findepi commented Jun 21, 2022

View reviewed changes

findepi force-pushed the findepi/iceberg-subsume-with-cast branch from 412e967 to 1ad18f7 Compare June 21, 2022 14:52

losipiuk approved these changes Jun 21, 2022

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/ConstraintExtractor.java Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/ConstraintExtractor.java Show resolved Hide resolved

findepi added 7 commits June 21, 2022 22:53

Short-circuit IcebergMetadata.applyFilter when only expression given

123a7fd

When `IcebergMetadata.applyFilter` is invoked with `Constraint` that has no useful summary (`TupleDomain`) and carries only expression or functional predicate, we can short-circuit the method execution.

Fix copy & paste error in comment

6292d5d

Also improve wording.

Handle null Constant value

ed316ad

Skip passing redundant argument

d65f724

`unwrapTimestampToDateCast` assumes the target type is `DATE` (and is invoked only when it is), so passing `targetType` is redundant.

Enhance exception reporting in PushPredicateIntoTableScan.computeEnfo…

7037442

…rced

Unwrap timestamptz cast to date in Iceberg

d6f2da1

findepi force-pushed the findepi/iceberg-subsume-with-cast branch from 1ad18f7 to d6f2da1 Compare June 21, 2022 20:55

findepi merged commit 49be4c2 into trinodb:master Jun 22, 2022

findepi mentioned this pull request Jun 22, 2022

Release notes for 387 #12870

Closed

github-actions bot added this to the 387 milestone Jun 22, 2022

findepi deleted the findepi/iceberg-subsume-with-cast branch June 22, 2022 11:23

findepi mentioned this pull request Jun 22, 2022

Derive predicate over base column for CAST(a_varchar AS date) > a_date_constant condition #12925

Closed

colebow mentioned this pull request Jun 22, 2022

Add Trino 387 release notes #12881

Merged

colebow mentioned this pull request Jun 22, 2022

Document casting timestamp_tz column to a date as part of Iceberg optimize #12943

Closed

Ankan0011 mentioned this pull request Aug 20, 2023

Document casting timestamp_tz column to a date as part of Iceberg optimize #12943 #18745

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Iceberg OPTIMIZE with WHERE casting timestamp_tz column to a date #12918

Support Iceberg OPTIMIZE with WHERE casting timestamp_tz column to a date #12918

findepi commented Jun 21, 2022 •

edited

Loading

losipiuk Jun 21, 2022

findepi Jun 21, 2022

findepi Jun 21, 2022

findepi Jun 21, 2022

findepi commented Jun 21, 2022 •

edited

Loading

erichwang commented Jun 22, 2022 •

edited

Loading

findepi commented Jun 22, 2022

colebow commented Jun 22, 2022

erichwang commented Jun 22, 2022

		* {@link TimestampWithTimeZoneType}. Such unwrap would not be monotonic. Within Iceberg, we know
		* that {@link TimestampWithTimeZoneType} is always in UTC zone (point in time, with no time zone information),

Support Iceberg OPTIMIZE with WHERE casting timestamp_tz column to a date #12918

Support Iceberg OPTIMIZE with WHERE casting timestamp_tz column to a date #12918

Conversation

findepi commented Jun 21, 2022 • edited Loading

losipiuk Jun 21, 2022

Choose a reason for hiding this comment

findepi Jun 21, 2022

Choose a reason for hiding this comment

findepi Jun 21, 2022

Choose a reason for hiding this comment

findepi Jun 21, 2022

Choose a reason for hiding this comment

findepi commented Jun 21, 2022 • edited Loading

erichwang commented Jun 22, 2022 • edited Loading

findepi commented Jun 22, 2022

colebow commented Jun 22, 2022

erichwang commented Jun 22, 2022

findepi commented Jun 21, 2022 •

edited

Loading

findepi commented Jun 21, 2022 •

edited

Loading

erichwang commented Jun 22, 2022 •

edited

Loading