You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ScalarFunction interface, which has practical description of what is needed for codegeneration and the benefits of it.
The functions we have that are likely highest priority:
truncate
bucket
zorder
date transformations
The text was updated successfully, but these errors were encountered:
kbendick
changed the title
Implement Spark’s FunctionCatalog for Existing Transform functions
Implement Spark’s FunctionCatalog for Existing Transformations
Jul 24, 2022
This will allow us to make use of Spark’s storage partitioned joins (aka bucket joins which is one subset of possible join optimizations of transformed columns) https://issues.apache.org/jira/browse/SPARK-37166
We need to implement Spark’s
FunctionCatalog
so that we can use the partition transformation functions in queries.This allows for using the partition transforms on non-partition columns in generated code.
This is necessary in order to write Catalyst rules which will pass
bucket
So that storage partitioned joins (aka bucketed joins) can be implemented.See also:
The functions we have that are likely highest priority:
The text was updated successfully, but these errors were encountered: