[WIP] [Coral-Hive] Make named_struct a Coral IR operator #412
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request, and why are they necessary?
This patch removes
HiveConvertletTable
implementation of theCAST
operator that convertnamed_struct
toCAST (ROW() AS ROW())
. Instead, it makesnamed_struct
a Coral IR operator. Engine translations on the RHS are also adapted to accommodate this change. This also eliminates the need to rewrite fromCAST (ROW() AS ROW())
tonamed_struct
on the Spark side, becausenamed_struct
is now maintained all along. I have not removed that transformation yet, but will remove either in this PR or a future PR.This PR is also introduces a Trino converter that takes the
SqlValidator
as an input so we can do type derivation on the RHS. This requires changes in engine integrations, and hence this PR is not ready to merge (unless we somehow are able to create theSqlValidator
from scratch without having to change the API to pass it on).Furthermore, this PR fixes the traversal logic in
Coral-Trino
transformer shuttle, and addsvisited
logic to transformers potentially resulting in infinite recursion so that they are called exactly once perSqlCall
.This PR should address #357.
How was this patch tested?
./gradlew build