Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add array support for round robin partition; Refactor pluginSupportedOrderableSig [databricks] #9067

Merged
merged 3 commits into from
Aug 18, 2023

Conversation

res-life
Copy link
Collaborator

closes #7807

  • Refactor pluginSupportedOrderableSig:
    All the references of pluginSupportedOrderableSig are:
RapidsDeltaUtils.scala:
  val orderableTypeSig = (GpuOverrides.pluginSupportedOrderableSig + TypeSig.DECIMAL_128
    + TypeSig.STRUCT).nested()

expr[SortOrder]:
	(pluginSupportedOrderableSig + TypeSig.DECIMAL_128 + TypeSig.STRUCT).nested() +
		TypeSig.ARRAY.nested(_gpuCommonTypes + TypeSig.DECIMAL_128)
  extra check: STRUCT is not supported as a child type for ARRAY

part[RangePartitioning]
	(pluginSupportedOrderableSig + TypeSig.DECIMAL_128 + TypeSig.STRUCT).nested() +
		TypeSig.ARRAY.nested(_gpuCommonTypes + TypeSig.DECIMAL_128)

// The SortOrder TypeSig will govern what types can actually be used as sorting key data type.
exec[TakeOrderedAndProjectExec]
	(pluginSupportedOrderableSig + TypeSig.DECIMAL_128 +
          TypeSig.ARRAY + TypeSig.STRUCT + TypeSig.MAP).nested()

// The SortOrder TypeSig will govern what types can actually be used as sorting key data type.
exec[TakeOrderedAndProjectExec]	Spark340
	(pluginSupportedOrderableSig + TypeSig.DECIMAL_128 +
		        TypeSig.ARRAY + TypeSig.STRUCT + TypeSig.MAP).nested()

// The SortOrder TypeSig will govern what types can actually be used as sorting key data type.
exec[SortExec]
	(pluginSupportedOrderableSig + TypeSig.DECIMAL_128 + TypeSig.ARRAY +
          TypeSig.STRUCT +TypeSig.MAP + TypeSig.BINARY).nested()

aggregate.scala
  private val orderable =
    (pluginSupportedOrderableSig + TypeSig.DECIMAL_128 + TypeSig.STRUCT).nested()

GpuShuffleExchangeExecBase.scala
  val orderableTypes = GpuOverrides.pluginSupportedOrderableSig + TypeSig.DECIMAL_128

ZOrderRules.scala
expr[UnaryExpression]
        // Same as RangePartitioningExec
        (pluginSupportedOrderableSig + TypeSig.DECIMAL_128 + TypeSig.STRUCT).nested()

Originally pluginSupportedOrderableSig is _gpuCommonTypes + TypeSig.STRUCT.nested(_gpuCommonTypes)
All the above references except GpuShuffleExchangeExecBase use (... + TypeSig.STRUCT).nested(), so we can referator
pluginSupportedOrderableSig to (_gpuCommonTypes + TypeSig.STRUCT).nested

  • all the references of _gpuCommonTypes plus decimal128, so put decimal128 into _gpuCommonTypes. Modify _gpuCommonTypes to public access

  • Enable array(gpuCommonTypes) type in GpuShuffleExchangeExecBase. And add test cases for this feature.

Signed-off-by: Chong Gao res_life@163.com

…OrderableSig

Signed-off-by: Chong Gao <res_life@163.com>
@res-life res-life changed the title Add array support for round robin partition; Refactor pluginSupportedOrderableSig [WIP] Add array support for round robin partition; Refactor pluginSupportedOrderableSig Aug 17, 2023
@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

Will change this PR to "Ready for review" after the permerge is passed.

@res-life
Copy link
Collaborator Author

Also closes this issue: #8686

Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

integration_tests/src/main/python/repart_test.py Outdated Show resolved Hide resolved
@sameerz sameerz added the bug Something isn't working label Aug 17, 2023
@res-life res-life changed the title [WIP] Add array support for round robin partition; Refactor pluginSupportedOrderableSig [WIP] Add array support for round robin partition; Refactor pluginSupportedOrderableSig [databricks] Aug 18, 2023
@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@res-life res-life changed the title [WIP] Add array support for round robin partition; Refactor pluginSupportedOrderableSig [databricks] Add array support for round robin partition; Refactor pluginSupportedOrderableSig [databricks] Aug 18, 2023
@res-life res-life marked this pull request as ready for review August 18, 2023 10:16
@revans2 revans2 merged commit 28bb659 into NVIDIA:branch-23.10 Aug 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Round robin partitioning sort check falls back to CPU for cases that can be supported
3 participants