Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce LORE framework. #11084

Merged
merged 21 commits into from
Jul 2, 2024
Merged

Introduce LORE framework. #11084

merged 21 commits into from
Jul 2, 2024

Conversation

liurenjie1024
Copy link
Collaborator

Close #10987.

@liurenjie1024 liurenjie1024 marked this pull request as draft June 19, 2024 10:05
@liurenjie1024
Copy link
Collaborator Author

It's not ready yet, we still need to handle aqe correctly, but it could provide an early review of how it works.

@liurenjie1024 liurenjie1024 marked this pull request as ready for review June 25, 2024 09:21
@liurenjie1024
Copy link
Collaborator Author

build

@sameerz sameerz added the feature request New feature or request label Jun 25, 2024
@liurenjie1024
Copy link
Collaborator Author

build

@liurenjie1024
Copy link
Collaborator Author

build

@liurenjie1024
Copy link
Collaborator Author

I think this pr is ready for review.

@liurenjie1024 liurenjie1024 mentioned this pull request Jun 26, 2024
Merged
jlowe
jlowe reviewed Jun 26, 2024
View reviewed changes
Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like this needs a "LORE: What it is and how to use it" doc somewhere, maybe under docs/dev/?

sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/GpuLore.scala Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/GpuLore.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/GpuLore.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/lore/replay.scala Outdated Show resolved Hide resolved
@liurenjie1024
Copy link
Collaborator Author

cc @jlowe I've added a dev doc and fixed comments, PTAL

@liurenjie1024
Copy link
Collaborator Author

build

@liurenjie1024
Copy link
Collaborator Author

build

@liurenjie1024 liurenjie1024 merged commit 9bb295a into NVIDIA:branch-24.08 Jul 2, 2024
45 checks passed
@liurenjie1024 liurenjie1024 deleted the renjie/issue-10987 branch July 2, 2024 01:40
liurenjie1024 added a commit to liurenjie1024/spark-rapids that referenced this pull request Jul 2, 2024
@liurenjie1024
Introduce LORE framework. (NVIDIA#11084)
7e503bd 
* Introduce lore id

* Introduce lore id

* Fix type

* Fix type

* Conf

* style

* part

* Dump

* Introduce lore framework

* Add tests.

* Rename test case

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>

* Fix AQE test

* Fix style

* Use args to display lore info.

* Fix build break

* Fix path in loreinfo

* Remove path

* Fix comments

* Update configs

* Fix comments

* Fix config

---------

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
liurenjie1024 added a commit to liurenjie1024/spark-rapids that referenced this pull request Jul 2, 2024
@liurenjie1024
Introduce LORE framework. (NVIDIA#11084)
7a26525 
* Introduce lore id

* Introduce lore id

* Fix type

* Fix type

* Conf

* style

* part

* Dump

* Introduce lore framework

* Add tests.

* Rename test case

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>

* Fix AQE test

* Fix style

* Use args to display lore info.

* Fix build break

* Fix path in loreinfo

* Remove path

* Fix comments

* Update configs

* Fix comments

* Fix config

---------

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
wjxiz1992 pushed a commit to nvliyuan/yuali-spark-rapids that referenced this pull request Jul 2, 2024
@liurenjie1024
Introduce LORE framework. (NVIDIA#11084) (#40)
dc02510 
* Introduce lore id

* Introduce lore id

* Fix type

* Fix type

* Conf

* style

* part

* Dump

* Introduce lore framework

* Add tests.

* Rename test case



* Fix AQE test

* Fix style

* Use args to display lore info.

* Fix build break

* Fix path in loreinfo

* Remove path

* Fix comments

* Update configs

* Fix comments

* Fix config

---------

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
wjxiz1992 added a commit to nvliyuan/yuali-spark-rapids that referenced this pull request Jul 5, 2024
@wjxiz1992 @xieshuaihu
@mythrocks @liurenjie1024 @firestarman
Max min by (#43)
66cf815 
* Fix match error in RapidsShuffleIterator.scala [scala2.13] (NVIDIA#11115)

Signed-off-by: xieshuaihu <xieshuaihu@gmail.com>

* Spark 4: Handle ANSI mode in sort_test.py (NVIDIA#11099)

* Spark 4: Handle ANSI mode in sort_test.py

Fixes NVIDIA#11027.

With ANSI mode enabled (like the default in Spark 4), one sees that some
tests in `sort_test.py` fail, because they expect ANSI mode to be off.

This commit disables running those tests with ANSI enabled, and add a
separate test for ANSI on/off.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Refactored not to use disable_ansi_mode.

These tests need not be revisited.  They test all combinations of ANSI mode,
including overflow failures.

Signed-off-by: MithunR <mithunr@nvidia.com>

---------

Signed-off-by: MithunR <mithunr@nvidia.com>

* Introduce LORE framework. (NVIDIA#11084)

* Introduce lore id

* Introduce lore id

* Fix type

* Fix type

* Conf

* style

* part

* Dump

* Introduce lore framework

* Add tests.

* Rename test case

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>

* Fix AQE test

* Fix style

* Use args to display lore info.

* Fix build break

* Fix path in loreinfo

* Remove path

* Fix comments

* Update configs

* Fix comments

* Fix config

---------

Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>

* Support minBy on GPU

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

---------

Signed-off-by: xieshuaihu <xieshuaihu@gmail.com>
Signed-off-by: MithunR <mithunr@nvidia.com>
Signed-off-by: liurenjie1024 <liurenjie2008@gmail.com>
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
Co-authored-by: xieshuaihu <xieshuaihu@gmail.com>
Co-authored-by: MithunR <mithunr@nvidia.com>
Co-authored-by: Renjie Liu <liurenjie2008@gmail.com>
Co-authored-by: Firestarman <firestarmanllc@gmail.com>
@liurenjie1024 liurenjie1024 mentioned this pull request Jul 5, 2024
Open
@jlowe jlowe mentioned this pull request Jul 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@binmahone binmahone binmahone left review comments

@jlowe jlowe jlowe approved these changes

@winningsix winningsix Awaiting requested review from winningsix

@res-life res-life Awaiting requested review from res-life

Assignees
No one assigned
Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Implement lore framework to support all operators.
5 participants
@liurenjie1024 @jlowe @winningsix @binmahone @sameerz