Skip to content

Commit

Permalink
[SPARK-7267][SQL]Push down Project when it's child is Limit
Browse files Browse the repository at this point in the history
SQL
```
select key from (select key,value from t1 limit 100) t2 limit 10
```
Optimized Logical Plan before modifying
```
== Optimized Logical Plan ==
Limit 10
  Project key#228
    Limit 100
      MetastoreRelation default, t1, None
```
Optimized Logical Plan after modifying
```
== Optimized Logical Plan ==
Limit 10
  Limit 100
    Project key#228
      MetastoreRelation default, t1, None
```
After this, we can combine limits

Author: Zhongshuai Pei <799203320@qq.com>
Author: DoingDone9 <799203320@qq.com>

Closes apache#5797 from DoingDone9/ProjectLimit and squashes the following commits:

70d0fca [Zhongshuai Pei] Update FilterPushdownSuite.scala
dc83ae9 [Zhongshuai Pei] Update FilterPushdownSuite.scala
485c61c [Zhongshuai Pei] Update Optimizer.scala
f03fe7f [Zhongshuai Pei] Merge pull request alteryx#12 from apache/master
f12fa50 [Zhongshuai Pei] Merge pull request alteryx#10 from apache/master
f61210c [Zhongshuai Pei] Merge pull request alteryx#9 from apache/master
34b1a9a [Zhongshuai Pei] Merge pull request alteryx#8 from apache/master
802261c [DoingDone9] Merge pull request alteryx#7 from apache/master
d00303b [DoingDone9] Merge pull request alteryx#6 from apache/master
98b134f [DoingDone9] Merge pull request alteryx#5 from apache/master
161cae3 [DoingDone9] Merge pull request alteryx#4 from apache/master
c87e8b6 [DoingDone9] Merge pull request #3 from apache/master
cb1852d [DoingDone9] Merge pull request #2 from apache/master
c3f046f [DoingDone9] Merge pull request #1 from apache/master
  • Loading branch information
pzzs authored and rxin committed Apr 30, 2015
1 parent 07a8620 commit 77cc25f
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,9 @@ object ColumnPruning extends Rule[LogicalPlan] {

Project(substitutedProjection, child)

case Project(projectList, Limit(exp, child)) =>
Limit(exp, Project(projectList, child))

// Eliminate no-op Projects
case Project(projectList, child) if child.output == projectList => child
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,23 @@ class FilterPushdownSuite extends PlanTest {

comparePlans(optimized, correctAnswer)
}

test("column pruning for Project(ne, Limit)") {
val originalQuery =
testRelation
.select('a,'b)
.limit(2)
.select('a)

val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer =
testRelation
.select('a)
.limit(2).analyze

comparePlans(optimized, correctAnswer)
}

// After this line is unimplemented.
test("simple push down") {
val originalQuery =
Expand Down

0 comments on commit 77cc25f

Please sign in to comment.