SPARK-1321 Use Guava's top k implementation rather than our BoundedPriorityQueue based implementation #229
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rxin
changed the title
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
All automated tests passed. |
| override def compare(l: T, r: T) = ord.compare(l, r) | ||
| } | ||
| collectionAsScalaIterable( | ||
| ordering.leastOf(asJavaIterator(input), num)).iterator |
There was a problem hiding this comment.
is this proper indendation? Should this be indented from the previous line?
|
Looks good to me. One small style comment. |
|
weird i missed that. fixed. |
| val ordering = new GuavaOrdering[T] { | ||
| override def compare(l: T, r: T) = ord.compare(l, r) | ||
| } | ||
| collectionAsScalaIterable(ordering.leastOf(asJavaIterator(input), num)).iterator |
There was a problem hiding this comment.
Why can't this just be
ordering.leastOf(input, num).iterator
?
There was a problem hiding this comment.
it could - but trying to be more explicit here. i'm really not a fan of implicits, especially in critical paths ..
… based implementation. Also updated the documentation for top and takeOrdered. Guava's top k implementation (in Ordering) is much faster than the BoundedPriorityQueue implementation for roughly sorted input (10 - 20X faster), and still faster for purely random input (2 - 5X).
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
All automated tests passed. |
|
Ok I've merged this. |
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
…iorityQueue based implementation Also updated the documentation for top and takeOrdered. On my simple test of sorting 100 million (Int, Int) tuples using Spark, Guava's top k implementation (in Ordering) is much faster than the BoundedPriorityQueue implementation for roughly sorted input (10 - 20X faster), and still faster for purely random input (2 - 5X). Author: Reynold Xin <rxin@apache.org> Closes apache#229 from rxin/takeOrdered and squashes the following commits: 0d11844 [Reynold Xin] Use Guava's top k implementation rather than our BoundedPriorityQueue based implementation. Also updated the documentation for top and takeOrdered.
jamesrgrinter
pushed a commit
to jamesrgrinter/spark
that referenced
this pull request
Apr 22, 2018
Igosuki
pushed a commit
to Adikteev/spark
that referenced
this pull request
Jul 31, 2018
* In hdfs soak test, updated hdfsclient image and specified hdfs service name. * Made the HDFS service name configurable. * Use the "app name" (begins with a slash) instead of service name. * Simplified the "hdfs-kerberos-delete-terasort-files" job * Updated service name and image version in 'hdfs-kerberos-delete-terasort-files' job * Make the executors run as root (for centos) * Templated the principal in the 'hdfs-kerberos-delete-terasort-files' job
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Fix the Mitaka devstack installation
arjunshroff
pushed a commit
to arjunshroff/spark
that referenced
this pull request
Nov 24, 2020
fishcus
pushed a commit
to fishcus/spark
that referenced
this pull request
Nov 18, 2021
AL-1864 upgrade jackson-databind to 2.9.10.8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Also updated the documentation for top and takeOrdered.
On my simple test of sorting 100 million (Int, Int) tuples using Spark, Guava's top k implementation (in Ordering) is much faster than the BoundedPriorityQueue implementation for roughly sorted input (10 - 20X faster), and still faster for purely random input (2 - 5X).