-
Notifications
You must be signed in to change notification settings - Fork 707
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Typedpipe partition #987
Typedpipe partition #987
Conversation
Merge from twitter/scalding
Looks like RichPipe has a |
well, I think we should keep TypedPipe as close as possible to scala.Iterable or Seq, so I think it is fine. It was probably a mistake in RichPipe, but oh well. Not all things are named the same between the two. LGTM. Merge when green. |
Makes sense. Travis seems to be having some trouble pulling down deps :(
I'll retry after a while I guess. |
change the http deps to https. We saw this earlier today. On Thu, Jul 31, 2014 at 1:43 PM, Ruban Monu notifications@github.com
Oscar Boykin :: @posco :: http://twitter.com/posco |
* | ||
* Sometimes what you really want is a groupBy in these cases. | ||
*/ | ||
def partition(p: T => Boolean): (TypedPipe[T], TypedPipe[T]) = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We may need some rules of thumb of when to use this V.S when to use Group operation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be partition when you want to split your 'dataflow' into two, groupBy when you want to create a grouping in the same dataflow?
Anything else that comes to mind? I'm still partially hung over in the fields API world so might be missing something.
(TypedPipe.groupBy does not exactly match scala Iterable.groupBy, gives you a Grouped
and not a Map[K, TypedPipe[T]
. If that were the case, then partition would be the same as groupBy on a boolean which gives you (TypedPipe[T], TypedPipe[T])
instead of a Map[Boolean, TypedPipe[T]]
.)
merge when green. |
closes #986 |
Addresses #986