-
Notifications
You must be signed in to change notification settings - Fork 707
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add OrderedSerialization2 support in Matrix2. #1457
Conversation
@johnynek Is this good to go? |
Sorry. Didn't see this. Thanks for bringing to my attention. I'll review On Saturday, December 19, 2015, Kevin Lin notifications@github.com wrote:
P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin |
Thanks, happy holidays! |
@@ -18,6 +18,7 @@ package com.twitter.scalding.mathematics | |||
import cascading.flow.FlowDef | |||
import cascading.pipe.Pipe | |||
import cascading.tuple.Fields | |||
import com.twitter.scalding.serialization.{ OrderedSerialization, OrderedSerialization2 } | |||
import com.twitter.scalding.TDsl._ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you remove TDsl while you are at it?
Need a test for this that some Matrix2 job works with the option: https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/Config.scala#L118 In fact, we should write the test first and find an example that fails (I guess adding two matrices) |
Moved the method implementation into OrderedSerialization2 companion object. |
I had troubles in importing serialization test's intOrderedSerialization ( Line 46 in 3985459
BTW, feel like we can use slack channel for discussions like this more frequently. |
def dynamicSize(i: Int) = staticSize | ||
} | ||
|
||
val p1: Pipe = Tsv("mat1", ('x1, 'y1, 'v1)).read |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this style is not recommended any more (.toTypedPipe), so let's just use TypedText.tsv the whole way:
val tp1 = TypedPipe.from(TypedText.tsv[(Int, Int, Double)]("mat1"))
Add OrderedSerialization2 support in Matrix2.
To resolve issue #1418