Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make an ADT for CoGrouped #1698

Merged
merged 1 commit into from
Jun 7, 2017
Merged

Make an ADT for CoGrouped #1698

merged 1 commit into from
Jun 7, 2017

Conversation

johnynek
Copy link
Collaborator

@johnynek johnynek commented Jun 4, 2017

CoGrouped was the last scalding typed trait to be a black box that we cannot look inside at planning.

This change continues to expose the old methods so you can use the direct composition of CoGrouped, or you can take them apart and re-assemble in a backend as may be more appropriate with global information or for particular compute substrates.

After this is merged, we can pretty easily add join support to #1697 without having to directly support N-way joins on the memory backend.

This will also be useful for spark, which does not support N-way joins in the RDD API.

@johnynek johnynek requested a review from piyushnarang June 4, 2017 20:28
@johnynek
Copy link
Collaborator Author

johnynek commented Jun 4, 2017

cc @fwbrasil

Copy link
Contributor

@benpence benpence left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable.

"This change continues to expose the old methods so you can use the direct composition of CoGrouped, or you can take them apart and re-assemble in a backend as may be more appropriate with global information or for particular compute substrates."

Am I understanding correctly that you expect users to use the case classes directly? Or do you mean for porting Scalding to different platforms?

@johnynek
Copy link
Collaborator Author

johnynek commented Jun 7, 2017

A normal user of scalding would never use this for anything that comes to mind now.

Someone writing some specific optimizer of scalding TypedPipes would use it, or someone writing a new scalding backend would use it.

Pretty much everyone else should never notice anything.

@johnynek johnynek merged commit d4f3655 into develop Jun 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants