Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve TypedParquetTuple #1302

Closed
johnynek opened this issue May 22, 2015 · 2 comments
Closed

Improve TypedParquetTuple #1302

johnynek opened this issue May 22, 2015 · 2 comments

Comments

@johnynek
Copy link
Collaborator

  1. add macros to generate ParquetReadSupport and ParquetWriteSupport
  2. We may need to change TypedParquetTupleScheme to accept instances rather than a class if reflecting a macro generated instance fails for some reason. If so, serialize and compress those instances (see Bijection's GzippedBase64String injection and JavaSerialization injection).
  3. Make it a one-liner to read and write Parquet using case classes.
@johnynek
Copy link
Collaborator Author

/cc @JiJiTang @avibryant @colinmarc

@JiJiTang
Copy link
Contributor

Hi @johnynek , yes we can improve this. And I've just made a commit about this improvement in my forked repo and will create PR later.

JiJiTang added a commit to JiJiTang/scalding that referenced this issue May 24, 2015
@JiJiTang
Improve TypedParquetTuple twitter#1302
08c7382
JiJiTang added a commit to JiJiTang/scalding that referenced this issue May 25, 2015
@JiJiTang
Improve TypedParquetTuple twitter#1302
ff16450
JiJiTang added a commit to JiJiTang/scalding that referenced this issue May 25, 2015
@JiJiTang
Improve TypedParquetTuple twitter#1302
60e7390
JiJiTang added a commit to JiJiTang/scalding that referenced this issue May 28, 2015
@JiJiTang
Improve TypedParquetTuple twitter#1302
c6980f3
@johnynek johnynek mentioned this issue May 29, 2015
Merged
JiJiTang added a commit to JiJiTang/scalding that referenced this issue May 29, 2015
@JiJiTang
Improve TypedParquetTuple twitter#1302
6995da4 
 *Refacto
 *Add example in README
johnynek added a commit that referenced this issue May 30, 2015
@johnynek
Merge pull request #1303 from JiJiTang/develop
7af2fae 
Improve TypedParquetTuple #1302
@ianoc ianoc mentioned this issue Aug 10, 2015
Closed
@johnynek johnynek closed this as completed Feb 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@johnynek @JiJiTang