Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++][Parquet] Parquet arrow::FileWriter accepts arrow::RecordBatch in the buffered row group mode #15216

Closed
2 tasks
wgtmac opened this issue Jan 6, 2023 · 0 comments · Fixed by #15240
Closed
2 tasks

[C++][Parquet] Parquet arrow::FileWriter accepts arrow::RecordBatch in the buffered row group mode #15216

wgtmac opened this issue Jan 6, 2023 · 0 comments · Fixed by #15240
Assignees
@wgtmac
Labels
Component: C++ Component: Parquet Type: enhancement
Milestone
11.0.0

Comments

@wgtmac
Copy link
Member

wgtmac commented Jan 6, 2023

Describe the enhancement requested

Current state of the parquet arrow::FileWriter

  • It does not support buffered row group mode. User has to write the whole column chunk one by one.
  • It only accepts arrow::Table in the tabular form, or arrow::Array/arrow::ChunkedArray in the columnar form. If one only has arrow::RecordBatch on hand, additional transformation is required before calling the writer.

The goal of this issue aims to enhance the arrow::FileWriter as below:

  • Accept writing arrow::RecordBatch.
  • Support buffered row group mode.

Component(s)

C++, Parquet
wgtmac added a commit to wgtmac/arrow that referenced this issue Jan 7, 2023
@wgtmac
apacheGH-15216: [C++][Parquet] Parquet writer accepts RecordBatch
0d9f0e2 
 - Parquet arrow::FileWriter supports buffered row group mode.
 - Parquet arrow::FileWriter accepts arrow::RecordBatch.
@github-actions github-actions bot mentioned this issue Jan 7, 2023
Merged
@wjones127 wjones127 mentioned this issue Jan 11, 2023
Open
wgtmac added a commit to wgtmac/arrow that referenced this issue Jan 12, 2023
@wgtmac
apacheGH-15216: [C++][Parquet] Parquet writer accepts RecordBatch
6415ade 
 - Parquet arrow::FileWriter supports buffered row group mode.
 - Parquet arrow::FileWriter accepts arrow::RecordBatch.
wjones127 pushed a commit that referenced this issue Jan 12, 2023
@wgtmac
GH-15216: [C++][Parquet] Parquet writer accepts RecordBatch (#15240)
427a865 
 - Parquet arrow::FileWriter supports buffered row group mode.
 - Parquet arrow::FileWriter accepts arrow::RecordBatch.
* Closes: #15216

Authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Will Jones <willjones127@gmail.com>
@wjones127 wjones127 added this to the 11.0.0 milestone Jan 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wgtmac wgtmac

Labels
Component: C++ Component: Parquet Type: enhancement
Projects
None yet
Milestone
11.0.0
Development

Successfully merging a pull request may close this issue.

GH-15216: [C++][Parquet] Parquet writer accepts RecordBatch wgtmac/arrow
3 participants
@kou @wgtmac @wjones127