[Rust] Writing optional fields

[ccakes]

I'm trying to write a dataset with optional fields but I can't figure out a way to do it using the existing API. While the only write methods are per-row group, I can't see a way to specify nulls via ColumnWriterImpl.write_batch()

Am I missing something obvious or is this something that isn't well supported until the Row writer implementation? I tried looking at the other lang libraries for inspiration (specifically C++) but it only gives an example using write_batch as effectively an individual row writer, other languages all give examples for optionals using dedicated row-level writer functionality.

[paddyhoran]

cc @sunchao

[sunchao]

@ccakes at the moment we don't have a high-level API for this yet, so you have to provide values, repetition levels, and definition levels. Please check here for how to use the API, and this tutorial for how to specify nulls via definition levels.

I do intend to implement a high-level API in the near future which will make this much easier. There's also a PR that aims at providing a serde_derive like procedure macro to serialize Rust structs into Parquet format.

[wesm]

Can you open a JIRA for this, or otherwise close it?

[sunchao]

I filed ARROW-4392 to track the high-level Parquet writer.

[wesm]

Sweet thanks