Async reader

[kylebarron]

Working async reader just for the arrow2 bindings.

TODO:

• Test with a parquet dataset
• Make read_row_group take a RowGroupMetaData instead of a FileMetaData?
• Ideally wouldn't need to copy the metadata object for each fetch, but not clear if it's possible in current wasm-bindgen for the async functions to take references as function parameters, since wasm-bindgen doesn't support lifetimes

---

Very early, just putting this up for planning.

Todo:

• working fetch example from rust (from here)
• [ ]

References:

• https://github.com/jorgecarleitao/arrow2/blob/v0.10.1/examples/s3/src/main.rs
• https://github.com/jorgecarleitao/parquet2/blob/7661cd4144/examples/s3/src/main.rs
• https://github.com/DataEngineeringLabs/ranged-reader-rs/blob/main/src/stream.rs

Future work:

• Update arrow2/parquet2 to not need an initial HEAD request for the length of the data. Should be able to use a negative byte range for the first fetch, and that should include all byte ranges.
• Configurable footer initial read size (right now hard coded in arrow2 to use 64kb)

[kylebarron]

I currently have issues with future cannot be sent between threads safely.

From rustwasm/wasm-bindgen#2409 (comment):

Futures generated with wasm-bindgen aren't Send so if you use async traits with wasm-bindgen you'll have to turn that off. Otherwise you'll need to arrange yourself for futures to be send by spawning tasks and such and using channels.

At least with arrow2 it's really designed around futures that implement Send, so it would be ideal to learn how to get this to work.

The async-executor example (https://github.com/wasm-rs/async-executor/blob/master/example/src/lib.rs) seems like a good place to start.

References:

• https://docs.rs/wasm-rs-async-executor/latest/wasm_rs_async_executor/
• https://docs.rs/tokio/0.1.22/tokio/sync/oneshot/index.html
• https://tokio.rs/tokio/tutorial/channels
• https://github.com/wasm-rs/async-executor/blob/master/example/src/lib.rs
• Allow async traits rustwasm/wasm-bindgen#2409 (comment)
• https://rustwasm.github.io/wasm-bindgen/api/wasm_bindgen_futures/fn.spawn_local.html
• Async rust book: https://rust-lang.github.io/async-book/04_pinning/01_chapter.html
• Executing futures in wasm rustwasm/wasm-bindgen#1126 (comment).
• https://stackoverflow.com/questions/66864010/future-created-by-async-block-is-not-send
• https://fasterthanli.me/articles/pin-and-suffering

[kylebarron]

Tons of progress:

• Structure of AsyncParquetFile class. Async constructor to fetch content length and metadata.
• Working async read_row_group! (for a single row group)

Future work:

• Right now read_row_group uses self instead of &self because the latter gives a lifetimes error. "Value does not live long enough". https://stackoverflow.com/questions/42503296/value-does-not-live-long-enough
• Figure out how to send only one request for the whole row group? I think right now it sends one request per column.
• Check that multiple requests are indeed being made concurrently, not sequentially

[kylebarron]

• Right now read_row_group uses self instead of &self because the latter gives a lifetimes error. "Value does not live long enough". stackoverflow.com/questions/42503296/value-does-not-live-long-enough

It looks like this isn't generally possible without consuming self. See: rustwasm/wasm-bindgen#1858 (comment), rustwasm/wasm-bindgen#1733 (comment), rustwasm/wasm-bindgen#2195 (comment).

Given this, a class-based solution will only work if the entire class can be cloned. At that point, I think it'll be simpler to have a functional API instead:

pub async fn read_row_group(
    url: String,
    content_length: usize,
    metadata: FileMetaData,
    i: usize,
) -> Result<Uint8Array, JsValue>

The inputs can be copied on the JS side. So to create an async iterator, you'd do

const metadata = await read_parquet_metadata(url);
const batches = [];
for (let i of metadata.numRowGroups) {
  const batch = await read_row_group(metadata, i);
  batches.push(batch);
}