Parquet ObjectStore Writer

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

#3370 added first-party support for reading parquet from object_store.

Describe the solution you'd like

I would like similar first-party integration for writing

Describe alternatives you've considered

Additional context

#5500 reworks the ObjectStore traits to facilitate this

[abhiaagarwal]

Anyone working on this? I'd like to give it a shot. I gained a lot of familiarity with the Upload trait after reworking delta-rs for the new object_store.

[Xuanwo]

Anyone working on this? I'd like to give it a shot. I gained a lot of familiarity with the Upload trait after reworking delta-rs for the new object_store.

Should be related to #5766, I'm working on it now.

[abhiaagarwal]

@Xuanwo ah, sounds great! I finished implementing it on my end, but it's a little bit lazy, It's a little bit inelegant, I'll defer to you :)

impl AsyncFileWriter for ParquetObjectWriter {
    fn write(&mut self, bs: Bytes) -> BoxFuture<'_, Result<()>> {
        self.upload.put_part(bs.into()).map_err(|e| {
            ParquetError::General(format!("ParquetObjectWriter::write error: {e}"))
        }).boxed()
    }

    fn complete(&mut self) -> BoxFuture<'_, Result<()>> {
        self.upload.complete().map_err(|e| {
            ParquetError::General(format!("ParquetObjectWriter::complete error: {e}"))
        }).map(|_| Ok(())).boxed()
    }
}