feat: add convert_to_delta

[junjunjd]

Description

Add a convert_to_delta operation for converting a Parquet table to a Delta Table in place.

Related Issue(s)

• closes Add convertToDelta command #1041
• closes CONVERT TO DELTA as a pure delta-rs API #1682

Documentation

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[rtyler]

Thanks for the pull request! This is a great start and I've left some comments. I can help bring this further if you need it, but also feel free to ping me on Slack if you need any further help preparing this for merge!

[MrPowers]

@junjunjd - did you see the comments from @rtyler? Anything else we can do to help you with this one?

[junjunjd]

@junjunjd - did you see the comments from @rtyler? Anything else we can do to help you with this one?

I've replied to comments from @rtyler. This MR is ready for review.

I also created four follow-up issues. I've started working on the first issue #1718.

• Allow users to define the data type and nullability of a partition column when converting a Parquet table to Delta table #1718
• Collect stats on parquet files when converting a Parquet table to a Delta table #1719
• Support parquet files with nanoseconds timestamps when converting a Parquet table to a Delta table #1721
• Support convert_to_delta for parquet2 #1723

[junjunjd]

Follow-up issue #1718 is done. The API now takes a vector of user-defined SchemaField for partition columns.

pub async fn convert_to_delta(
    storage: ObjectStoreRef,
    partition_schema: Vec<SchemaField>,
) -> Result<DeltaTable, Error> 

The changes are pushed to this MR.

[junjunjd]

@rtyler @wjones127 @roeap This MR is ready for final review.

[junjunjd]

@rtyler @wjones127 @roeap Is there anything I can do to help get this merged?

[junjunjd]

@rtyler @wjones127 @roeap Just a gentle ping for review.

[roeap]

Thanks for this great wortk @junjunjd! Left some comments in the code.

More generally speaking, I believe we should model CONVET TO DELTA as an operation (i.e. config builder / IntoFuture Trait ...) and also place it in the operations module rather then next to the table. We are slowly but surely working towards also modelling things as logical plans and exposing sql APIs so convert to delta should be compatible whith the other operations we expose.

[junjunjd]

@roeap Thanks for the comments. I updated the MR to model CONVET TO DELTA as an operation. This MR is ready for another review.

[roeap]

Great work @junjunjd and thanks for sticking with it.

Just some minor nits to look at, but overall I think we are good to go!

[roeap]

It seems we do have some compiler errors going on. I would disregard the ones related to a future not being Send at first until the other issues are fixed, they may just resolve themselves :).

[junjunjd]

@roeap Thanks for the review! CI is fixed. MR is ready for final review/merge.

[roeap]

Looking great @junjunjd, thanks for sticking with it!