CONVERT TO DELTA as a pure delta-rs API

[ericsun2]

Description

The equivalent API in delta-rs to CONVERT TO DELTA parquet.'s3://my-bucket/parquet-data'; directly in Python.

It is https://docs.delta.io/latest/api/python/index.html#delta.tables.DeltaTable.convertToDelta for delta-rs without Spark.

Use Case
Let's say that we have a bunch of Parquet directories which can be quickly/effectively converted to Delta.

2nd request: generate Uniform 3.0 manifest via delta-rs. That will be great.

Related Issue(s)
#1041

[rtyler]

@ericsun2 I wrote and use oxbow for these purposes. I'm not sure what an API like that would look like in Python. What you reference is a Spark SQL command, which we don't exactly have an equivalent interface in the Python connector. How would you envision that?

[MrPowers]

Here are the PySpark APIs for that functionality:

# Convert unpartitioned parquet table at path 'path/to/table'
deltaTable = DeltaTable.convertToDelta(spark, "parquet.`path/to/table`")

# Convert partitioned parquet table at path 'path/to/table' and partitioned by integer column named 'part'
partitionedDeltaTable = DeltaTable.convertToDelta(spark, "parquet.`path/to/table`", "part int")

Perhaps we could expose something similar in delta-rs?

[ericsun2]

Both SparkSQL and PySpark have similar statement or API already, but it makes sense to have the same API in delta-rs so that we can directly invoke such "convert to delta" function in a light-weight fashion inside a non-Spark context (such as microservice, Lambda, Python script) w/o having a running SQL Warehouse or spinning up a PySpark process.

Especially when we have some services which can generate a lot of Parquet files instead of CSV/JSON files, it will be quite useful to generate the Delta + Iceberg + Hudi manifest metadata directly and swiftly before we convert and compact such Parquet directories into Delta Table. I envision this "convert to delta" function can support the additional option to generate Iceberg and Hudi metadata optionally if user choose to opt-in as well. This will be perfectly aligned with Uniform 3.0 standard.