Allow users to define the data type and nullability of a partition column when converting a Parquet table to Delta table

[junjunjd]

Description

In the PySpark APIs for convertToDelta command, users can specify the data type of a partition column:

# Convert partitioned parquet table at path 'path/to/table' and partitioned by integer column named 'part'
partitionedDeltaTable = DeltaTable.convertToDelta(spark, "parquet.`path/to/table`", "part int")

It makes sense to support user-defined data type and nullability for partition columns in convert_to_delta. Currently the convert_to_delta function sets the data type of all partition columns as string.

Use Case

Related Issue(s)
#1041, #1682, #1686

[junjunjd]

The convert_to_delta API now takes a vector of SchemaField for partition columns defined by user.

pub async fn convert_to_delta(
    storage: ObjectStoreRef,
    partition_schema: Vec<SchemaField>,
) -> Result<DeltaTable, Error> 

The changes are pushed to #1686.