Ability to pass empty Arrow tables/datasets to write_deltalake with rust engine

[jorritsandbrink]

Description

It's currently not possible to pass empty Arrow tables or datasets to write_deltalake when using the rust engine.

Write empty Arrow table:

import pyarrow as pa
from deltalake import write_deltalake

arrow_table = pa.Table.from_pydict(
    {"foo": [1, 2], "bar": [True, False]}
)
empty_arrow_table = arrow_table.schema.empty_table()

write_deltalake("my_delta_table", arrow_table, mode="append", engine="rust")  # this creates the Delta table
write_deltalake("my_delta_table", empty_arrow_table, mode="append", engine="rust")  # this errors (but works with `pyarrow` engine)

Error:

DeltaError: Generic error: No data source supplied to write command.

Write empty dataset:

import pyarrow.parquet as pq

pq.write_table(empty_arrow_table, "my_empty_parquet_file.parquet")
empty_arrow_dataset = pa.dataset.dataset("my_empty_parquet_file.parquet")
write_deltalake("my_delta_table", empty_arrow_dataset, mode="append", engine="rust")  # this errors on both `pyarrow` and `rust` engine

Error (when using rust engine, pyarrow throws different error):

PanicException: called `Result::unwrap()` on an `Err` value: CDataInterface("Index error: list index out of range. Detail: Python exception: IndexError")

Use Case
I now use this flow to handle an empty table:

if arrow_table.num_rows > 0:
    write_deltalake(...)

And this flow to handle an empty dataset:

arrow_table = arrow_dataset.to_table()

if arrow_table.num_rows > 0:
    write_deltalake(...)

The dataset case is particulary unpleasant, because you need to eagerly materialize the dataset to a table in memory just to check if it's empty.

It would be nice if we could simply use

write_deltalake(..., data=potentially_empty_arrow_table_or_dataset, ...)

and leave handling of the "empty case" to delta-rs.

Related Issue(s)

• Create empty table failed on rust engine #2354 (only regards initial table creation, not subsequent writes)

[sherlockbeard]

can you share full code example for pyarrow dataset ?.

[jorritsandbrink]

@sherlockbeard yes, here it is:

import pyarrow as pa
import pyarrow.parquet as pq
from deltalake import write_deltalake

arrow_table = pa.Table.from_pydict(
    {"foo": [1, 2], "bar": [True, False]}
)
empty_arrow_table = arrow_table.schema.empty_table()

pq.write_table(empty_arrow_table, "my_empty_parquet_file.parquet")
empty_arrow_dataset = pa.dataset.dataset("my_empty_parquet_file.parquet")

write_deltalake("my_delta_table", arrow_table, mode="append", engine="rust")  # this creates the Delta table
write_deltalake("my_delta_table", empty_arrow_dataset, mode="append", engine="rust")  # this errors on both `pyarrow` and `rust` engine