to_pyarrow_dataset() ignores dictionary_columns in read_options

[Kuhlwein]

Environment

Delta-rs version: 0.6.3

Binding: python

Environment:

• Cloud provider: Azure
• OS: Windows/linux
• Other:

---

Bug

When passing parquet_read_options to to_pyarrow_dataset, it is possible to use dictionary_columns to control which columns should be dictionary encoded as they are read.

What happened:
Such columns are not dictionary encoded when they are read, rather they are of type string, as they would be had dictionary_columns been empty.

What you expected to happen:
I expect the particular columns to be dictionary encoded.

How to reproduce it:

import pyarrow.dataset as ds

dt = DeltaTable(path)
read_options = ds.ParquetReadOptions(dictionary_columns=["test"])
data = dt.to_pyarrow_dataset(parquet_read_options=read_options)
table = data.to_table()
print(table.schema)

I expect the column "test" to be dictionary encoded, rather than just of type string.

More details:
I believe that the problem is in table.py at line 338, the schema of self.schema.to_pyarrow() has the wrong type for the columns which are to be dictionary encoded.

Maybe it is possible to use the physical_schema property of the fragments defined just above to get the right schema, or otherwise parse the read options to modify the schema?

[wjones127]

Delta Tables have a specific schema, and we enforce that when reading we always use that exact schema. Dictionary types are different types in Arrow, not just a minor detail of the array, so reading as dictionary array would mean a different schema.

But perhaps we could parse the read_dictionary and have special handling for that. It does seem desirable to be able to read columns as dictionaries.