read_parquet(use_pyarrow=True) does not work for parquet datasets (directory)

[karenkewitsch]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

Seems to be working up till 0.15.14, but now broken.

Reproducible example

from pathlib import Path

import pyarrow
import pyarrow.parquet as pq
import polars

parquet_path = Path(__file__).parent / "my_dataset"
test_table = pyarrow.table([["foo", "bar"], [1, 2]], names=["name", "age"])
pq.write_to_dataset(test_table, str(parquet_path), partition_cols=["age"])

polars_df = polars.read_parquet(str(parquet_path), use_pyarrow=True)

Expected behavior

I would expect it to behave like previous versions and load all the partitions in the dir.

Installed versions

---Version info---
Polars: 0.15.17
Index type: UInt32
Platform: macOS-13.1-arm64-arm-64bit
Python: 3.10.9 (main, Jan 11 2023, 08:18:22) [Clang 14.0.0 (clang-1400.0.29.202)]
---Optional dependencies---
pyarrow: 10.0.1
pandas: 1.5.2
numpy: 1.23.5
fsspec: 2022.11.0
connectorx: <not installed>
xlsx2csv: <not installed>
deltalake: <not installed>
matplotlib: 3.6.3```

</details>

[ritchie46]

Read parquet never intended to do that. You can use globbing patterns to read multiple files. E.g.: pl.read_parquet("mydir/*.parquet").

[karenkewitsch]

@ritchie46 , that seems to contradict the past behaviour and the documentation of the function, which states "... If the path is a directory, that directory will be used as partition aware scan." regarding the 'source' argument.

[ritchie46]

@ritchie46 , that seems to contradict the past behaviour and the documentation of the function, which states "... If the path is a directory, that directory will be used as partition aware scan." regarding the 'source' argument.

Hmm.. that shouldn't be there I believe. 🤔

[ritchie46]

It does work if pyarrow is false?