[GCS] Scanning hive-partitioned dataset results in orders of magnitude more network traffic than it should

[jhostetler]

Describe the bug, including details regarding any error messages, version, and platform.

I have a hive-partitioned dataset in a Google Cloud Storage bucket. Its size is around 54MB according to gsutil du and verified by downloading it and checking locally. However, if I open it with ds = pyarrow.dataset.dataset("gs://...", partitioning="hive", format="parquet") and then traverse it with ds.to_batches(), it results in multiple GB of inbound network traffic and takes much longer than simply downloading the data.

This is with PyArrow 10.0.1. MacOS 12.6 with Apple M1 CPU.

Component(s)

Python

[djouallah]

I am having the same issue when used with delta table
delta-io/delta-rs#931