Error index out of bounds when scanning multiple CSV files from S3 with .scan_csv

[mghen]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
import os

path = r"s3://your_bucket_containing_multiple_csv_files/*.csv"  # where this bucket contains multiple csv files of the same structure
sql = f"SELECT * FROM {path}"

storage_options = {
    "aws_access_key_id": os.environ['AWS_ACCESS_KEY_ID'],
    "aws_secret_access_key": os.environ['AWS_SECRET_ACCESS_KEY'],
    "aws_session_token": os.environ['AWS_SESSION_TOKEN'],
    "aws_region": "ap-southeast-2"
}

polars_scan = pl.scan_csv(path, storage_options=storage_options)
print(polars_scan)  # this and any actions attempting to use the lazy frame cause an error when there is more than one CSV in the bucket.

Log output

Traceback (most recent call last):
  File "C:\Users\mghenri\AppData\Roaming\JetBrains\PyCharm2024.1\scratches\scratch2.py", line 25, in <module>
    print(polars_scan)
  File "C:\Program Files\Python311\venvs\orchestration-H9WLSma6-py3.11\Lib\site-packages\polars\lazyframe\frame.py", line 626, in __str__
    {self.explain(optimized=False)}\
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python311\venvs\orchestration-H9WLSma6-py3.11\Lib\site-packages\polars\lazyframe\frame.py", line 1109, in explain
    return self._ldf.describe_plan()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: index out of bounds: the len is 1 but the index is 1

Issue description

Fail scenario: Upload a few csv files to your_bucket_containing_multiple_csv_files and then run the code.

Success scenario: Retry with just one csv in the bucket.

Success scenario: Retry with the same (multiple) csv files in a local folder instead of S3 (and remove storage_options from code).

Expected behavior

A lazy frame is successfully created and usable.

Installed versions

--------Version info---------
Polars: 1.4.1
Index type: UInt32
Platform: Windows-10-10.0.19045-SP0
Python: 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:
cloudpickle:
connectorx:
deltalake:
fastexcel:
fsspec: 2024.3.1
gevent:
great_tables:
hvplot:
matplotlib:
nest_asyncio: 1.6.0
numpy: 1.26.4
openpyxl:
pandas: 2.2.1
pyarrow: 15.0.2
pydantic:
pyiceberg:
sqlalchemy:
torch:
xlsx2csv:
xlsxwriter:

[mghen]

Also worth noting that if these are parquet files instead of csv files, then pl.scan_parquet works fine.

[ritchie46]

@nameexhaustion can you take this one?

[mghen]

Wow that was fast! Thanks heaps!