Hugging Face support

[c-peters]

Description

Hugging face has a platform datasets which hosts ML datasets. Right now we have limited support (in the form of pyarrow or ffspec) which does not allow for lazy evaluation.

Goal:

• Allow reading from hf://..... url's in a lazy manner
• Allow for globbing

pl.scan_parquet("hf://datasets/roneneldan/TinyStories/data/train-*-of-*.parquet")

Pointers:

• We should be able to convert the hf:// url to their https:// counterparts. Either by implementing our own custom ObjectStore or by converting the hf:// url into a list of https:// somewhere Hugging Face allows listing files in directories using the following endpoint https://huggingface.co/api/datasets/{reponame}/tree/{revision}/{encoded_path} this spits out paginated json which we could traverse to manually glob all the matching files
• The file can be found under the endpoint `https://huggingface.co/{reponame}/resolve/{revision}/{encoded_path}

Examples URLs:
https://huggingface.co/roneneldan/TinyStories-1M/resolve/main/.gitattributes
https://huggingface.co/api/datasets/roneneldan/TinyStories/tree/main

In the future, add support for:

• Private or gated repos with authentication (storage_options?)
• Auto converted parquet files (extra @parquet.. in the url)
• Other file types scan_csv and scan_(nd)json
• Add write support?
• Add a chapter in the user-guide (Polars and upstream)