Feat: add support for HuggingFace datasets

[deependujha]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

• Index HF datasets and then stream them.

import litdata as ld
from litdata.streaming.item_loader import ParquetLoader

hf_uri = "hf://datasets/open-thoughts/OpenThoughts-114k/data"

ld.index_parquet_dataset(hf_uri, "hf-cache")

# -------

ds = ld.StreamingDataset(hf_uri, index_path="hf-cache", item_loader = ParquetLoader())

for _ds in ds:
    print(f"{_ds=}")

---

Or, just Steam HF dataset directly 🚀

• It'll automatically index hf dataset and cache it.

import litdata as ld

hf_uri = "hf://datasets/open-thoughts/OpenThoughts-114k/data"

ds = ld.StreamingDataset(hf_uri)

for _ds in ds:
    print(f"{_ds=}")

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Attention: Patch coverage is 89.56522% with 24 lines in your changes missing coverage. Please review.

Project coverage is 79%. Comparing base (f09eb65) to head (e1a4706).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff          @@
##           main   #462    +/-   ##
====================================
+ Coverage    78%    79%    +1%     
====================================
  Files        37     38     +1     
  Lines      5370   5547   +177     
====================================
+ Hits       4193   4386   +193     
+ Misses     1177   1161    -16     

[justusschock]

could we get some sort of tests for this?

[tchaton]

It would be good to make a few more tests. This looks a bit light for the quantity of code ;)

[lantiga]

Looks great @deependujha! just a few comments