Lightweight data loader

[chrisflesher]

I was reading the notes about data loading and was surprised that both options (tensorflow-datasets and pytorch) seem to require 1-3GB installs! Is there a minimal data loader that doesn't require so much disk space?

[hawkinsp]

I don't think tensorflow_datasets depends on tensorflow any more, so the wheel download size is more like 10MB.

There are other libraries (e.g., https://github.com/google/grain) that you may want to look into, although the public docs on grain seem minimal.

[chrisflesher]

I tried running the tensorflow-datasets example but without tensorflow it crashed when loading the MINST dataset. Maybe for custom datasets this might work? I don't have that much experience using it.

May be interested in grain but couldn't find any documentation...

[adam-hartshorne]

Have you tried jax_dataloader ?
https://github.com/BirkhoffG/jax-dataloader

[chrisflesher]

Thanks, will look at this.

[moskomule]

huggingface's datasets now supports jax!
https://huggingface.co/docs/datasets/use_with_jax

[chrisflesher]

Wow, this seems like a really great option for what I'm doing. Lightweight, fast, and widely used.