Hugging Face Loader: Add lazy load

[eyurtsev]

Add lazy load to HF datasets loader

Unfortunately, there are no tests as far as i can tell. Verified code manually.

[eyurtsev]

Same feature as this: https://github.com/hwchase17/langchain/pull/3501/files#diff-257a9ed31010c03aefa6e28d08934579d4b596d5e61090ad6d6d6583c5f8f317

[vowelparrot]

This would be even better if we use streaming datasets https://huggingface.co/docs/datasets/stream

[eyurtsev]

if i understood correctly streaming=True doesn't download the data locally, but streaming=False does?

it sounds like we should pass a streaming param to dataset, rather than use streaming=True by default. Is that correct?

[vowelparrot]

Ya I don't think we should change the behavior right now

[eyurtsev]

@vowelparrot I'm going to merge as is. My understanding is that adding a default streaming=True may change behavior in an undesired way (data isn't downloaded locally -- which I think is often desirable). Instead we can follow up with a PR to expose more parameters in the init that will be passed to the load datasets function.