Add code and tests + supporting files

[mariosasko]

Add the filesystem implementation with additional tweaks (cp_file, rm_file, etc.) needed for the datasets integration.

Also, add/update supporting files to align the repo with the existing HF projects.

[lhoestq]

Woohoo love it !

[mariosasko]

I've addressed the comments and made additional improvements to make the code cleaner/more performant. Let me know what you think!

I'm still trying to figure out if I can avoid the temp file creation in write mode in HfFile because this goes against fsspec's "philosophy". For instance, simplecache already uses a temp file for caching remote writes, so in our case, this would mean creating 2 temp files and occupying 2x space.

Having thought about it, I agree it's OK to depend on huggingface_hub for simplicity. With this in mind, some things that would be cool to have in huggingface_hub (cc @Wauplin) to simplify the code further:

• the endpoint param in hf_hub_url to support custom endpoints
• an op for efficient copying of repo files in the commit API (advanced use-case)

[Wauplin]

@mariosasko I did not have the time to review the PR yet but about your two last points:

the endpoint param in hf_hub_url to support custom endpoints

Yes, would definitely be good to have that. Could you create an issue for that please ?

an op for efficient copying of repo files in the commit API (advanced use-case)

Hmm, I see the purpose in hffs since we need to comply to the specification. However as you said it seems a very niche use-case. You can still create an issue for it but since it would require some changes in moon-landing and that it's not really a priority, it is most likely that we won't implement it anytime soon. I would say the sub-optimal solution where the file is copied locally twice is not too bad, hoping that no-one uses it for large files.

[Wauplin]

This looks good, thanks for working on it @mariosasko !

I have added comments, most of them been about stuff that can be leverage from huggingface_hub. If would require huggingface_hub>=0.10.0 instead of huggingface_hub>=0.9.0 if that's fine.

[lhoestq]

Hi ! Any news on this ? @severo and I would like to use it to store datasets in Parquet format on the Hub under git references (for the datasets-server project and to showcase how to get a arbitrary slices of data)

Happy to help if I can :)

[mariosasko]

@Wauplin @lhoestq Let me know if everything looks good now

[lhoestq]

Thanks ! looks all good :)

[Wauplin]

Looks good ! Thanks for making the changes :)