Tips to upload large models/datasets #1565

Wauplin · 2023-07-20T16:05:31Z

We've talked several times about having a section in the docs with some tips for users wanting to upload a large amount of data. I tried to sum-up everything with two distinct aspects:

technical limitations (max size per file, max file per commit, max file per folder, max file per repo)
practical tips (start small, use hf_transfer, expect failures)

I also reorganized a bit the Upload guide in general. It's becoming a quite long page but I still think it makes sense to have everything in one place.

HuggingFaceDocBuilderDev · 2023-07-20T16:10:58Z

The documentation is not available anymore as the PR was closed or merged.

Wauplin · 2023-07-20T16:15:44Z

cc @Pierrci @coyotte508 can you check the info is accurate. I took it from #995 (comment) but prefer to double-check it's not outdated (already 1y old 😄)

cc @lhoestq might be interested for datasets users as well

lhoestq

Awesome !

docs/source/guides/upload.md

stevhliu

Awesome! Once merged, it may be nice to link to this from the Datasets docs as well 😄

docs/source/guides/upload.md

Wauplin · 2023-07-20T17:02:14Z

Thanks for the review and valuable feedback @stevhliu @lhoestq! I'll update to PR to make the short list more explicit about what is a hard limit and what is a recommendation. And also move it up in the section :)

osanseviero

Cool stuff! 🔥

osanseviero · 2023-07-20T21:19:49Z

docs/source/guides/upload.md

@@ -371,11 +376,89 @@ In addition to [`upload_file`] and [`upload_folder`], the following functions al

 For more detailed information, take a look at the [`HfApi`] reference.

-## Push files with Git LFS
+## Tips and tricks for large uploads


Let's be careful to not break this url

I would not be too worried about it to be honest. I've checked and #upload-files-with-git-lfs is referenced nowhere in our internal docs (both hfh and hf_docs). Doesn't mean that such a URL doesn't exist in the wild but I would expect probability to be quite low. And even if it's the case, users will be redirected to the correct page even though it's not the correct section. Since Git LFS upload is mostly deprecated it should be fine

(btw it's quite easy to add backward compat for a particular url anchor if needed)

docs/source/guides/upload.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

Wauplin · 2023-07-21T12:59:11Z

I have addressed all the comments:

merged the PR suggestions
suggest to contact us via Discord or datasets@huggingface.co (I expect repos with TBs of data to be 99% datasets)
compiled technical recommendation in a table as suggested by @stevhliu:

@stevhliu Could you have a second look to the docs and check it looks good? Thanks in advance!

docs/source/guides/upload.md

julien-c · 2023-07-21T14:35:33Z

docs/source/guides/upload.md

@@ -371,11 +377,95 @@ In addition to [`upload_file`] and [`upload_folder`], the following functions al

 For more detailed information, take a look at the [`HfApi`] reference.

-## Push files with Git LFS
+## Tips and tricks for large uploads


are we sure we don't want to put that content on its own doc page? (no strong opinio)

No strong opinion either. I thought the content was a bit light to get its own page

Actually, I was initially looking for this PR in hub-docs, so yeah it could make sense haha, but it seems to me that some of the advice and limits are specific to huggingface_hub (or at least to uploads through the HTTP API), so we can keep it here, at least for now

julien-c

let's wait for @Pierrci quick review before merging please 🙏

Co-authored-by: Julien Chaumond <julien@huggingface.co>

…e/huggingface_hub into tutorial-upload-large-dataset

stevhliu

Nice, love the additional columns you added to the table! 🤗

docs/source/guides/upload.md

Wauplin · 2023-07-21T16:30:54Z

Good, thanks @stevhliu I made the little change.

Final review from @Pierrci and we are good to go! 🎉

Pierrci

I was finally able to take a look 😄

I don't think we should advertise hard limits, if you reach them it means your repo is going to be very hard to manage (both for the user and for us); we really want to steer people as much as we can toward numbers lower than those. We also want to keep some leeway and be able to change those limits at our discretion to protect our infra if needed.

So I pushed suggestions that advertise recommendations instead - and as for any recommendations, you can choose to ignore them if you're a player, but sometimes it's not a good idea, particularly if you go way above them :)

docs/source/guides/upload.md

Pierrci · 2023-08-15T22:55:10Z

docs/source/guides/upload.md

@@ -371,11 +377,95 @@ In addition to [`upload_file`] and [`upload_folder`], the following functions al

 For more detailed information, take a look at the [`HfApi`] reference.

-## Push files with Git LFS
+## Tips and tricks for large uploads


Actually, I was initially looking for this PR in hub-docs, so yeah it could make sense haha, but it seems to me that some of the advice and limits are specific to huggingface_hub (or at least to uploads through the HTTP API), so we can keep it here, at least for now

Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>

Wauplin · 2023-08-16T13:16:11Z

Thanks for the review @Pierrci! Completely understand your point here about not to commit to much on the hard limits. I have applied your recommendations. Could you have a final look and we are good to merge? :)

Pierrci · 2023-08-16T15:22:54Z

Thanks @Wauplin! I did a Grammarly pass and pushed additional changes (the VSCode extension is really nice for that!), everything LGTM on my end!

Wauplin · 2023-08-16T15:38:27Z

Perfect! Thanks for the final pass, I'm finally merging this PR 😄

julien-c · 2023-08-21T09:46:45Z

docs/source/guides/upload.md

+      For example, json files can be merged into a single jsonl file, or large datasets can be exported as Parquet files.
+    - The maximum number of files per folder cannot exceed 10k files per folder. A simple solution is to
+      create a repository structure that uses subdirectories. For example, a repo with 1k folders from `000/` to `999/`, each containing at most 1000 files, is already enough.
+- **File size**: In the case of uploading large files (e.g. model weights), we strongly recommend splitting them **into chunks of around 5GB each**.


late to the party but i would have done larger, e.g. 20GB or at least 10GB (Cloudfront caches up to 30GB if i'm not mistaken)

@Pierrci @huggingface/moon-landing-back

yes, but I don't really see the interest; as mentioned just below in the doc splitting into small chunks is better for uploading/downloading and retries, while I'm not sure there are a lot of advantages in doing chunks of 30GB?

severo · 2023-08-24T15:23:18Z

@julien-c about https://discuss.huggingface.co/t/is-there-a-size-limit-for-dataset-hosting/14861/13?u=severo, we had the information in a previous version of this PR (see the table above: #1565 (comment)).

Should we add it again?

julien-c · 2023-08-25T18:39:06Z

i think we didn't want to do a table with "recommended limits" and "hard limits", so maybe just add a sentence like:

in all cases no single LFS file will be able to be >50GB. I.e. 50GB is the hard limit for single file size.

severo · 2023-08-25T19:35:06Z

OK: #1624

tips about large uplaods

438d639

Wauplin added the documentation Improvements or additions to documentation label Jul 20, 2023

Wauplin requested review from julien-c and stevhliu July 20, 2023 16:05

typos

53caabe

lhoestq approved these changes Jul 20, 2023

View reviewed changes

docs/source/guides/upload.md Outdated Show resolved Hide resolved

docs/source/guides/upload.md Outdated Show resolved Hide resolved

docs/source/guides/upload.md Outdated Show resolved Hide resolved

stevhliu approved these changes Jul 20, 2023

View reviewed changes

coyotte508 approved these changes Jul 20, 2023

View reviewed changes

osanseviero approved these changes Jul 20, 2023

View reviewed changes

Wauplin and others added 3 commits July 21, 2023 11:44

Merge branch 'main' into tutorial-upload-large-dataset

273883c

Apply suggestions from code review

16dd98a

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

add table

5487aea

julien-c reviewed Jul 21, 2023

View reviewed changes

docs/source/guides/upload.md Outdated Show resolved Hide resolved

julien-c reviewed Jul 21, 2023

View reviewed changes

julien-c requested a review from Pierrci July 21, 2023 14:35

julien-c reviewed Jul 21, 2023

View reviewed changes

Wauplin and others added 3 commits July 21, 2023 17:29

Apply suggestions from code review

e94cf1f

Co-authored-by: Julien Chaumond <julien@huggingface.co>

fix

4193f9b

Merge branch 'tutorial-upload-large-dataset' of github.com:huggingfac…

1383f0e

…e/huggingface_hub into tutorial-upload-large-dataset

stevhliu approved these changes Jul 21, 2023

View reviewed changes

docs/source/guides/upload.md Outdated Show resolved Hide resolved

Mt erge branch 'main' into tutorial-upload-large-dataset

67250e7

Pierrci reviewed Aug 15, 2023

View reviewed changes

Apply suggestions from code review

81847ff

Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>

grammarly pass

4c2bb66

Pierrci approved these changes Aug 16, 2023

View reviewed changes

Wauplin merged commit 9d418ae into main Aug 16, 2023
4 checks passed

Wauplin deleted the tutorial-upload-large-dataset branch August 16, 2023 15:38

julien-c reviewed Aug 21, 2023

View reviewed changes

Wauplin mentioned this pull request Sep 29, 2023

Harmonize warnings/errors/documentation related to file size limit #995

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tips to upload large models/datasets #1565

Tips to upload large models/datasets #1565

Wauplin commented Jul 20, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jul 20, 2023 •

edited

Loading

Wauplin commented Jul 20, 2023

lhoestq left a comment

stevhliu left a comment

Wauplin commented Jul 20, 2023

osanseviero left a comment

osanseviero Jul 20, 2023

Wauplin Jul 21, 2023

julien-c Jul 21, 2023

Wauplin commented Jul 21, 2023

julien-c Jul 21, 2023

Wauplin Jul 21, 2023

Pierrci Aug 15, 2023

julien-c left a comment

stevhliu left a comment

Wauplin commented Jul 21, 2023

Pierrci left a comment •

edited

Loading

Pierrci Aug 15, 2023

Wauplin commented Aug 16, 2023

Pierrci commented Aug 16, 2023

Wauplin commented Aug 16, 2023

julien-c Aug 21, 2023

Pierrci Aug 21, 2023

severo commented Aug 24, 2023

julien-c commented Aug 25, 2023

severo commented Aug 25, 2023

Tips to upload large models/datasets #1565

Tips to upload large models/datasets #1565

Conversation

Wauplin commented Jul 20, 2023 • edited Loading

HuggingFaceDocBuilderDev commented Jul 20, 2023 • edited Loading

Wauplin commented Jul 20, 2023

lhoestq left a comment

Choose a reason for hiding this comment

stevhliu left a comment

Choose a reason for hiding this comment

Wauplin commented Jul 20, 2023

osanseviero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wauplin commented Jul 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julien-c left a comment

Choose a reason for hiding this comment

stevhliu left a comment

Choose a reason for hiding this comment

Wauplin commented Jul 21, 2023

Pierrci left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wauplin commented Aug 16, 2023

Pierrci commented Aug 16, 2023

Wauplin commented Aug 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

severo commented Aug 24, 2023

julien-c commented Aug 25, 2023

severo commented Aug 25, 2023

Wauplin commented Jul 20, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jul 20, 2023 •

edited

Loading

Pierrci left a comment •

edited

Loading