[Feature] HF_HUB_ENABLE_HF_TRANSFER=0? Extremely slow Model downloads from Hugging Face

[lucasmelogithub]

Priority

Undecided

OS type

Ubuntu

Hardware type

Xeon-GNR

Running nodes

Single Node

Description

Is there a technical reason for HF_HUB_ENABLE_HF_TRANSFER=0 on the compose.yaml files?
I did not run into download issues early last year. But starting in December, Hugging Face Models downloads(ex: Intel/neural-chat-7b-v3-3) are now taking 24 minutes to download.

I updated the Xeon ChatQnA compose.yaml file to HF_HUB_ENABLE_HF_TRANSFER=1 and it now only took 15 seconds.

If no concerns, I can submit a PR to update the compose.yaml files.

Per docs https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables#hfhubenablehftransfer

HF_HUB_ENABLE_HF_TRANSFER
Set to True for faster uploads and downloads from the Hub

[xiguiw]

Some more info about HF_HUB_ENABLE_HF_TRANSFER
hf_transfer is an experimental feature, so it may not be enabled by default in all Hugging Face libraries. Make sure your huggingface_hub library is up to date to use this feature.

HF_HUB_ENABLE_HF_TRANSFER=1
What it does: Enables the hf_transfer library, which is a custom, high-performance file transfer mechanism developed by Hugging Face.

Purpose: It is designed to significantly speed up file downloads from the Hugging Face Hub, especially for large files or in environments with high latency or limited bandwidth.

How it works: hf_transfer uses optimizations like parallel downloads and better connection handling to improve download speeds.

When to use: Set this to 1 if you want faster downloads and are working with large datasets or models from the Hugging Face Hub.

HF_HUB_ENABLE_HF_TRANSFER=0
What it does: Disables the hf_transfer library and falls back to the default file transfer mechanism (usually Python's requests library or similar).

Purpose: This is the default behavior if hf_transfer is not explicitly enabled.

How it works: Downloads files using standard HTTP requests, which may be slower for large files or in suboptimal network conditions.

When to use: Set this to 0 if you encounter issues with hf_transfer or prefer to use the standard download mechanism.

[lucasmelogithub]

We have automated the deployment on Xeon end-to-end. https://github.com/opea-project/GenAIExamples/tree/main/ChatQnA#-automated-terraform-deployment-using-intel-optimized-cloud-modules-for-terraform

HF_HUB_ENABLE_HF_TRANSFER=0 is "unusable" due to the amount of time it takes to download the Models (~25 minutes).

HF_HUB_ENABLE_HF_TRANSFER=1 reduced that time to seconds.

[eero-t]

@lianhao Please check above. Helm charts default to HF_HUB_ENABLE_HF_TRANSFER=0, but now that they've been switched to using a specific version of hf-downloader, maybe it would make sense to change that value?

[lianhao]

@lianhao Please check above. Helm charts default to HF_HUB_ENABLE_HF_TRANSFER=0, but now that they've been switched to using a specific version of hf-downloader, maybe it would make sense to change that value?

Issue opea-project/GenAIInfra#744 created for tracking purpose

[eero-t]

Per docs https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables#hfhubenablehftransfer

I just noticed this in that doc:

hf_transfer lacks several user-friendly features such as resumable downloads and proxies

=> Lack of proxy support would make this a no-go. Lack of resumable downloads is pretty significant downside too.