RuntimeError: Unsloth: llama.cpp GGUF seems to be too buggy to install.

[Jiar]

prerequisites

%%capture
# Installs Unsloth, Xformers (Flash Attention) and all other packages!
!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install --no-deps "xformers<0.0.26" trl peft accelerate bitsandbytes

run code:

if True: model.push_to_hub_gguf(hf_model_name, tokenizer, quantization_method = "q4_k_m", token = token)

error:

/usr/bin/ld: unicode-data.cpp:(.text._ZNSt8multimapIjjSt4lessIjESaISt4pairIKjjEEEC2ESt16initializer_listIS4_ERKS1_RKS5_[_ZNSt8multimapIjjSt4lessIjESaISt4pairIKjjEEEC5ESt16initializer_listIS4_ERKS1_RKS5_]+0xbe): undefined reference to `std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)'
collect2: error: ld returned 1 exit status
make: *** [Makefile:956: vdot] Error 1
make: Leaving directory '/content/llama.cpp'
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-19-4b22232c01dc> in <cell line: 3>()
      1 # Save to q4_k_m GGUF
      2 #if True: model.save_pretrained_gguf(model_name, tokenizer, quantization_method = "q4_k_m")
----> 3 if True: model.push_to_hub_gguf(hf_model_name, tokenizer, quantization_method = "q4_k_m", token = token)
      4 
      5 # Save to 16bit GGUF

2 frames
/usr/local/lib/python3.10/dist-packages/unsloth/save.py in install_llama_cpp_old(version)
    783     # Check if successful
    784     if not os.path.exists("llama.cpp/quantize"):
--> 785         raise RuntimeError(
    786             "Unsloth: llama.cpp GGUF seems to be too buggy to install.\n"\
    787             "File a report to llama.cpp's main repo since this is not an Unsloth issue."

RuntimeError: Unsloth: llama.cpp GGUF seems to be too buggy to install.
File a report to llama.cpp's main repo since this is not an Unsloth issue.

screenshot

[danielhanchen]

@Jiar Might be relevant: #476

Please try uninstalling peft==0.11.0 then install peft==0.10.0 as a temporary fix

[Jiar]

@Jiar Might be relevant: #476

Please try uninstalling peft==0.11.0 then install peft==0.10.0 as a temporary fix

Thanks!

[joytsay]

@danielhanchen @Jiar
for what is worth this RuntimeError: Unsloth: llama.cpp GGUF seems to be too buggy to install. happened again
and it was less a hassle to save to VLLM hf first

model.save_pretrained_merged("/your/awesome/model/", tokenizer, save_method = "merged_16bit",)

and then use llama.cpp to convert-hf-to-gguf

cd unsloth/llama.cpp

python3 convert-hf-to-gguf.py /your/awesome/model/

[danielhanchen]

@joytsay So sorry on the delay! Ye sadly sometimes directly using Unsloth for GGUF might be worse than simply using GGUF directly - if using it directly works, then that's good!