[Question] Can I load a the huggingface llama model aswell?

[maxkraft7]

I have downloaded the llama-model from here. There it got converted to be compatible with pytorch. But the biggest advantage is that it is actually available. The magnet link from that PR has no trackers so it's not starting to download at least for me. And the IPFS files always have a different checksum when I download them. So since I only have the huggingface-version is it possible use their model with llama.cpp somehow?

[KASR]

With some modifications you might be able to use this: alpaca-convert-colab, I haven't tested it however.

You would only need to run the first two blocks up to torch.save(model, "consolidated.01.pth") (and renaming the file) once you have the .pth and the params.json files you can use the conversion scripts of llama cpp to obtain the ggjt files.

[MillionthOdin16]

Here's what I used last night. I'm not sure if this is the same thing KASR is mentioning or not.

Convert.py

[Technetium1]

@maxkraft7 I enjoy automatic tracker additions via https://github.com/c0re100/qBittorrent-Enhanced-Edition but you can add these manually to anything https://github.com/ngosang/trackerslist/blob/master/trackers_all.txt

[ahoho]

I did it in two steps: I modified export_state_dict_checkpoint.py from alpaca-lora to create a consolidated file, then used a slightly modified convert-pth-to-ggml.py from llama.cpp (I didn't want to bother with sharding logic, but the conversion script expects multiple .pth checkpoints). Don't forget to clean up the intermediate files :)

Here's a gist with my changes. Tested with the 7B-30B LLaMA models from decapoda-research

I mentioned it elsewhere, but if you're quantizing, I've had better results with q4_0 than q4_1. YMMV though

[tonyaw]

Here's what I used last night. I'm not sure if this is the same thing KASR is mentioning or not.

Convert.py

@MillionthOdin16 , I got following error via your script. Could you please help to check?
[root@qd-graphics koboldai-client]# bin/micromamba run -r runtime -n koboldai python3 sandbox/convert/convert.py models/gpt4-x-alpaca/
Traceback (most recent call last):
File "/home/root/sandbox/convert/convert.py", line 674, in
main()
File "/home/root/sandbox/convert/convert.py", line 665, in main
model, model_path = load_some_model(args.model)
File "/home/root/sandbox/convert/convert.py", line 621, in load_some_model
model = lazy_load_torch(path)
File "/home/root/sandbox/convert/convert.py", line 497, in lazy_load_torch
model = unpickler.load()
TypeError: 'staticmethod' object is not callable