Has anyone tried Dolly-like models?

[kha84]

I just watched the latest video of my favorite youtuber - https://www.youtube.com/watch?v=AWAo4iyNWGc&t=14s and was wondering, if someone has already quantized & converted one of these to be compatible with llama.cpp?
The beauty of Dolly-like models is that they're based on open source gpt-j-6B from EleutherAI, so noone will be hunting us for using them without an ask.

[pikalover6]

ggml already supports gpt-j, you should just be able to convert and quantize them.

[gorborukov]

I did not succeed with the convert-h5-to-ggml.py script and the model Dolly_GPT-J-6b. Script thinks for a while and crashes with the status "Killed".

[kha84]

I did not succeed with the convert-h5-to-ggml.py script and the model Dolly_GPT-J-6b. Script thinks for a while and crashes with the status "Killed".

Prob out-of-memory killer played his part? Check journalctl / syslog and try to monitor system resources while running it

[Ghatage]

@leroycep I'm wondering if I should try to use float32 during quantization of databricks/dolly-v2-12b. Perhaps that would give better accuracy (but require 2x the memory)

[Ghatage]

Just checking back, I converted the new dolly model with using float32 for the weights. The accuracy has greatly improved compared to the float16 linked above. However the model size grown 6 times the original (7gb -> 47 gb).

[leroycep]

It looks like the dolly (and pythia/GPTNeoX) were trained using 16 bit, so if the accuracy is improving, it's likely that only the computations need to changed, and not the stored values.

• 12b config shows torch_dtype as bfloat16: https://huggingface.co/databricks/dolly-v2-12b/blob/main/config.json
• GPTNeoX was trained using fp16: https://huggingface.co/docs/transformers/model_doc/gpt_neox
• The pythia dataset uses fp16 precision: https://github.com/EleutherAI/pythia/blob/main/models/12B/pythia-12b.yml#L61-L69

I've started work on separating the GPTNeoX runner from the cformers library into a standalone executable. I'm using the Zig language, and I've tweaked the library to mmap the file, which makes the file load much faster. I haven't gotten evaluation working yet. There have been changes to GGML that break the cformers code for GPTNeoX, and I don't understand enough about GGML or GPTNeoX to fix them. I think I'll work through Karpathy's Neural Network course before trying to continue this effort.

[ghost]

I think there'll be a lot more interest on these courses going forwards with all these new models coming out. The other applications didn't really do it for me but with LLMs I feel that I've been bitten by the AI bug...

[ggerganov]

GPT-NeoX basic inference is now demonstrated in the ggml repo, via the new StableLM Alpha models:

https://github.com/ggerganov/ggml/tree/master/examples/stablelm

[jon-chuang]

I will investigate this as I would like to see foundation models and their variants outside of llama be blessed with the gift of ggml.