support phi 3 #27
support phi 3 #27
Conversation
|
@davidgxue We have validated this PR with modfications on real model quant with pre/post PPL. Ready for merge. Thank you for the Phi3 code! |
|
Test Result: Phi-3-medium-4k-instruct |
|
Awesome work and thank you for adding me on here haha! AutoGPTQ library still seems glitchy? Both LLAMA 3 and Phi3 quants from AutoGPTQ has gibberish output issues when using with transformers... Hopefully this repo is better maintained! |
|
@davidgxue Phi3 appears to contain fused modules and from our experience, any fused modules, will result in less than ideal calibration. That's why you see the Phi-3 post quant much higher (diff vs pre-quant) than other models. As such, worse output is expected but I do see an issue with bad/gibberish. Can you provide me with examples of gibberish (are these generated on native or gptq qunated) models? With our team, expert in gptq quants, we may be able to src the bug. I would highly recommend you to re-quant them use GPTQModel as our code vs AutoGPTQ will in many cases by default generate higher quality quants due to the optimizations we have made. We also added lots of protection against setting toggles that generate bad quants.
This is our pledge, mission statement, and core reason why I pushed our team to fork AutoGPTQ. |
|
Oh yeah the fused modules part is definitely expected. What I meant is LLAMA 3 and Phi 3 (well i guess they are both llama family maybe) literally not working. I have had this issue open for a long time on AutoGPTQ repo, and im not the only one cuz I released a few GPTQ quants on huggingface and im being spammed in DM or huggingface issues about people experiencing the exact same thing.
Love it! Will follow you guys and see if there are stuff I can help out on! |
* initial attempt to add phi 3 (may need unfuse) * fix Phi3GPTQ base_modules --------- Co-authored-by: David Xue <xuegdxw@gmail.com>
No description provided.