[MODEL] Add Telechat2 (China Telecom)

[1096125073]

Add support for the Telechat model (VLLM now supports the Telechat model).

[Qubitium]

@1096125073 Do you have HF model url for this model? We need to download this model and ci test

[1096125073]

@1096125073 Do you have HF model url for this model? We need to download this model and ci test

Yes, you can try this one
https://huggingface.co/Tele-AI/TeleChat2-7B

[Qubitium]

@1096125073 Thank you for the PR. I will have @LRL-ModelCloud take over the PR changes from here. He will fix this PR so it can run correctly in GPTQModel with CI tests. Right now, this code will not run since gptqmodel uses different structure on how to define models.

[1096125073]

@1096125073 Thank you for the PR. I will have @LRL-ModelCloud take over the PR changes from here. He will fix this PR so it can run correctly in GPTQModel with CI tests. Right now, this code will not run since gptqmodel uses different structure on how to define models.

Sorry, I just discovered that the properties of this library are different from AutoGPTQ and have been updated

[Qubitium]

@1096125073 Thank you for the PR. I will have @LRL-ModelCloud take over the PR changes from here. He will fix this PR so it can run correctly in GPTQModel with CI tests. Right now, this code will not run since gptqmodel uses different structure on how to define models.

Sorry, I just discovered that the properties of this library are different from AutoGPTQ and have been updated

No problem. We wil fix this. But stop force-pushing so we can fix this PR.

[Qubitium]

@1096125073 We are testing.

1. If you are with telechat2 team, please ask manager to add torch_dtype to config.json

https://huggingface.co/Tele-AI/TeleChat2-7B/blob/main/modeling_telechat2.py#L186

The assert codes here are strange. Why is it forcing (asserting) cuda? GPTQModel supports inference on multiple devices and does not require cuda.

[1096125073]

@1096125073 We are testing.

1. If you are with telechat2 team, please ask manager to add torch_dtype to config.json

https://huggingface.co/Tele-AI/TeleChat2-7B/blob/main/modeling_telechat2.py#L186

The assert codes here are strange. Why is it forcing (asserting) cuda? GPTQModel supports inference on multiple devices and does not require cuda.

Sorry, this is being handled by another team, but we are currently organizing the code and submitting the PR to the transformer library. I believe this will be resolved soon.

[Qubitium]

@1096125073 We are running into dtype assert bugs in forward if we load the original model as bfloat16. float16 can run. Can you confirm the model .bin files are native bfloat16 or float16?

[1096125073]

@1096125073 We are running into dtype assert bugs in forward if we load the original model as bfloat16. float16 can run. Can you confirm the model .bin files are native bfloat16 or float16?

yes, 7b run as float16