[Feature] support Nemo

[lihan]

Motivation

The best multilingual open source small LM.

https://mistral.ai/news/mistral-nemo/

Related resources

No response

Additional context

No response

[zhyncs]

The structure is essentially the same as that of a Llama, supporting it shouldn't be too difficult, please stay tuned. ref https://github.com/vllm-project/vllm/pull/6548/files

[maxin9966]

@zhyncs Does it support the AWQ format?

[AllentDan]

AllentDan@d14ce34

I found that mistral and llama used different input_dim for attention.output layer. Do we need another type or name argument to distinguish the model architecture? @lvhan028 @lzhangzz
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L199
https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L303

[lzhangzz]

The input dim of attention.output should be computed as head_num * head_dim. The use of hidden_units_ is a bug.