added "tie_word_emebeddings" setting necessary for Llama 3.2

[martin-gorner]

Starting from Llama 3.2, the models use tied embeddings which means that the checkpoints no longer have a separate set of weights for reverse embeddings. This change allows the Transformer "tie_word_embeddings" setting to be read from config.json and instantiates the Llama3 ReversibleEmbedding class with the correct tie_weights setting.

Without this, loading Llama 3.2 errors out with the following error message:
"SafetensorError: File does not contain tensor lm_head.weight"

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[martin-gorner]

As implemented, the "tie_word_emebeddings" is a mandatory config parameter but I have checked that it appears in all Lllama releases to date:
https://huggingface.co/meta-llama/Meta-Llama-3-70B/blob/main/config.json
https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct/blob/main/config.json
https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/config.json
https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/config.json
https://huggingface.co/meta-llama/Llama-3.2-3B/blob/main/config.json
https://huggingface.co/meta-llama/Llama-Guard-3-1B/blob/main/config.json
https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/config.json
https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct/blob/main/config.json
https://huggingface.co/meta-llama/Llama-2-70b-chat-hf/blob/main/config.json
https://huggingface.co/meta-llama/Llama-2-70b-hf/blob/main/config.json
https://huggingface.co/meta-llama/CodeLlama-34b-hf/blob/main/config.json
https://huggingface.co/meta-llama/CodeLlama-13b-Python-hf/blob/main/config.json
https://huggingface.co/meta-llama/CodeLlama-7b-Instruct-hf/blob/main/config.json
etc...

with the exception of some Llama 2 models that do not even have a config.json file like this one:
https://huggingface.co/meta-llama/Llama-2-70b-chat/tree/main
Loading will fail on those with or without the fix.

[osanseviero]

Hi 👋 The Llama 2 model you link to (https://huggingface.co/meta-llama/Llama-2-70b-chat) is not a transformers compatible repository but the original research checkpoints that were released. The transformers compatible repo is https://huggingface.co/meta-llama/Llama-2-70b-chat-hf.

Since Llama 3, Meta has released the models with the transformers-compatible weights as the primary release artifact, with the original research checkpoints in an original repository on the Hub.

[divyashreepathihalli]

Thanks for the update Martin!