Support for LLaMA models

[zphang]

Can be merged after huggingface/transformers#21955

[dustydecapod]

Have you tested this? In my local testing previously, implementing this exact change, I ran into an issue where the trainers didn't like not having a pad token in the tokenizer.

[younesbelkada]

@zoidbb does adding:

tokenizer.pad_token = tokenizer.eos_token

helps solving your issue?

[zphang]

Have you tested this? In my local testing previously, implementing this exact change, I ran into an issue where the trainers didn't like not having a pad token in the tokenizer.

Yes, but I tested with my own data pipeline/data collator.

[dustydecapod]

@zoidbb does adding:

tokenizer.pad_token = tokenizer.eos_token

helps solving your issue?

Oddly not, I tried this and it still complained. I'll provide more details this evening when I'm done with work.

[tloen]

@zoidbb you may also need:

tokenizer.pad_token_id = tokenizer.eos_token_id

[pacman100]

Thank you @zphang for adding this! 🤗