Training Script for LLama and GPT

[awsaf49]

I think it is really great to see this repo has some SOTA nlp models like GPT 2 and LLama v2. I think it would be nice to have a training script. Any plans to publish a training code?

[leondgarse]

Currently I only got a torch one on tinyshakspear torch_text_train_script.py from Github karpathy/nanoGPT. The basic components for training text model are not so hard, and we have bunch of examples in TF tutorial. May try a script if some spare time.
BTW, also got a torch_clip_train_script.py for comparing with TF one, but my training results using TF one is not so good. Torch one can drop to loss ~=1.6, while TF loss > 2.5 using almost similar hyper parameters. Still needs more tests.

[awsaf49]

For llamav2 training you may also check karpathy/llama2.c/train.py.

Regarding clip, which torch script are using for getting loss~=1.6 ?

[leondgarse]

It's this one torch_clip_train_script.py. The actual loss value doesn't matter, as it's only on my test datsaset coco_dog_cat.tar.gz.