Did you have exploding gradients (loss = nan)?

[vetka925]

I tried to fine-tune LLaMA with LoRA on my dataset. Firstly, it has learnt good - loss decreased. But after a few hundred steps loss started increasing and became nan.

[AngainorDev]

This would usually point to either a noisy dataset, either too high a LR.

Make sure the various settings, like warmup and batch size are consistent with your dataset size.