Add Chunked SimPO Loss

[pramodith]

Summary

This PR adds the Simple Preference Optimization Loss function. The only difference between SimPO and CPO is a margin term gamma which specifies that the preferred response should be atleast gamma logit points better than the losing response.

$$SimPOLoss = -\log(\sigma(\beta\log(\pi_\theta(y_c|x)) - \beta\log(\pi_\theta(y_r|x)) - \gamma))$$

Note that SimPO explicitly specifies that $$\pi_\theta(y|x)$$ needs to be normalized by length, unlike DPO.

This corresponds to Eq 6 in the paper.

Testing Done

GPU A100-80G-SXM

• Hardware Type:
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[shivam15s]

Great PR @pramodith ! Could you rebase?

[pramodith]

Great PR @pramodith ! Could you rebase?

Good to go now!

[shivam15s]

lgtm

[shivam15s]

Shouldn't this be normalized by length, as you said in the description?

[shivam15s]

Nvm, the logps are already averaged