Added error check to RLOO, PPOv2, OnlineDPO that ref_policy and policy have different identities

[RylanSchaeffer]

As discussed in Issue 2046, we add an error check to make sure that the ref_policy and policy have different identities.

This is just a quick and dirty demonstration to show what I have in mind. Feedback is very welcome!

[RylanSchaeffer]

As a quick reminder, is is a way of checking whether two Python objects have the same identity: https://stackoverflow.com/a/3647560/4570472

x = [1, 2, 3]
y = x
z = [1, 2, 3]
y == x  # True
y is x    # True
z == x  # True
z is x  # False

[lewtun]

Thanks a lot for adding this sanity check @RylanSchaeffer ! If I understand correctly, this scenario can occur for any trainer where there is a reference model - would you like to expand your check to the other trainers as well?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Useful check, thanks!

[qgallouedec]

Failing test not related to this PR

[qgallouedec]

Thanks @RylanSchaeffer!