KTO - support loading the adapter twice

[claralp]

For DPOTrainer there exists the option to load the Adapter from SFT training twice, as in Reference model considerations with PEFT - load-the-adapter-twice:

# Initialize the trainer, without a ref_model param.
dpo_trainer = DPOTrainer(
    model,
    ...
    model_adapter_name="train",
    ref_adapter_name="reference",
)

In DPOTrainer this allows me to use a larger batch size and speed up training.
Would be cool to have this option in KTOTrainer as well. Anyone working on this already?

[PhilipMay]

Hi @lewtun , we had a discussion about KTO. Do you already work on this or should we come up with a PR?

We would try and use the code from DPO and apply it to KTO to implement this.

[claralp]

@younesbelkada @kashif, is this a desired feature for you as well?
If yes, is someone already implementing it or should we come up with a PR?

[kashif]

I believe would be good to have yes, and no one is working on it

[PhilipMay]

That feature would be super useful @claralp . Thanks.