Direct Preference Optimization

[maxjeblick]

This PR adds DPO (https://github.com/eric-mitchell/direct-preference-optimization) as a new problem type.
Also, IPO can be selected via the associated loss function.

Apart from adding the new problem type, the following changes have been made:

• A new dataset for DPO is loaded by default https://huggingface.co/datasets/Intel/orca_dpo_pairs
• Code to create HH DPO dataset compatible to llm studio format has been added
• Default Dataset creation will now show a progress pop up that informs the user
• Some small refactoring in various places

Possible follow-up work:

• Add better insights; Validation Insights and Train Data Insights do not show the rejected answer. I haven't added it to make the PR slimmer.
• Add Reward margin plot to charts.
• Add problem type dropdown selection to dataset import

[pascal-pfeiffer]

Thanks a LOT @maxjeblick

Very high quality PR, and it works flawless on many different setups and datasets that I tested.
One thing, that we should change in the future is the dataset import. Currently, only the default settings for Causal modeling can be set during import, so one always needs to change that when starting an experiment in e.g. DPO training.

Also, rewards are logged, but never displayed (only when using neptune) -> potential good new feature as you mentioned.

While still not easy to get better results over standard fine-tuning, as DPO is way more user friendly, I am rooting for fully replacing RLHF (by PPO) with it in a subsequent PR.

Minor change needed:

• Rejected Answer Column is missing a tooltip

[pascal-pfeiffer]

Nitpick:
Maybe, we should replace by a "h2ogpt-gm" fine-tune that already has the same prompting style.

[maxjeblick]

Do you have a specific model in mind? I could also change the default prompt style values.

[pascal-pfeiffer]

Yes, probably just change default values to something that works. Such as using mistralai/Mistral-7B-Instruct-v0.1 and its prompting style.