Clean up DPO example

[lewtun]

What does this PR do?

This PR standardises the dpo.py script to use the ultrafeedback_binarized dataset instead of the huge Anthropic one. I also tweaked the hparams so that they "just work" and use a more realistic SFT model.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lewtun]

This type of debugging arg shouldn't live in the lib IMO

[qgallouedec]

Let's remove them all

[lewtun]

Good idea! Done in ddf30cb

[lewtun]

There seems to be some rich related issues with the tests now that I removed the sanity_check arg. I'm taking a look

[lewtun]

Very strange, running with the rich callback gives an issue where the progress_bar is set to None during evaluation even though it is correctly initialised at the start of training:

Traceback (most recent call last):
  File "/fsx/lewis/git/hf/trl/trl/commands/scripts/dpo.py", line 176, in <module>
    metrics = trainer.evaluate()
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 3666, in evaluate
    output = eval_loop(
  File "/fsx/lewis/git/hf/trl/trl/trainer/dpo_trainer.py", line 1651, in evaluation_loop
    initial_output = super().evaluation_loop(
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 3888, in evaluation_loop
    self.control = self.callback_handler.on_prediction_step(args, self.state, self.control)
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer_callback.py", line 503, in on_prediction_step
    return self.call_event("on_prediction_step", args, state, control)
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer_callback.py", line 507, in call_event
    result = getattr(callback, event)(
  File "/fsx/lewis/git/hf/trl/trl/trainer/callbacks.py", line 124, in on_prediction_step
    self.prediction_task_id = self.prediction_bar.add_task(
AttributeError: 'NoneType' object has no attribute 'add_task'
Traceback (most recent call last):
  File "/fsx/lewis/git/hf/trl/trl/commands/scripts/dpo.py", line 176, in <module>
    metrics = trainer.evaluate()
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 3666, in evaluate
    output = eval_loop(
  File "/fsx/lewis/git/hf/trl/trl/trainer/dpo_trainer.py", line 1651, in evaluation_loop
    initial_output = super().evaluation_loop(
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer.py", line 3888, in evaluation_loop
    self.control = self.callback_handler.on_prediction_step(args, self.state, self.control)
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer_callback.py", line 503, in on_prediction_step
    return self.call_event("on_prediction_step", args, state, control)
  File "/fsx/lewis/miniconda3/envs/trl/lib/python3.10/site-packages/transformers/trainer_callback.py", line 507, in call_event
    result = getattr(callback, event)(
  File "/fsx/lewis/git/hf/trl/trl/trainer/callbacks.py", line 124, in on_prediction_step
    self.prediction_task_id = self.prediction_bar.add_task(
AttributeError: 'NoneType' object has no attribute 'add_task'

Command to repro:

TRL_USE_RICH=True CUDA_VISIBLE_DEVICES="" trl dpo --max_steps 1 --output_dir tmp-dpo --model_name_or_path trl-internal-testing/tiny-random-LlamaForCausalLM --dataset_name trl-lib/ultrafeedback_binarized --learning_rate 1e-4 --lr_scheduler_type cosine --dataset_num_proc 48

[lewtun]

FYI @kashif @qgallouedec @edbeeching we should not add this logic into the example scripts IMO - it's best solved by adding support for something like the dataset mixer we have in the handbook or H4 repo

[kashif]

ah yes my bad!