Various args and test fix

[qgallouedec]

Some test doesn't pass locally. Consequently, I propose the following fix/improvement:

1. Remove unused peft_test marker

tests/test_ppo_trainer.py:910
  /fsx/qgallouedec/trl/tests/test_ppo_trainer.py:910: PytestUnknownMarkWarning: Unknown pytest.mark.peft_test - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.peft_test

2. Don't report to wandb

wandb does not cope well with multiple executions in the same process, and generates errors such as

wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "model/num_parameters" from 1658681 to 1663801

the easiest is not to report during test. Already fixed for reward tester in #1895

3. Don't share model across tests

Also partially addressed in #1895. When sharing the model, it creates conflicts between tests, due to model re-training mostly.

FAILED tests/test_dpo_trainer.py::DPOTrainerTester::test_dpo_trainer_w_dataset_num_proc - _pickle.PicklingError: Cannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.

The solution is to replace setUpClass by setUp (so that the setup is not shared)

4. Fixes configuration argument being overrided by deprecated default values

Fixes #1905 and adds tests.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Finally 😮‍💨