Can't have load_best_model_at_end with explicit test_datasets

[chiragjn]

Please check that this issue hasn't been reported before.

• I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Ideally, it should be possible to have load_best_model_at_end: True with test_datasets: [...] as it is already allowed with val_set_size.

Not sure why it was restricted, tried doing a git blame, but there was a massive refactor so hard to trace

Current behaviour

Currently it crashes with

.../lib/python3.10/site-packages/transformers/trainer_callback.py", line 578, in on_train_begin
    assert args.load_best_model_at_end, "EarlyStoppingCallback requires load_best_model_at_end = True"
AssertionError: EarlyStoppingCallback requires load_best_model_at_end = True

because load_best_model_at_end gets disabled

https://github.com/OpenAccess-AI-Collective/axolotl/blob/ea00dd08528399405943f880b27ecc60e4d83cdf/src/axolotl/core/trainer_builder.py#L849-L859

Steps to reproduce

Train qlora model but set

test_datasets:
 - ...
val_set_size: 0
load_best_model_at_end: True

Config yaml

No response

Possible solution

Slightly modify the condition to allow load_best_model_at_end when either test_datasets is set or val_set_size > 0

Which Operating Systems are you using?

• Linux
• macOS
• Windows

Python Version

3.10

axolotl branch-commit

main/ea00dd08528399405943f880b27ecc60e4d83cdf

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this bug has not been reported yet.
• I am using the latest version of axolotl.
• I have provided enough information for the maintainers to reproduce and diagnose the issue.

[dameikle]

I raised a pull request here that I believe resolves this as I had a similar need #1291