Best practices for automated parameter sweeps

[HenrikAsmuth]

If have started playing around with pytorch-lightning and I am really happy with how it streamlines many aspects of training and evaluation of pytorch models. Particularly, I like how the LightningCLI facilitates the creation and usage of yaml config files.

However, so far I haven't found a really satisfying solution for doing parameter sweeps. In my own set-ups (without Lightning) I typically have some default config file that is used to set-up the training and then another config which only specifies which parameters of the default config should be swept over (usually providing a list of values).

I came across a few solutions but none of them seem optimal:

• The lightning Tuner class can be used for sweeping learning rate and batch size but is not general in the sense that any parameter (model hyperparameters, settings of the dataset, etc.) can be changed

• I came across many posts combining their lightning set-up with Optuna or others. But obviously that adds some boilerplate code which lightning was supposed to get rid of and will add another library to the whole set-up.

• In an old official Lightning tutorial they use LightningApp and the Sweep class from training studio. The solution looks very generalizable and tidy. But it seems that everything in Lightning-AI Universe (which includes LightningApp and training studio) is discontinued. At least the repo is archived on github. I am assuming because they commercialized all those applications now. I am therefore skeptical how smart it is to build on some old repo that is not maintained anymore.

Would love to hear if anyone has a good solution for that.

[mauvilsa]

Some time ago I created https://github.com/mauvilsa/ray-tune-cli. Though it is only an example and I haven't been able to develop that further. Also note that there is lightray (mentioned in mauvilsa/ray-tune-cli#1 (comment)) which you can pip install, see https://pypi.org/project/lightray/. I haven't used it but would be worth to try.

[HenrikAsmuth]

Thanks, that is basically what I was looking for. For now, my main goal is to do a manually defined sweep, not using an automated tuner. But the building blocks I need to parse the arguments for the sweep are basically the same as used for ray in those examples.