prune Tune arguments from Trainer init - LearnRate & BatchSize as callback

[Borda]

🚀 Feature

Move the auto_scale_batch_size and auto_lr_find to the .tune method

Motivation

it is quite confusing setting some auto finds which needs to be called anyway with .tune() which uses another set of arguments.

Pitch

As a naive user, I would expect that the tuning is executed at the begging of .fit()

Alternatives

Also in such cases, the user is not much under control in which order they are called, for example, my use case is:

1. find the largest batch size
2. do something extra around the model
3. find the best LR

ad this moment I would need to create two Traner instances :(

I am also experiencing some OOM memory failer when I run Batch size and LR, have not the concrete case for debugging yet

Additional context

The only justification would be usage with Lightning CLI

[ananthsub]

I see 3 options:

• keep as is
• move constructor arguments to specific method they apply to (applies to tune, fit, validate, test, predict as well)
• create separate objects for users to interact with: Trainer, Evaluator, Predictor, Tester

the last one is the most extreme, but i wanted to mention for completeness. between the first two, i prefer the second one. there's a similar conversation here: #9006

[SkafteNicki]

Additional option:
Move lr finder and batch scaler to callbacks, removing the tune method
auto_scale_batch_size and auto_lr_find could still exist, setting them to True would just add the corresponding callback.

[Borda]

Move lr finder and batch scaler to callbacks, removing the tune method

I like this one and it may be the cleanest way as we don't add any other logic pattern/layer

[ananthsub]

@SkafteNicki @Borda - as a callback, this can be pretty invasive in setting data inside of the trainer. Is that a precedent we should be setting? In what callback hooks would these run? https://github.com/PyTorchLightning/pytorch-lightning/blob/69f66fd6bb361a7932a82291e4ef001f4f381f99/pytorch_lightning/tuner/batch_size_scaling.py#L115-L125

[Borda]

I assume that all would run in the before fit hook...

[tchaton]

IMO, I trust each entrypoint to the trainer to be independent as it makes sure features are fully encapsultated and reduce risks from bugs.

move constructor arguments to specific method they apply to (applies to tune, fit, validate, test, predict as well)

I think this is fine for tune which is pretty particular, but It don't think we should do this for other functions.

Best,
T.C

[SkafteNicki]

I am also fine with moving the arguments to the tune method.
Tuning is the odd one out of tune, fit, validate, test, predict so I think we can do it without having to move other arguments.

[Borda]

I model practical use-case is combination with FineTune which would unfreeze backbone on 10th epoch
so on:

• 0.epoch - freeze backbone
• 0.epoch - run BatchSize -> 45
• 0.epoch - run LRFinder -> 0.005
• 9.epoch - unfreeze backbone
• 9.epoch - run BatchSize -> 14
• 9.epoch - run LRFinder -> 0.001

this gives more freedom to combine these methods during the training cycle
so the callback will be executed on epoch start...