[RLlib] Add option to use torch.lr_scheduler classes for learning rate schedules.

[simonsays1980]

Why are these changes needed?

In many state-of-the-art papers complex learning rate schedules are used (for example cosinus learning rate schedules). Most learning rate schedules are implemented in the torch.lr_scheduler module, but cannot be used in RLlib's algorithms, yet. This PR proposes a way to add a list of (module-specific) learning rate schedulers for each optimizer. This empowers users to use all kind of learning rate schedules and simplifies reproducing or verifying results from state-of-the-art papers as well as supplementing industrial appications with appropriate learning rate schedules.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

As this is experimental for now (and similar to the grad scaler class option), let's merge it as-is. We do have to revisit the Learner API and figure out how to better allow for these more complex Learner customizations:

• Using a custom optimizer (non-Adam).
• Using a custom lr scheduler.
• Using a second/third/... optimizer (per RLModule).
• Using custom grad scaling behavior.

One important learning from the old API stack was that we shouldn't build in all kinds of loose features like this, b/c users don't really benefit from these in that they still don't understand what's going on under the hood and how RLlib translates these extra, convenience options into Learner logic. In the long run, I do think it's better to improve the root Learner API instead, then have example scripts that show how these customizations can be done (w/o us having to provide all these extra config options and built-in logics). But this is a mid-term TODO and cleanup request ...