Issues with rllib_training Example

[Accommodus]

I am currently working on integrating the Ray RLlib PPO algorithm into the Basilisk environment as outlined in the rllib_training.html example. While I'm able to initiate the environment and have it sample random actions, I encounter errors when attempting to train.

1. When building the default configuration provided by the example, I receive the following error:

   ValueError: Argument `learning_rate` should be float, or an instance of LearningRateSchedule, or a callable (that takes in the current iteration value and returns the corresponding learning rate value). Received instead: learning_rate=<tf.Variable 'default_policy_wk5/lr:0' shape=() dtype=float32, numpy=3e-05>
   

2. After attempting to use a LearningRateSchedule, I encounter this error:

   ValueError: The initializer passed is not valid. It should be a callable with no arguments and the shape should not be provided or an instance of `tf.keras.initializers.*` and `shape` should be fully defined.
   

Could anyone provide a working example that properly sets up the training without these errors? Any help would be greatly appreciated.
Thank you!

[Mark2000]

Ran into this recently, a combination of a keras and rllib update are incompatible with each other. I recommend the solution here of dropping your tf install by one version: ://github.com/ray-project/ray/issues/45050

[Mark2000]

Also, you’ll probably run into this issue: ray-project/ray#45147

At a minimum, you’ll probably need to make those changes locally, but if you bump the issue there it may encourage the rllib maintainers to merge that fix into a release :)