[RLlib] New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule, Algo, and Learner example classes). #47885

sven1977 · 2024-10-03T02:33:57Z

New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule, Algo, and Learner example classes).
These new classes will replace the core/testing/bc_... classes and serve as:

simple example implementations for things like: using a shared encoder in multi-agent.
example for how to implement a custom Algorithm and AlgorithmConfig.
example for yet another custom Learner.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <svenmika1977@gmail.com>

simonsays1980

LGTM. Just a small nit and a deisgn question.

simonsays1980 · 2024-10-03T13:39:37Z

rllib/examples/algorithms/classes/vpg.py

+            )
+
+        # Reduce and return collected metrics.
+        return self.metrics.reduce()


It's so simple now :)

simonsays1980 · 2024-10-03T13:41:47Z

rllib/examples/rl_modules/classes/vpg_rlm.py

+
+    @override(TorchRLModule)
+    def _forward_inference(self, batch, **kwargs):
+        with torch.no_grad():


simonsays1980 · 2024-10-03T13:43:50Z

rllib/examples/rl_modules/classes/vpg_using_shared_encoder_rlm.py

+    def _common_forward(self, batch):
+        # Features can be found in the batch under the "encoder_features" key.
+        features = batch["encoder_features"]
+        logits = self.policy_head(features)


self._policy_head(features)?

simonsays1980 · 2024-10-03T13:52:13Z

rllib/examples/rl_modules/classes/vpg_using_shared_encoder_rlm.py

+        )
+
+    @override(MultiRLModule)
+    def _run_forward_pass(self, forward_fn_name, batch, **kwargs):


This is still a strange form - passing in the forward_fn_name. Why is this needed? To have a single method to be overridden in inheritance?

This will be gone in another PR that's already in (your) review :)

Great question!

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…odule_do_over_bc_default_module_02_pg_algo

… Algo, and Learner example classes). (ray-project#47885) Signed-off-by: Tina Wu <j6vupz97@gmail.com>

… Algo, and Learner example classes). (ray-project#47885) Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>

sven1977 added 3 commits October 3, 2024 04:19

wip

8143aaf

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

3bf0d09

Signed-off-by: sven1977 <svenmika1977@gmail.com>

wip

09bac52

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 requested review from ArturNiederfahrenhorst and simonsays1980 as code owners October 3, 2024 02:33

sven1977 enabled auto-merge (squash) October 3, 2024 02:35

github-actions bot added the go add ONLY when ready to merge, run all tests label Oct 3, 2024

simonsays1980 approved these changes Oct 3, 2024

View reviewed changes

wip

7bab1ea

Signed-off-by: sven1977 <svenmika1977@gmail.com>

github-actions bot disabled auto-merge October 3, 2024 17:39

sven1977 enabled auto-merge (squash) October 3, 2024 17:55

Merge branch 'master' of https://github.com/ray-project/ray into rl_m…

ded31fe

…odule_do_over_bc_default_module_02_pg_algo

github-actions bot disabled auto-merge October 4, 2024 20:55

sven1977 enabled auto-merge (squash) October 4, 2024 22:22

sven1977 merged commit cbde03c into ray-project:master Oct 4, 2024
6 checks passed

sven1977 deleted the rl_module_do_over_bc_default_module_02_pg_algo branch October 4, 2024 23:51

tinaxfwu pushed a commit to tinaxfwu/ray that referenced this pull request Oct 5, 2024

[RLlib] New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule,…

e86f5a9

… Algo, and Learner example classes). (ray-project#47885) Signed-off-by: Tina Wu <j6vupz97@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule, Algo, and Learner example classes). #47885

[RLlib] New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule, Algo, and Learner example classes). #47885

sven1977 commented Oct 3, 2024 •

edited

Loading

simonsays1980 left a comment

simonsays1980 Oct 3, 2024

simonsays1980 Oct 3, 2024

simonsays1980 Oct 3, 2024

sven1977 Oct 3, 2024

simonsays1980 Oct 3, 2024

sven1977 Oct 3, 2024

sven1977 Oct 3, 2024

[RLlib] New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule, Algo, and Learner example classes). #47885

[RLlib] New API stack: (Multi)RLModule overhaul vol 02 (VPG RLModule, Algo, and Learner example classes). #47885

Conversation

sven1977 commented Oct 3, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

simonsays1980 left a comment

Choose a reason for hiding this comment

simonsays1980 Oct 3, 2024

Choose a reason for hiding this comment

simonsays1980 Oct 3, 2024

Choose a reason for hiding this comment

simonsays1980 Oct 3, 2024

Choose a reason for hiding this comment

sven1977 Oct 3, 2024

Choose a reason for hiding this comment

simonsays1980 Oct 3, 2024

Choose a reason for hiding this comment

sven1977 Oct 3, 2024

Choose a reason for hiding this comment

sven1977 Oct 3, 2024

Choose a reason for hiding this comment

sven1977 commented Oct 3, 2024 •

edited

Loading