Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Quick-fix for default RLModules in combination with a user-provided config-sub-dict (instead of a full DefaultModelConfig). #47965

Merged
merged 4 commits into from
Oct 10, 2024
Merged

[RLlib] Quick-fix for default RLModules in combination with a user-provided config-sub-dict (instead of a full DefaultModelConfig). #47965

merged 4 commits into from
Oct 10, 2024

Conversation

sven1977
Copy link
Contributor

@sven1977 sven1977 commented Oct 10, 2024 •
edited
Loading

Quick-fix for default RLModules in combination with a user-provided config-sub-dict (instead of a full DefaultModelConfig).

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@sven1977
wip
2d9117c 
Signed-off-by: sven1977 <svenmika1977@gmail.com>
simonsays1980
simonsays1980 approved these changes Oct 10, 2024
View reviewed changes
Copy link
Collaborator

@simonsays1980 simonsays1980 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Some general questions and maybe we need to add this also to BCRLModule

rllib/algorithms/dqn/dqn_rainbow_rl_module.py Outdated
# TODO (sven): Temporary fix (until we have figured out the final config
# architecture for default models). If self.model_config is a dict (which
# it should always be, make sure to merge it with the default config).
self.model_config = dataclasses.asdict(DefaultModelConfig()) | self.model_config
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that the user needs to do this everytime she defines a new module? We do here actually the same we did before with the AlgorithmConfig.model_config property.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that the user needs to do this everytime she defines a new module? We do here actually the same we did before with the AlgorithmConfig.model_config property

Nope, this is only(!) needed for RLlib's default models. See my other answer here. Users are completely free to request any keys they need in their custom model classes and then - of course - have to provide these keys in the model_config dict they pass into the RLModuleSpec. Users can also decide to use default values inside their RLModule classes for certain keys.

rllib/algorithms/sac/sac_rl_module.py Outdated
# TODO (sven): Temporary fix (until we have figured out the final config
# architecture for default models). If self.model_config is a dict (which
# it should always be, make sure to merge it with the default config).
self.model_config = dataclasses.asdict(DefaultModelConfig()) | self.model_config
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we need to add this also to the BCRLModule then.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also: We might need to add this to all the examples that are not derived from the algorithm modules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! Done ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the non-default models, this is fine, b/c those are 100% user-driven and thus don't use the catalogs. The user decides, which keys are accessed in the model_config and only those need to be provided (or even have default values inside the custom RLModule code).

This is really only a problem for RLlib's default models.

@sven1977
wip
58e71fb 
Signed-off-by: sven1977 <svenmika1977@gmail.com>
@sven1977 sven1977 enabled auto-merge (squash) October 10, 2024 10:30
@sven1977 sven1977 added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Oct 10, 2024
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Oct 10, 2024
@sven1977 sven1977 added rllib RLlib related issues rllib-models An issue related to RLlib (default or custom) Models. rllib-newstack labels Oct 10, 2024
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into quic…
b62c697 
…k_fix_default_model_config_problem
@sven1977
fix
726b175 
Signed-off-by: sven1977 <svenmika1977@gmail.com>
@sven1977 sven1977 enabled auto-merge (squash) October 10, 2024 13:29
@sven1977 sven1977 merged commit 59af152 into ray-project:master Oct 10, 2024
6 checks passed
@sven1977 sven1977 deleted the quick_fix_default_model_config_problem branch October 10, 2024 16:13
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
7f7e893 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
503d445 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
e62052a 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
7d2dbce 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
55950af 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
75399e4 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
8958582 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
d04f8d3 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@sven1977 @ujjawal-khare
[RLlib] Quick-fix for default RLModules in combination with a user-pr…
611b645 
…ovided config-sub-dict (instead of a full `DefaultModelConfig`). (ray-project#47965)

Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonsays1980 simonsays1980 simonsays1980 approved these changes

Assignees

@simonsays1980 simonsays1980

Labels
go add ONLY when ready to merge, run all tests rllib RLlib related issues rllib-models An issue related to RLlib (default or custom) Models. rllib-newstack tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sven1977 @simonsays1980