Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[rllib] Auto clip actions to Box space range; deprecate squash_to_range #3426

Merged
merged 8 commits into from
Dec 4, 2018
Merged

[rllib] Auto clip actions to Box space range; deprecate squash_to_range #3426

merged 8 commits into from
Dec 4, 2018

Conversation

ericl
Copy link
Contributor

@ericl ericl commented Nov 29, 2018

What do these changes do?

Per discussion here https://groups.google.com/forum/#!topic/ray-dev/loJOOU5LzDk, squash_to_range incorrectly zeros out gradients and can cause divergence.

@ericl ericl mentioned this pull request Nov 29, 2018
Closed
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9653/
Test FAILed.

@ericl
Copy link
Contributor Author

ericl commented Nov 29, 2018

jenkins retest this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9659/
Test FAILed.

@ericl
Copy link
Contributor Author

ericl commented Nov 29, 2018

jenkins retest this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9668/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9685/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9686/
Test FAILed.

richardliaw
richardliaw reviewed Dec 2, 2018
View reviewed changes
python/ray/rllib/agents/agent.py
@@ -61,6 +61,8 @@
# Whether to clip rewards prior to experience postprocessing. Setting to
# None means clip for Atari only.
"clip_rewards": None,
# Whether to np.clip() actions to the action space low/high range spec.
"clip_actions": True,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when would you want this to be false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For backwards compatibility I think it's a reasonable option to have.

python/ray/rllib/evaluation/sampler.py Outdated Show resolved Hide resolved
python/ray/rllib/evaluation/sampler.py Outdated Show resolved Hide resolved
@richardliaw richardliaw changed the title Auto clip actions to Box space range; deprecate squash_to_range [rllib] Auto clip actions to Box space range; deprecate squash_to_range Dec 2, 2018
@richardliaw @ericl
Apply suggestions from code review
3be7545 
Co-Authored-By: ericl <ekhliang@gmail.com>
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9696/
Test FAILed.

@ericl ericl added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Dec 3, 2018
@ericl ericl merged commit d820597 into ray-project:master Dec 4, 2018
@ericl ericl mentioned this pull request Jan 18, 2019
Closed
@saudet saudet mentioned this pull request Aug 5, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardliaw richardliaw richardliaw approved these changes

Assignees

@richardliaw richardliaw

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ericl @AmplabJenkins @richardliaw