[RLlib] SAC/DQN activate multi-agent learning tests and small bug fix in MultiAgentEpisode.
#45542
Conversation
sven1977
requested review from
ArturNiederfahrenhorst and
simonsays1980
as code owners
…dqn_multi_agent_debugging
rllib/BUILD
Outdated
| @@ -303,6 +292,15 @@ py_test( | |||
| args = ["--as-test", "--enable-new-api-stack"] | |||
| ) | |||
|
|
|||
| py_test( | |||
There was a problem hiding this comment.
Yey! And we are there! :)
rllib/BUILD
Outdated
| @@ -452,6 +450,15 @@ py_test( | |||
| args = ["--as-test", "--enable-new-api-stack"] | |||
| ) | |||
|
|
|||
| py_test( | |||
There was a problem hiding this comment.
Next step: synchronized sampling.
rllib/algorithms/algorithm.py
Outdated
| EPISODE_RETURN_MIN, | ||
| EPISODE_RETURN_MAX, | ||
| ]: | ||
| if must_have not in results[ENV_RUNNER_RESULTS]: |
There was a problem hiding this comment.
Yup, understood. But this means, if a user collects custom metrics via callback and uses Tune on this ... an error could result due to this metrics not being available in an iteration. We should document this somewhere - maybe at the MetricsLogger API ...
There was a problem hiding this comment.
Correct, that's why I put the comment there. We should actually fix Tune, there is no other solution to this problem, imo.
Tune didn't probably think of this b/c in SL, you don't have "strangely behaving" episodes that don't deliver data sometimes.
rllib/env/multi_agent_episode.py
Outdated
| @@ -873,7 +873,12 @@ def concat_episode(self, other: "MultiAgentEpisode") -> None: | |||
| ) | |||
|
|
|||
| # Concatenate the env- to agent-timestep mappings. | |||
| self.env_t_to_agent_t[agent_id].extend(other.env_t_to_agent_t[agent_id]) | |||
| j = self.env_t | |||
| for i, val in enumerate(other.env_t_to_agent_t[agent_id][1:]): | |||
There was a problem hiding this comment.
This is one of the nasty parts of the MAE. Awesome catch!
| .rl_module( | ||
| # Settings identical to old stack. | ||
| model_config_dict={ | ||
| "fcnet_hiddens": [256], | ||
| "fcnet_activation": "relu", | ||
| "fcnet_activation": "tanh", |
| model_config_dict={ | ||
| "fcnet_hiddens": [256, 256], | ||
| "fcnet_activation": "relu", | ||
| # "post_fcnet_hiddens": [], |
There was a problem hiding this comment.
This part is old stack equivalent.
| @@ -131,7 +131,7 @@ def add( | |||
| """ | |||
| episodes: List["MultiAgentEpisode"] = force_list(episodes) | |||
|
|
|||
| new_episode_ids: List[str] = {eps.id_ for eps in episodes} | |||
| new_episode_ids: Set[str] = {eps.id_ for eps in episodes} | |||
| # TODO (sven, simon): Is there always a mapping? What if not? | ||
| # Is then module_id == agent_id? | ||
| module_id = ma_episode._agent_to_module_mapping[agent_id] | ||
| module_id = ma_episode.module_for(agent_id) |
There was a problem hiding this comment.
Sweet, this method is just clean!
Signed-off-by: sven1977 <svenmika1977@gmail.com>
…dqn_multi_agent_debugging
…dqn_multi_agent_debugging
…ti_agent_debugging
…dqn_multi_agent_debugging Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # rllib/utils/metrics/stats.py
…dqn_multi_agent_debugging Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # rllib/BUILD # rllib/algorithms/sac/torch/sac_torch_learner.py # rllib/tuned_examples/sac/multi_agent_pendulum_sac.py # rllib/utils/metrics/stats.py
…dqn_multi_agent_debugging
…dqn_multi_agent_debugging
MultiAgentEpisode.concat(): self.env_t_to_agent_t is not properly built in resulting episode object.Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.