[RLlib] MultiAgentEnv API enhancements (related to defining obs-/action spaces for agents).

[sven1977]

MultiAgentEnv API enhancements (related to defining obs-/action spaces for agents).

The MultiAgentEnv API gets the following face-lift:

• The MultiAgentEnv will have a new get_observation_space(agent_id=...) method, which is the only path that RLlib will use to get an agent's space.
• We remove/deprecate other methods to simplify the MultiAgentEnv class a bit. E.g. observation_space_sample() .
• If users want (b/c they have a lot of agents, some of which share the same spaces), users can now override this method in MultiAgentEnv and always return the correct (single-agent) space for the given agent_id. This relieves one from having to include all agent IDs in the agent-to-space dict. In fact, you don't even have to define such a dict anymore at all.
• Alternatively to overriding get_observation_space(), one can define self.observation_spaces (plural!) to be a dict mapping AgentIDs to individual spaces (<- currently the only supported mechanism).
• Alternatively to overriding get_observation_space(), one can define self.observation_space (singular!) to be a single observation space that applies to all agents in the env (<- "lazy" mechanism for when all agents have the same space anyways).

^ Exact same for action spaces.

On top of that, to make things more explicit and more similar to PettingZoo, we should also require users to provide the two attributes in their MultiAgentEnv

self.agents = [list of agents currently in the episode]
self.possible_agents = [list of all possible agent that can ever appear in any episode]

The method self.get_agent_ids() will be deprecated soon (replaced by self.agents or self.possible_agents).
The private attribute self._agent_ids will be deprecated soon (replaced by self.agents or self.possible_agents).

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simonsays1980]

LGTM. We should test, if checkpointing runs error-free with these changes. I don't see an obvious problem, but let's quickly check this.

[simonsays1980]

Nice!!

[simonsays1980]

This might also help in the sampling from MultiAgentEpisodeBuffers