Add support for MultiDiscrete/MultiBinary observation spaces

[araffin]

• preprocessing.py and policy.py need to be updated (cf Stable-Baselines for the preprocessing)
• tests using envs from identity_env.py

@rolandgvc is working that one

[rolandgvc]

I thought get_obs_shape was supposed to return a tuple, but from the two existent implementations I see it return integers.

def get_obs_shape(observation_space: spaces.Space) -> Tuple[int, ...]:
    """
    Get the shape of the observation (useful for the buffers).

    :param observation_space: (spaces.Space)
    :return: (Tuple[int, ...])
    """
    if isinstance(observation_space, spaces.Box):
        return observation_space.shape
    elif isinstance(observation_space, spaces.Discrete):
        # Observation is an int
        return 1

Should the returns remain integers or be made tuples?

[araffin]

but from the two existent implementations I see it return integers.

so, this method is only used in the buffer, good catch, this should return a tuple :/ (so 1, instead of 1 normally)
I have to check again, otherwise, we will need to change slightly the type hint.

[araffin]

Wait, looking at master, it returns 1,, so it's fine

[rolandgvc]

ok, so no ( , ) tuples, right?

[araffin]

stable-baselines3/stable_baselines3/common/preprocessing.py

Line 84 in 257a40e

return 1,

It is a tuple

[rolandgvc]

My bad :D

[H-Park]

Playing devil's advocate here, but since tuple observation (and action spaces) is on the roadmap, wouldn't that make this feature a redundancy?

Basically, MultiDiscrete([a, b, c, ...]) vs Tuple((Discrete(a), Discrete(b), Discrete(c), ...)).

[Miffyli]

@H-Park Good point. On one hand, it adds one more layer of "modifications" (turn MultiDiscrete into bunch of Discretes), but on the second hand it will reduce if-elseing and manual handling. I say we still implement MultiDiscrete for now (as in original stable-baselines), and once we get to tuple obs/actions we handle this question.

[araffin]

Playing devil's advocate here, but since tuple observation (and action spaces) is on the roadmap, wouldn't that make this feature a redundancy?

So, yes and no. Yes, because what you describe is equivalent to MultiDiscrete space.
No, because supporting MultiDiscrete separately is cleaner and would be anyway a building block, as done in the flatdim() method:

https://github.com/openai/gym/blob/b2727d6fd396f7fe2b19a541ef3a6ea47559cdb3/gym/spaces/utils.py#L24

Also, MultiDiscrete is a very special case when all the sub-spaces are of the same type. Usually, in that case, you don't need Dict or Tuple spaces (e.g. multiples Box spaces can be re-combined into one).