[RLlib] Action masking example for new API stack.

[simonsays1980]

Why are these changes needed?

This PR adds an example for using action masking in the new API stack to the repository. In addition it makes a small change to the SingleAgentEnvRunner to deal with Dict observation spaces.

Related issue number

Closes #44780 #44452

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[sven1977]

Super cool PR! One more example script TODO down.

We need to add this example to BUILD as well.

[sven1977]

Strange: Shouldn't this already have caused a bug in the existing flatten obs example (which also uses a dict obs space)?

[simonsays1980]

I think, the flatten obs example silently snuck around it: The unbatch(obs) was only forgotten in _sample_episodes and not in _sample_timesteps (has cost me a lot of time to figure it out thoug). Because the flatten obs does no evaluation - it never ran into it :D

[sven1977]

Ahhh, yes, makes perfect sense. Thanks for catching this!

[lonsdale8734]

With ray==2.24.0, batch_mode="complete_episodes" will cause the bug. And obs = unbatch(obs) should outter the for-loop like what _sample_timesteps do:

        obs, infos = self.env.reset()
        obs = unbatch(obs)
        for env_index in range(self.num_envs):
            episodes[env_index].add_env_reset(
                observation=obs[env_index],

[sven1977]

Thanks for the additional fixes and answers @simonsays1980 . Very nice PR!

[aslonnie]

approval for CI files.