Adds multiple env support to the SB3 wrapper

[edbeeching]

Adds an option to interact with parallel Godot executables using the n_parallel argument. This only works in on an export env.

Example usage with 4 parallel envs:

env = StableBaselinesGodotEnv(env_path=args.env_path, n_parallel=4)

[visuallization]

Uuuuh nice, this looks really good. Also nice small improvements regarding typings, documentation and warnings. I will test this straight away and will get back to you.

[Ivan-267]

Looks great and worked properly in my quick test.

Regarding the last commit:

godot_rl_agents/examples/stable_baselines3_example.py

Line 28 in 9c25e76

parser.add_argument("--n_parallel", default=1, type=int, help="whether to speed up the physics in the env")

A small note is that the help tip can be updated as well, e.g. "How many instances of the environment executable to launch - requires --env_path to be set if > 1.".

I'm not sure whether this should be added, it's just here as a note, potentially setting a unique seed for each environment could encourage exploration:

from:

godot_rl_agents/godot_rl/wrappers/stable_baselines_wrapper.py

Line 17 in 9c25e76

self.envs = [GodotEnv(env_path=env_path, convert_action_space=True, port=port+p, **kwargs) for p in range(n_parallel)]

to e.g. (untested):

        self.envs = [GodotEnv(env_path=env_path, convert_action_space=True, port=port+p, seed=p, **kwargs) for p in range(n_parallel)]

[edbeeching]

@Ivan-267 , good point about the seed and help description. Thanks.
I also added a check in the StableBaselinesGodotEnv init method to ensure env_path is not None when n_parallel>1

[araffin]

the truncated should be stored in info, and info should include terminal obs too, see https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/vec_env/dummy_vec_env.py#L61-L70

[edbeeching]

@araffin Thanks for this info. I have a question.

On the Godot side, envs are automatically reset when an episode terminates / truncates, I am not sure how feasible it is to send the terminal obs in the truncated setting without refactoring all the envs. Does the sb3 PPO implementation use the terminal obs for a value preduction in the truncated setting? If it is not possible to refactor in the short term, would you recommend just setting the terminated flag? (with trunc = False for all steps)

[araffin]

Does the sb3 PPO implementation use the terminal obs for a value preduction

yes https://github.com/DLR-RM/stable-baselines3/blob/5abd50a853e0f667f48ae10769f478c4972eda35/stable_baselines3/common/on_policy_algorithm.py#L194-L205

If it is not possible to refactor in the short term, would you recommend just setting the terminated flag?

you can use done = truncated or terminated (and not set the terminal obs), although it might impact performance for infinite-horizon tasks.
(as a quick check, you can use this wrapper: https://sb3-contrib.readthedocs.io/en/master/common/wrappers.html#timefeaturewrapper and see if there is any performance diff)