Update Gymnasium to v1.0.0

[pseudo-rnd-thoughts]

This PR updates SB3 to Gymnasium v1.0, read the release-notes to see all the changes.

Motivation and Context

Gymnasium is the core API used in SB3, therefore would be helpful for both SB3 to use the latest version and that SB3 provides a great testing ground to check for that the Gymnasium release works as intended.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have opened an associated PR on the SB3-Contrib repository (if necessary)
• I have opened an associated PR on the RL-Zoo3 repository (if necessary)
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)
• I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[pseudo-rnd-thoughts]

There are only two main issues to resolve + need to rewrite VecVideoRecorder

1. As we have removed Env.__getattr__, then we need to update to use Env.get_wrapper_attr
2. Registration of external environment is not automatical, i.e., atari, therefore a hack was added to fix it

[pseudo-rnd-thoughts]

The CI seems to have failed due to reasons unrelated to the version change

[araffin]

Thanks for the PR =)

As we have removed Env.getattr, then we need to update to use Env.get_wrapper_attr

if possible (and if not too hacky), I would add backward compat changes to handle both gymnasium 0.29 and 1.x.

[pseudo-rnd-thoughts]

Thanks for the PR =)

No worries, all the errors seem expected and no unexpected bugs are found

As we have removed Env.getattr, then we need to update to use Env.get_wrapper_attr

if possible (and if not too hacky), I would add backward compat changes to handle both gymnasium 0.29 and 1.x.

I believe the changes made should be backward compatible. Just updating VecRecordEnv needs to be fully updated / rewritten

[Kallinteris-Andreas]

Assuming the Environment which used gymnasium==0.29 is not broken from the gymansium==1.0 update, it should just work without any additional compatibility work in SB3

[araffin]

Assuming the Environment which used gymnasium==0.29 is not broken from the gymansium==1.0 update, it should just work without any additional compatibility work in SB3

I'm talking about allowing people to use 0.29 with SB3.

I believe the changes made should be backward compatible.

mmh, I would double check the getattr() part, I remember it was warning the user

[pseudo-rnd-thoughts]

mmh, I would double check the getattr() part, I remember it was warning the user

If the code works with 1.0.0a1 then it will work with 0.29 but possibly not the other way around

[pseudo-rnd-thoughts]

@araffin I believe I have fixed all the issues except for updating VecVideoRecorder.
I don't know how SB3 works internals, would you be able to get one of your devs to update that?

[araffin]

I don't know how SB3 works internals, would you be able to get one of your devs to update that?

Currently, there is only one active dev (me...), Quentin (@qgallouedec ) is helping me with answering questions and doing code reviews, for the rest, we have to rely on the community.

In the meantime, you could try running tests in SB3 contrib and RL Zoo with this branch, that should unveil other bugs/issues.

[Kallinteris-Andreas]

i tested this: (from https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#record-a-video)

import gymnasium as gym
from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv

env_id = "CartPole-v1"
video_folder = "logs/videos/"
video_length = 100

vec_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")])

obs = vec_env.reset()

# Record the video starting at the first step
vec_env = VecVideoRecorder(vec_env, video_folder,
                       record_video_trigger=lambda x: x == 0, video_length=video_length,
                       name_prefix=f"random-agent-{env_id}")

vec_env.reset()
for _ in range(video_length + 1):
  action = [vec_env.action_space.sample()]
  obs, _, _, _ = vec_env.step(action)
# Save the video
vec_env.close()

and I get this error

python test.py
Traceback (most recent call last):
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/test.py", line 17, in <module>
    vec_env.reset()
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/vec_video_recorder.py", line 66, in reset
    self.start_video_recorder()
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/vec_video_recorder.py", line 78, in start_video_recorder
    self.video_recorder.capture_frame()
    ^^^^^^^^^^^^^^^^^^^
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/base_vec_env.py", line 420, in __getattr__
    return self.getattr_recursive(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/base_vec_env.py", line 445, in getattr_recursive
    attr = getattr(self.venv, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DummyVecEnv' object has no attribute 'video_recorder'

i am using gymnasium==1.0.0a1, sb3 this PR's latest commit (in march)

[pseudo-rnd-thoughts]

Just updating VecRecordEnv needs to be fully updated / rewritten

@Kallinteris-Andreas Yes, this is the only part of the PR that still needs to be done.
I'm tempted to just copy and paste the old video recorder in as a solution. Might try to do this evening

[qgallouedec]

Feel free to ping me if necessary

[pseudo-rnd-thoughts]

@araffin I've updated to alpha 2, however, this removed Dict space using OrderedDict for samples as python 3.7+ dicts are ordered, therefore, there is no need for OrderedDict.
However, this causes mypy issues, do you understand the problem better?

[Kallinteris-Andreas]

gymnasium==1.0.0 is release

@araffin is there anything blocking a new SB3 release

[richardjozsa]

Just a question, with the new updates in Gymnasium v1.0.0, I can see a change how the environments resetting.
Correct me if I'm wrong, it is just assumptions.

Doesn't that change should effect how we step and store information from the envs? I mean this part:
in offpolicy for example

#As the VecEnv resets automatically, new_obs is already the
        # first observation of the next episode
        for i, done in enumerate(dones):
            if done and infos[i].get("terminal_observation") is not None:
                if isinstance(next_obs, dict):
                    next_obs_ = infos[i]["terminal_observation"]
                    # VecNormalize normalizes the terminal observation
                    if self._vec_normalize_env is not None:
                        next_obs_ = self._vec_normalize_env.unnormalize_obs(next_obs_)
                    # Replace next obs for the correct envs
                    for key in next_obs.keys():
                        next_obs[key][i] = next_obs_[key]
                else:
                    next_obs[i] = infos[i]["terminal_observation"]
                    # VecNormalize normalizes the terminal observation
                    if self._vec_normalize_env is not None:
                        next_obs[i] = self._vec_normalize_env.unnormalize_obs(next_obs[i, :])

        replay_buffer.add(
            self._last_original_obs,  # type: ignore[arg-type]
            next_obs,  # type: ignore[arg-type]
            buffer_action,
            reward_,
            dones,
            infos,
        )

On this page it says that it will be handled by autoreset, I think it has to be handled in the code above.

[pseudo-rnd-thoughts]

If I understand correctly, SB3 has its only vector environments, VecEnv so don't use the gymnasium vector environment so is unaffected by this change

[araffin]

is there anything blocking a new SB3 release

There are different things blocking:

• time and priority from my side
• testing with SB3 contrib and RL Zoo
• backward compat and tests with 0.28.1
• if possible, remove the copy-pasted VideoRecorder, or at least bring it to SB3 standards in term of documentation/typing/tests

[Kallinteris-Andreas]

1. I will (try to test) backcompat for gymnasium==0.29.1
2. Is there a reason you need the VideoRecorder to be inside SB3, can't you just use the one directly from gymnasium
3. unfortunately my time machine no longer works, so I can not give you more time

[araffin]

Is there a reason you need the VideoRecorder to be inside SB3, can't you just use the one directly from gymnasium

we don't: #1837 (comment)

[pseudo-rnd-thoughts]

Is there a reason you need the VideoRecorder to be inside SB3, can't you just use the one directly from gymnasium

SB3 doesn't use Gymnasium's VectorEnv and has their own VecEnv and so have their own wrappers like VecVideoRecorder. This is largely a wrapper about the Gymnasium old monitor code however as that was removed in v1.0, then either SB3 needs the original code, a stripped down version or a complete rewrite of the wrapper.