Gymnasium integration

[smelfungus]

Hello! I am excited to share my initial contribution towards integrating the Gymnasium (formerly OpenAI Gym) toolkit with SharpNeat. While the current solution is still in its early stages and requires further refinement, I believe it serves as a foundation for discussion and future development.

Integration overview:

• The Gymnasium task will be managed by a single set of classes: GymnasiumEvaluator, GymnasiumEvaluationScheme, and GymnasiumExperimentFactory. These classes aim to create a generic Gymnasium "bridging" task that accommodates the various environments available in the toolkit.
• Gymnasium's UI is managed internally. For now, GymnasiumControl simply decorates GenomeControl and allows for OnGenomeUpdated propagation. In future iterations, it would be beneficial to decouple the task UI from Windows/.NET (this also applies to GymnasiumExperimentUi and GymnasiumExperimentUiFactory).
• GymnasiumEpisode is responsible for bridging, evaluating, and communicating with the Gymnasium environment. It uses named pipes for inter-process communication; however, alternatives for better cross-platform support and performance, such as memory-mapped files, should be considered. The script assumes the correct Python version is available in the system path, but future iterations could explore configurability and virtual environments. Communication involves two primary message types: action and observation+reward+termination. Both continuous and discrete action spaces should be supported, although additional work is necessary. GymnasiumEpisode also contains some stateless utility methods, which hopefully will be migrated to separate packages in the next iterations.
• GymnasiumEvaluationScheme and GymnasiumEvaluator are relatively straightforward, but they require additional parameterization to accommodate various inputs, outputs, rewards, and stopping conditions.
• Lastly, main.py is a Python script responsible for the counterpart of communication and Gymnasium environment control.

I look forward to your feedback and suggestions! Thank you for your time.

First evolution attempts:

BipedalWalker-v3.mp4

ref
https://gymnasium.farama.org/
https://www.gymlibrary.dev/
https://gymnasium.farama.org/environments/box2d/bipedal_walker/

[colgreen]

Hi. Thanks for the PR. I was just taking a look to try and understand the various parts and dependencies.

The python/python dependency is easy to provide/configure by simply installing the latest python for Windows, and clicking the option to add python to the PATH environment variable.

main.py seems to listen for commands/actions from the C#/.NET code, and GymnasiumEpisode.cs spawns/starts instances of main/py, and connects to the named pipe.

I think/hope I understood all of that correctly.

To get Gymnasium I have tried pip install "gymnasium[box2d]", but box2d seems to have another dependency:

distutils.errors.DistutilsPlatformError: Unable to find vcvarsall.bat

Full output

pip install "gymnasium[box2d]"
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: gymnasium[box2d] in c:\users\colin\appdata\roaming\python\python311\site-packages (0.28.1)
Requirement already satisfied: numpy>=1.21.0 in c:\users\colin\appdata\roaming\python\python311\site-packages (from gymnasium[box2d]) (1.24.3)
Requirement already satisfied: jax-jumpy>=1.0.0 in c:\users\colin\appdata\roaming\python\python311\site-packages (from gymnasium[box2d]) (1.0.0)
Requirement already satisfied: cloudpickle>=1.2.0 in c:\users\colin\appdata\roaming\python\python311\site-packages (from gymnasium[box2d]) (2.2.1)
Requirement already satisfied: typing-extensions>=4.3.0 in c:\users\colin\appdata\roaming\python\python311\site-packages (from gymnasium[box2d]) (4.5.0)
Requirement already satisfied: farama-notifications>=0.0.1 in c:\users\colin\appdata\roaming\python\python311\site-packages (from gymnasium[box2d]) (0.0.4)
Collecting box2d-py==2.3.5
  Using cached box2d-py-2.3.5.tar.gz (374 kB)
  Preparing metadata (setup.py) ... done
Collecting pygame==2.1.3
  Using cached pygame-2.1.3-cp311-cp311-win_amd64.whl (10.4 MB)
Requirement already satisfied: swig==4.* in c:\users\colin\appdata\roaming\python\python311\site-packages (from gymnasium[box2d]) (4.1.1)
Installing collected packages: box2d-py, pygame
  DEPRECATION: box2d-py is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for box2d-py ... error
  error: subprocess-exited-with-error

  × Running setup.py install for box2d-py did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Using setuptools (version 65.5.0).
      running install
      C:\Program Files\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build\lib.win32-cpython-311
      creating build\lib.win32-cpython-311\Box2D
      copying library\Box2D\Box2D.py -> build\lib.win32-cpython-311\Box2D
      copying library\Box2D\__init__.py -> build\lib.win32-cpython-311\Box2D
      creating build\lib.win32-cpython-311\Box2D\b2
      copying library\Box2D\b2\__init__.py -> build\lib.win32-cpython-311\Box2D\b2
      running build_ext
      building 'Box2D._Box2D' extension
      swigging Box2D\Box2D.i to Box2D\Box2D_wrap.cpp
      swig.exe -python -c++ -IBox2D -small -O -includeall -ignoremissing -w201 -globals b2Globals -outdir library\Box2D -keyword -w511 -D_SWIG_KWARGS -o Box2D\Box2D_wrap.cpp Box2D\Box2D.i
      error: command 'swig.exe' failed: None
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> box2d-py

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: python.exe -m pip install --upgrade pip

[colgreen]

We'll need step by step instructions in a readme.md to help people get these dependencies in place, otherwise each person trying to use this will have to work through these dependencies and issues each time.

[colgreen]

Update:

I followed the pip upgrade instructions, like so:

python.exe -m pip install --upgrade pip

Running pip install "gymnasium[box2d]" again progresses further, but fails on a missing dependency on swig.exe.

I downloaded swig 4.1.1 from https://sourceforge.net/projects/swig/, unzipped, and added swig.exe to the Windows PATH environment variable. pip install "gymnasium[box2d]" now succeeds, but main.py fails...

python main.py
Traceback (most recent call last):
  File "D:\tmp-sharpneat\sharpneat\src\SharpNeat.Windows.App\gymnasium\main.py", line 9, in <module>
    import win32pipe
ModuleNotFoundError: No module named 'win32pipe'

So I did pip install win32pipe, and... success!

[smelfungus]

Thank you for taking a look, @colgreen!
I apologize for not including detailed setup steps earlier. Sure, I'll make several more iterations to document and streamline the setup, make experiment configuration more flexible, include required dependencies etc.

In addition, I'm currently working on a distributed evaluation implementation that will allow us to evaluate computationally-intensive tasks across multiple machines. With a single SharpNeat server handling the genetic algorithm component, and multiple node-executors taking care of receiving, decoding, evaluating, and sending results back to the server, we'll be able to tackle even more complex challenges. I'm currently testing this setup with the Bipedal Walker Hardcore task on four 60-vCPU machines, fingers crossed for success! The tech stack includes gRPC and AspNetCore.

I've also been working on improving the named pipes integration and have added support for the BSD socket Python component. This means we'll be able to evaluate our project on a wider variety of operating systems and reduce our reliance on Windows named pipes.

I'm eager to share the results, iterations, and updates with you as soon as they're ready.