[RLlib; docs] Docs do-over (new API stack): Env pages vol 02. #48542
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…_redo_cleanup_old_api_stack_01
…_redo_cleanup_old_api_stack_01
…_redo_cleanup_old_api_stack_01 Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # doc/source/rllib/images/rllib-envs.svg
sven1977
requested review from
maxpumperla,
simonsays1980 and
a team
as code owners
sven1977
added
rllib
…_redo_cleanup_old_api_stack_01
…_redo_cleanup_old_api_stack_01
sven1977
changed the title
angelinalg
reviewed
Dec 12, 2024
doc/source/rllib/external-envs.rst
Outdated
| In many situations, it does not make sense for an RL environment to be "stepped" by RLlib. | ||
| For example, if we train one or more policies inside a complex simulator, for example, a game engine | ||
| or a robotics simulation, it would be more natural and user friendly to flip this setup around | ||
| and - instead of RLlib "stepping" the env - allow the simulations and the agents to fully control |
There was a problem hiding this comment.
Suggested change
| and - instead of RLlib "stepping" the env - allow the simulations and the agents to fully control | |
| and - instead of RLlib "stepping" the env - allow the simulations and the agents to fully control |
| "traffic light" agents interacting simultaneously, whereas in a board game, | ||
| two or more agents may act in a turn-based sequence. | ||
|
|
||
| Several different policy networks may be used to control the various agents. |
There was a problem hiding this comment.
Suggested change
| Several different policy networks may be used to control the various agents. | |
| You can use several different policy networks to control the various agents. |
|
I only got as far as doc/source/rllib/multi-agent-envs.rst but I will pick up again tomorrow. |
angelinalg
approved these changes
Dec 14, 2024
There was a problem hiding this comment.
Good work on all this writing. If this is a good time to change titles to sentence case, that would be more consistent with our style guide. Otherwise, don't worry about it. It's definitely not a blocker. Hope the suggestions are helpful. Will approve to not block you. Sorry for the delay.
| two or more agents may act in a turn-based sequence. | ||
|
|
||
| Several different policy networks may be used to control the various agents. | ||
| Thereby, each of the agents in the environment maps to exactly one particular policy. This mapping is |
There was a problem hiding this comment.
Suggested change
| Thereby, each of the agents in the environment maps to exactly one particular policy. This mapping is | |
| Each agent in the environment maps to exactly one particular policy. Define this mapping |
|
|
||
| Several different policy networks may be used to control the various agents. | ||
| Thereby, each of the agents in the environment maps to exactly one particular policy. This mapping is | ||
| determined by a user-provided function, called the "mapping function". Note that if there |
There was a problem hiding this comment.
Suggested change
| determined by a user-provided function, called the "mapping function". Note that if there | |
| with a user-provided function, called the "mapping function". Note that if there |
| Several different policy networks may be used to control the various agents. | ||
| Thereby, each of the agents in the environment maps to exactly one particular policy. This mapping is | ||
| determined by a user-provided function, called the "mapping function". Note that if there | ||
| are ``N`` agents mapping to ``M`` policies, ``N`` is always larger or equal to ``M``, |
There was a problem hiding this comment.
Suggested change
| are ``N`` agents mapping to ``M`` policies, ``N`` is always larger or equal to ``M``, | |
| are ``N`` agents mapping to ``M`` policies, ``N`` must be equal or greater to ``M``, |
There was a problem hiding this comment.
Not sure if the rewrite is too strong, but I'm guessing you're trying to say that.
| :width: 600 | ||
|
|
||
| **Multi-agent setup:** ``N`` agents live in the environment and take actions computed by ``M`` policy networks. | ||
| The mapping from agent to policy is flexible and determined by a user-provided mapping function. Here, `agent_1` |
There was a problem hiding this comment.
Suggested change
| The mapping from agent to policy is flexible and determined by a user-provided mapping function. Here, `agent_1` | |
| The mapping from agent to policy is flexible and determined by a user-provided mapping function. In this diagram, `agent_1` |
|
|
||
| .. hint:: | ||
|
|
||
| This paragraph describes RLlib's own :py:class`~ray.rllib.env.multi_agent_env.MultiAgentEnv` API, which is the |
There was a problem hiding this comment.
Suggested change
| This paragraph describes RLlib's own :py:class`~ray.rllib.env.multi_agent_env.MultiAgentEnv` API, which is the | |
| This paragraph describes RLlib's :py:class`~ray.rllib.env.multi_agent_env.MultiAgentEnv` API, which is the |
There was a problem hiding this comment.
Just a suggestion to remove a second instance of "own".
doc/source/rllib/rllib-env.rst
Outdated
| .. seealso:: | ||
| 1. **Vectorization within a single process:** Many environments achieve high | ||
| frame rates per core but are limited by policy inference latency. To address | ||
| this, create multiple environments per process and thus batch the policy forward pass |
There was a problem hiding this comment.
Suggested change
| this, create multiple environments per process and thus batch the policy forward pass | |
| this limitation, create multiple environments per process to batch the policy forward pass |
doc/source/rllib/rllib-env.rst
Outdated
| across these vectorized environments. Set ``config.env_runners(num_envs_per_env_runner=..)`` | ||
| to create more than one environment copy per :py:class:`~ray.rllib.envs.env_runner.EnvRunner` | ||
| actor. Additionally, you can make the individual sub-environments within a vector | ||
| independent processes (through python's multiprocessing used by gymnasium). |
There was a problem hiding this comment.
Suggested change
| independent processes (through python's multiprocessing used by gymnasium). | |
| independent processes through Python's multiprocessing used by gymnasium. |
doc/source/rllib/rllib-env.rst
Outdated
|
|
||
| External Application Clients | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
| Multi-agent setups are not vectorizable yet. The Ray team is working on a solution for |
There was a problem hiding this comment.
Suggested change
| Multi-agent setups are not vectorizable yet. The Ray team is working on a solution for | |
| Multi-agent setups aren't vectorizable yet. The Ray team is working on a solution for |
doc/source/rllib/rllib-env.rst
Outdated
| External Application Clients | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
| Multi-agent setups are not vectorizable yet. The Ray team is working on a solution for | ||
| this restriction by utilizing `gymnasium >= 1.x` custom vectorization feature. |
There was a problem hiding this comment.
Suggested change
| this restriction by utilizing `gymnasium >= 1.x` custom vectorization feature. | |
| this restriction by using the `gymnasium >= 1.x` custom vectorization feature. |
doc/source/rllib/rllib-env.rst
Outdated
| This low-level API models multiple agents executing asynchronously in multiple environments. | ||
| A call to ``BaseEnv:poll()`` returns observations from ready agents keyed by 1) their environment, then 2) agent ids. | ||
| Actions for those agents are sent back via ``BaseEnv:send_actions()``. BaseEnv is used to implement all the other env types in RLlib, so it offers a superset of their functionality. | ||
| Some environments may require substantial resources to initialize and run. Should your environments require |
There was a problem hiding this comment.
Suggested change
| Some environments may require substantial resources to initialize and run. Should your environments require | |
| Some environments may require substantial resources to initialize and run. If your environments require |
sven1977
commented
Dec 18, 2024
sven1977
commented
Dec 18, 2024
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Sven Mika <sven@anyscale.io>
…_redo_cleanup_old_api_stack_01
Signed-off-by: Sven Mika <sven@anyscale.io>
simonsays1980
approved these changes
Dec 19, 2024
| .. figure:: images/envs/external_env_setup_client_inference.svg | ||
| :width: 600 | ||
| **External application with client-side inference**: An external simulator (for example a game engine) | ||
| connects to RLlib, which runs as a server through a tcp-cabable, custom EnvRunner. |
There was a problem hiding this comment.
Do we want to use inline code formatting for all classes like EnvRunner?
| .. scale: 75 % | ||
| .. A Unity3D soccer game being learnt by RLlib via the ExternalEnv API. | ||
|
|
||
| RLlib provides an `external messaging protocol <https://github.com/ray-project/ray/blob/master/rllib/env/utils/external_env_protocol.py>`__ |
There was a problem hiding this comment.
Yeah, let's make this a widely adopted standard! :)
| The RLlink Protocol | ||
| ------------------- | ||
|
|
||
| RLlink is a simple, stateful protocol designed for communication between a reinforcement learning (RL) server (ex., RLlib) and an |
There was a problem hiding this comment.
Dumb question: Why not using plain HTTP/2? It is standard and provides security and serialization via Protobuf?
There was a problem hiding this comment.
Definitely in the next iteration! Trying to keep it as simple as possible for this very first iteration. For now, this is just about the message types (what to say when and what to expect back from server?), not really the actual implementation of the messages.
| top-level: action_0 -------------------------------------> action_1 -> | ||
| low-level: action_0 -> action_1 -> action_2 -> action_3 -> action_4 -> | ||
|
|
||
| Alternatively, you could implement an environment, in which the two agent types don't act at the same time (overlappingly), |
There was a problem hiding this comment.
Awesome explanation!
There was a problem hiding this comment.
As I mentioned a while ago: For the long run it might be cool imo, if we get some design support such that it gets a more professional look and feel
|
|
||
| This paragraph describes RLlib's own :py:class`~ray.rllib.env.multi_agent_env.MultiAgentEnv` API, which is the | ||
| recommended way of defining your own multi-agent environment logic. However, if you are already using a | ||
| third-party multi-agent API, RLlib offers wrappers for :ref:`Farama's PettingZoo API <farama-pettingzoo-api>` as well |
There was a problem hiding this comment.
We might want to leave some comment about the game form we implement and PettingZoo (I think it is extensive form game)/OpenSpiel
…_redo_cleanup_old_api_stack_01
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Do-over of RLlib docs (new API stack):
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.