Skip to content

Commit

Permalink
Adds known issue docs for simulation stepping (#241)
Browse files Browse the repository at this point in the history
# Description

As pointed out in #234 and #240, the environment returns stale values on
reset. Since there is no immediate fix for this, this MR adds the issue
to "Known issues" page in the documentation.

## Type of change

- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./orbit.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have run all the tests with `./orbit.sh --test` and they pass
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
  • Loading branch information
Mayankm96 authored Feb 12, 2024
1 parent 939a768 commit fa175aa
Showing 1 changed file with 36 additions and 0 deletions.
36 changes: 36 additions & 0 deletions docs/source/refs/issues.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,41 @@
Known issues
============

.. attention::

Please also refer to the `Omniverse Isaac Sim documentation`_ for known issues and workarounds.

Stale values after resetting the environment
--------------------------------------------

When resetting the environment, some of the data fields of assets and sensors are not updated.
These include the poses of links in a kinematic chain, the camera images, the contact sensor readings,
and the lidar point clouds. This is a known issue which has to do with the way the PhysX and
rendering engines work in Omniverse.

Many physics engines do a simulation step as a two-level call: ``forward()`` and ``simulate()``,
where the kinematic and dynamic states are updated, respectively. Unfortunately, PhysX has only a
single ``step()`` call where the two operations are combined. Due to computations through GPU
kernels, it is not so straightforward for them to split these operations. Thus, at the moment,
it is not possible to set the root and/or joint states and do a forward call to update the
kinematic states of links. This affects both initialization as well as episodic resets.

Similarly for RTX rendering related sensors (such as cameras), the sensor data is not updated
immediately after setting the state of the sensor. The rendering engine update is bundled with
the simulator's ``step()`` call which only gets called when the simulation is stepped forward.
This means that the sensor data is not updated immediately after a reset and it will hold
outdated values.

While the above is erroneous, there is currently no direct workaround for it. From our experience in
using IsaacGym, the reset values affect the agent learning critically depending on how frequently
the environment terminates. Eventually if the agent is learning successfully, this number drops
and does not affect the performance that critically.

We have made a feature request to the respective Omniverse teams to have complete control
over stepping different parts of the simulation app. However, at this point, there is no set
timeline for this feature request.


Blank initial frames from the camera
------------------------------------

Expand Down Expand Up @@ -34,3 +69,4 @@ This is then replicated across other references of the same asset since physics
are stored in the instanceable asset's USD file and not in its stage reference's USD file.

.. _instanceable assets: https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/tutorial_gym_instanceable_assets.html
.. _Omniverse Isaac Sim documentation: https://docs.omniverse.nvidia.com/isaacsim/latest/known_issues.html

0 comments on commit fa175aa

Please sign in to comment.