Parallel rendering support in MJX?

[StoneT2000]

I saw there were some other posts (#1101 (comment), #1264) but couldn't seem to see if there was a clear solution. I would like to train on some mjx powered tasks with GPU simulation in addition to rendering every parallel environment for visual-based RL. Is this possible and if not, are there any plans to support this?

Thanks!

[erikfrey]

Hi there - on HEAD you can now find mjx.ray which mirrors mj_ray for planes, spheres, capsules, boxes, and meshes, so you can simulate LiDAR. For simple scenes it should be quite performant.

For RGB, there's nothing currently on the MJX roadmap, but we would love to support such an effort. MJX can interoperate with other GPU programs via dlpack and there seem to be a number of python 3d rendering libraries that this might work with, such as Open3D.

If anyone would like to try this, please let us know!

[Andrew-Luo1]

mjx.ray indeed appears to be quite performant. I discuss a simple depth camera implementation here, which renders up to several million FPS.

[dantes-ai]

Interested in this as well. This currently means that MJX does not support scaling up visual-based RL...