Skip to content

VR and Gazebo

Jump to bottom
Catharine McGhan edited this page Jan 23, 2019 · 33 revisions

Table of Contents

ROS + Gazebo and VR

Official Gazebo support for VR headsets:

Gazebo 3 through Gazebo 5 support the Oculus Rift DK1 headset (see link). From Gazebo 6 onwards, both the Oculus Rift DK1 and DK2 have been supported (see link).

Instructions for getting the Oculus DK2 working with Gazebo 6+ are here: link

The OSRF has no plans to try and support the HTC Vive, but there is community interest, see link.

"Unofficial" Gazebo support for VR headsets (efforts by our group and others)

Last updated 2019 Jan 23:

More updates on this project will be coming soon, but the TL;DR is: success!! :) We have the HTC Vive working with Gazebo 7 under ROS kinetic and Ubuntu 16.04 LTS on a mid-level "VR-ready" system. It's a bit noticeably laggy, but we're working on some simple modifications that should make it faster/better. (For now, turning off collision-detection and using simpler robot models is recommended.)

The source code (BSD 3-clause) with some barebones instructions is currently available here:

This has installation dependencies of: ROS kinetic, Gazebo 7.0, Vrui 4.5, OpenCV, SteamVR for Linux.

If you run into any issues with the setup, please post them under issues at the zhenyushi/vrui_mdf repository. If you'd like to help us with testing, please write to us under the AS4SR/general_info issues page here, instead! :)

We have published a conference paper detailing our current setup.

We'll be linking to a pdf of that conference paper, and also a copy of the conference presentation, soon. (A technical note has also been submitted and is currently in-review.)

Currently, we have not tested this under Ubuntu 18.04, but that is in the pipeline. Also note that right now, this is for the HTC Vive only, 'not' the HTC Vive Pro (sorry).

Note that we're also currently comparing this single-computer Linux-based system to the ROS# + Unity(+ ROS + Gazebo) setup mentioned below, because the ROS# + Unity interface now has support for the HTC Vive and several other headsets.

  • This ROS# + Unity setup solves the headset display lag issue by decoupling the headset display from Gazebo, using the Unity environment to handle the visuals/rendering-only. The robot and general environmental responsiveness itself is still restricted by the speed at which Gazebo (the dynamics simulator) can update the scene using the physics engine.

Status of HTC Vive + Gazebo:

Last updated 2018 Apr 12:

Work is underway to get the HTC Vive working with Gazebo 7 under ROS kinetic and Ubuntu 16.04 LTS on a VR-capable system. This is currently a side project for the AS4SR lab, though.

It looks like there are a couple possible paths forward. Currently the two most-promising are:

  1. Just trying to set up Gazebo to stream visuals from a stereo camera feed to an application made from Doc-ok's Vrui for the HTC Vive
    • This has been tested-working under Ubuntu 16.04 with the new hardware stack enabled (4.10 kernels beyond 16.04's 4.4 kernels) via sudo apt-get install --install-recommends linux-generic-hwe-16.04 xserver-xorg-hwe-16.04 and with an AMD card of the newest AMDGPU-PRO drivers.
    • Vrui links: main blog with explanation and Vrui main site (with generic/non-Vive-specific download and quick-install pages)
    • reddit thread with help on the libraries: link (though pay attention specifically to this part on down to the solution, which involves modding the libraries+paths referenced in the RunViveTracker.sh file)
    • Note that you do need to install SteamVR (and Steam) under Linux also, since Vrui depends on their half-working VR for Linux support. This is discussed (and done) as part of the Vrui installation instructions (see above).
  2. Actually editing the current Gazebo plugin to connect to the HTC Vive instead of the Oculus Rift DK1/DK2
    • This would still involve looking at the Vrui toolkit to get the headset parameters to figure out the windowing, and potentially also involve using parts of Vrui to connect to the Lighthouse sensors to get the relative position and orientation of the headset to update the stereo camera position and orientation in the Gazebo sim; it may be less work though, and could potentially update far more quickly/efficiently than the 'first path forward' given above. It could potentially take more work, though, since the code will have to be far more closely integrated with Gazebo via the modified plugin.
Note that we are currently ***NOT*** attempting to get the Oculus Rift CV1 working with Gazebo. This is because Oculus has (1) closed out their Linux SDK libraries before supporting the CV1, and (2) the CV1 has much better hardware.
  • (2) may be counterintuitive, but the difficulty presented is this: the CV1 has two separate hardware displays internal to the headset, one for each eye. This requires compiling programs specifically for the CV1 for things to display properly.
    • Oculus used to have dual-monitor support for the DK1 and DK2 headsets, because those headsets have a single display; thus, they could be easily treated as a second monitor screen by a video card / computer system when plugged in via an HDMI port.
    • Oculus discontinued "dual-monitor" support for the CV1 headset for this reason -- it's not one display that can be "extended" easily -- it's actually two displays, and trying to treat two displays like one display is a horrible kludge.
      • In other words, Oculus would rather "force" developers to create VR programs that work properly, so that the update framerates and display resolutions are kept up to snuff / won't make people sick.
  • ...and yes, this is why it's likely easier to get the HTC Vive working with Gazebo's Oculus plugin than Oculus' own CV1 -- the Vive has a single hardware display like the DK1 and DK2
Source code and files all-in-one-place:

Other VR-related resources:

Last updated 2018 Apr 12:

ROS Indigo:
  • VR and Gazebo with Android (online textbook, good resource): link

Some newer Vive-ROS projects to look at:

Last updated 2018 Apr 12:

ROS + VR with Android phone-headsets

Last updated 2018 Apr 12:

Going at this sideways -- ROS + Unity and Microsoft HoloLens:

Last updated 2018 Apr 12:

It looks like there's some VR support for Microsoft HoloLens and Unity (from the Unity side) plus ROS via rosbridge. This requires a Windows computer running Unity and supporting the Microsoft HoloLens, and an Ubuntu Linux computer running ROS and rosbridge.

See holoROS project:

Potential alternate path for Oculus Rift CV1

Last updated 2018 Apr 12:

Once Microsoft HoloLens is working with Unity, could maybe get Oculus Rift CV1 also working in a similar way:

  • https://developer.oculus.com/downloads/unity/
Clone this wiki locally