Tutorial 3: Video and RGBD

Previous 2: ROS

Next 4: Extending to a new robot

Passing Kinect or other RGBD data to Unity

Displaying Kinect or other RGBD data using this package follows the same structure as a normal ROS camera. For RGBD data you will need to pass the RGB image, the depth image, and the camera pose. You can view both the RGB image and the point cloud.

Passing in the data

Like the standard ROS framework you will need to publish images and the pose on certain topics. Do not pass the point cloud directly, as this is an data-inefficient method. This unity project calculates the point cloud using a Unity shader.

Set the topics and parameters of the images and pose using the Unity panel.

Note: There are two common errors when using the Kinect view from this package.

1. The depth image must be passed as an int16 (short in C-sharp).
2. The width and height of the depth image listener are not automatically detected. The current setting are for a qHD (quarter HD) image. You can change the width and height from the Unity GUI.

Getting the camera pose

The camera pose is necessary to know how to display the point cloud relative to the robot. If you do not have a convenient method for getting the camera pose, you can "grab" the virtual kinect in Unity and move it around until the point cloud matches up with what is expected. You can also hard code the point cloud position and rotation.

In our lab, we use a motion capture system with reflective spheres in calibrated locations on both the robot and the kinect.