Computer Vision and ROS

ROS/Images/OpenCV

Useful links

• image_pipeline is the stack for dealing with ROS Image messages. The packages below are part of image_pipeline:

  1. image_view
     • Package used to view ROS Image messages.
  2. image_proc
     • Package used to process images (often takes input from camera driver or rosbag) and undistorts it. See stereo_image_proc for the stereo equivalent.
  3. camera_calibration
     • Package used to calibrate the camera (to get intrinsic parameters).
     • After calibrating the camera, the camera node (i.e. usb_cam) will take care of publishing the calibration information for that camera on the /camera_info topic along with the raw image topic.
  • Intrinsic camera parameters are those that are independent of the camera position or orientation. They are essentially focal length and the distortion coefficients.
  • Extrinsic camera parameters are those that describe the "coordinate system transformations from 3D world coordinates to 3D camera coordinates." In other words, how to turn the pixels in an image to points in 3D space.

• image_transport is the hidden layer underneath nodes that use ROS messages. Essentially, it takes care of publishing all the extra "topics" for the different compression formats (image_transport doesn't use ROS topics, but something very similar and transparent for the user).

  • ROS Image messages can be in RAW format (not encoded or compressed), or they can be compressed (in JPEG format mostly), in which case they would be CompressedImage ROS message types. In order to take care of all this nonsense compression magic, image_transport is used as a hidden layer underneath nodes that use ROS Image messages (such as image_view or image_proc).

• Notes on Images in ROS

• Besides the image_resizer we have, you can also compress/decompress images in rosbags by playing them and recording them at the same time using the different transport (compressed or not). Also, check out this image_compressor node (this one uses PNG compression).

How to rosbag (record) camera image data

1. Make sure the topic you want to record camera messages from is running. Use image_view to check (see below section).
2. run rosbag record -O my_bag_name.bag /camera/image_raw/compressed. This assumes you want your output bag file to be named my_bag_name.bag, and that the topic you're recording is named /camera/image_raw/compressed. Make sure you only record compressed image data! See this issue for details.

How to display rosbag'ged camera data in ROS using image_view

1. Find the name of the image topic you want to display. I recommend using rostopic list or the node graph in rqt. Let's say your topic name is /camera/image_raw.
2. Play your rosbag data using rosbag play my_bag_name.bag. You can use the -l parameter to make the bag loop like this rosbag play -l my_bag_name.bag.
3. Now run the image_view node: rosrun image_view image_view image:=/camera/image_raw. If you bagged compressed image data which you should do, then throw _image_transport:=compressed at the end of the previous command (this basically changes the _image_transport parameter to compressed).

Using image_transport republish to change compress/uncompress ROS image messages

Example launch file code to republish a raw ROS Image stream into CompressedImage:

<node pkg="image_transport" type="republish" name="republish" output="screen" args="raw in:=/left/image_raw compressed out:=/camera/image_raw"/>

Cameras and Lenses

• List of companies:

  • Pointgrey
  • Axis
  • Edmund Optics
  • Thor Labs

• Axis Lens Guide

• Edmund Optics Imaging Resource Guide

• Read all this about lenses and here also.

• What to lookout for:

  • The lens focal length and the camera image sensor size determine the field of view you get.
  • Also watch out for aperture size and vignetting.

• How to calculate Angle of view:

  • First find the image sensor dimensions (check link below).

  • Then find the lens focal length.

  • Then plug them into this equation:

    

• Don't worry too much about lens mount types because there's adapters out there. Usually C-mount or CS-mount are what we'll need. (CS cam and C lens can be used with a spacer, but not vice versa)

• List of image sensor dimensions.

• Possibly use DSLR cameras: use Python scripts to remote control cameras.

• Global vs Rolling shutter

• CCD vs CMOS sensor types

• Iris features (basically controllable aperture. Auto-iris works best for changing lighting conditions but might suffer diffraction lens flare artifacts under bright light)