Home

Welcome to the MIR Object Recognition ROS2 Code Base wiki!

Process flow

Note: Make sure to source the ROS rolling and devel in all the terminals

Camera feed

Step 1:

• If using a realsense camera, connect the camera to your system where you are running this codebase and run the following command in terminal 1:

ros2 launch realsense2_camera rs_launch.py pointcloud.enable:=true pointcloud.ordered_pc:=true depth_module.profile:=640x480x30 rgb_camera.profile:=640x480x30

• The ros2-beta branch of realsense2_camera package has a bug that doesn't set the pointcloud.ordered_pc parameter to true. So, we have to set it manually using the ros2 param command.

ros2 param set /camera pointcloud.ordered_pc true

• To align the pointcloud depth properly, set the below parameter to true.

ros2 param set /camera align_depth.enable true

Or

• If running in the simulation, run the bag file in loop in terminal 1 using the command below:

ros2 bag play -l bag_files/bag_file_name

• If you dont have the bag file, download one from the below link and save it in ~/mir_object_recognition/bag_files:

https://drive.google.com/file/d/1okPBwca5MgtF6kc3yL3oOEA8TWGFyu-0/view

Note:

• In order for the object recognition to work properly, the RGB image and the Pointcloud data should be of same size and in sync.

• If you want to collect a bag file and use if for later, use the following command in terminal to record the bag file with the required topics:

ros2 bag record /camera/color/camera_info /camera/color/image_raw /camera/depth/color/points /clock /tf /tf_static

Step 2:

• If you are using a realsense camera, you have to publish the tf link between the camera and the base_link in another terminal:

ros2 run tf2_ros static_transform_publisher 0.298 -0.039 0.795 0.0 1.16 -0.055  base_link camera_link

• Change the tf values according to your camera setup.
• The tf information is needed while collecting the bag file too.
• Without tf information between camera_link and base_link, the transformations will not work.

multimodal_object_recognition node

Step 3:

• Run the multimodal_object_recognition launch file in terminal 2 using the command below (make sure to source your workspace).

ros2 launch mir_object_recognition multimodal_object_recognition.launch.py 

• Once the node is launch is up and running, move to step 4

lifecycle_controller node

• lifecycle controller is a simple independent service client used to control a lifecycle node. Please follow the below steps for running.

Step 4:

• Run the lifecycle_controller in terminal 3 using the command below.

ros2 run lifecycle_controller lifecycle_controller --ros-args -p lc_name:=mmor

• lifecycle_controller needs the lifecycle node name (lc_name) as a parameter to run.
• Here, we are passing mmor for our multimodal_object_recognition (mmor) node.
• This lifecycle_controller can be used to control any lifecycle node if the services are written according to ROS2 demo convention.
• After running the lifecycle_controller we can see the following output as shown below.

• If the parameter arguments are not given, lifecycle_controller throws a warning and exits as shown below.

Step 5:

• For the sake of simplicity let us control the lifecycle_talker that is available in ROS2 lifecycle demos
• Using the keyboard inputs, we can control the lifecycle node by changing states. The current state is also displayed.

• Simultaneously we can observe the activity in the lifecycle_talker node.

• If a lifecycle node is not available the following error is displayed. In this case lifecycle_controller is exited after 10 seconds and re-run once the lifecycle node is available.

• If a lifecycle node is manually killed then the lifecycle_controller must be manually stopped by pressing T(terminate).

Dynamic parameter reconfigure

Step 6:

• Run the rqt_reconfigure to dynamically change parameters via gui in terminal 4 using the command below:

ros2 run rqt_reconfigure rqt_reconfigure

• Click 'Enter' after changing any input field values.

RGB Recognizer

Step 7:

• Run the RGB recognizer script in terminal 5 using the command below:

ros2 launch mir_recognizer_scripts rgb_recognizer.launch.py

Visualization

Step 8:

• Run the rviz2 to view the object recognition output and other relevant data in terminal 6 using the command below:

rviz2

• Once the rviz is open, load the ~/mir_object_recognition/src/mir_object_recognition/ros/rviz/mir_object_recognition.rviz file to view the recognized objects and their poses.

Recognition

Step 9:

To perform RGB object recognition, follow the steps below:

• The mmor node will be in unconfigured state by default.
• Change the state to Inactive by entering C in the lifecycle_controller terminal, during which all the parameters, publishers, subscribers and other configurations take place.
• We are disabling the PointCloud (PC) recognizer in the dynamic parameters as we don't have a model for it for now.
• We are also disabling the roi to visualize all the objects detected in the field of view.
• We enable the debug mode to visualize the RGB recognizer output, bounding boxes, object clusters, and object poses in the RVIZ.
• Refresh the rqt_reconfigure gui to see the updated parameters.
• To start processing the data, change the mmor node state to Active by entering A in the lifecycle_controller terminal.
• The mmor node starts subscribing to the image and cloud topics from the RGB-D camera.
• A message filter will synchronize the image and cloud topics based on the timestamp. Once the image and cloud are synchronized, a callback will be triggered.
• The pointcloud is transformed to the target frame_id.
• The transformed cloud is then used to:
  • Fit a plane using RANSAC.
  • Find the clusters based on tunable parameters.
  • Fit bounding boxes to those clusters.
  • Get workspace height.
• Publish the clustered pointclouds and the image to the PC and RGB recognizers respectively.
• The pc recognizer will give the recognized object list which contains object name, pose, bounding box. (as of now this is not tested in this codebase due to lack of pc recognition models with us).
• The RGB recognizer used here is based on YOLOv5 from b-it-bots.
• The RGB recognizer will give the recognized object list which contains object name, bounding box, roi (region of interest).
• The roi information from RGB recognizer is then used to find the respected clusters from the pointcloud.
• Once the clusters for each recognized objects from RGB are extracted, they are used to find the pose of the objects using PCA.
• Both recognized object list from RGB and PC are combined into one object list.
• Based on the reachability of the robot arm, the objects which are outside of roi will be named as decoy.
• Then the poses of all the objects are adjusted:
  • pitch is set to 0 to do a top-pick.
  • for round objects, yaw is ignored.
  • for heavy objects, pose is adjusted to account for the center of gravity.
• Now, the combined object list is published to the object list merger (in case of recognized list from PC and RGB), which merged objects from PC and RGB recognizer into one single list and stores it in the knowledge-base.
• When the debug mode is enabled, the mmor node will publish the bounding boxes from segmentation, RGB and PC object clusters, and RGB and PC pose arrays of all objects.
• To terminate the mmor node, enter X in the lifecycle_controller terminal, which will shut down the node.

Data collection

• Follow the steps for the camera feed and run either the bagfile or the realsense node.

• In a new terminal with the workspace sourced, run the launch file for the data collector component

ros2 launch mir_object_recognition data_collector.launch.py

• In another terminal, run the lifecycle controller node and pass 'data_collector' as the lc_name argument.

ros2 run lifecycle_controller lifecycle_controller --ros-args -p lc_name:=data_collector

• Press C to transition the data_collector component from UNCONFIGURED to INACTIVE state, and then press A to transition it to ACTIVE state. In this state, the component will start saving the pointcloud clusters and the RGB image. By default, the location is the '/tmp/' directory, but if you want to change this, you can provide as an argument to the launchfile like the following example

ros2 launch mir_object_recognition data_collector.launch.py log_directory:=/home/user/Pictures

CLI commands for composition

• To list active components and component container:

ros2 component list

• To load a component to an active component container:

ros2 component load /<component_container> <package_name> <namespace>::<component_name> 

eg:

ros2 component load /MMOR_container mir_object_recognition perception_namespace::DataCollector 

• To unload a component:

ros2 component unload /<component_container> <id_from_component_list>

eg: