Semantic DSP Map

Particle-based Instance-aware Semantic Occupancy Mapping in Dynamic Environments

Gang Chen, Zhaoying Wang, Wei Dong, Javier Alonso-Mora

Accepted by IEEE T-RO [PDF]

Description

This repository contains the code for paper "Particle-based Instance-aware Semantic Occupancy Mapping in Dynamic Environments". This map is an instance-aware ego-centric semantic occupancy map for dynamic environments with objects like vehicles and pedestrians. The map uses particles as the basic representation and outputs occupancy voxels for tasks like motion planning. Static environment is also suitable.

The C++ source file for mapping is in the include/ folder and a ROS1 node example that uses the map is given in src/mapping.cpp

The following gifs shows a mapping result using the Virtual Kitti 2 dataset (left) and using a ZED2 camera in the real world.

Quick Installation

Tested Environment: Ubuntu 20.04 + ROS Noetic

• Our code uses yaml-cpp. Follow instructions in yaml-cpp to install it.

• Then download and compile the mapping code by

mkdir -p semantic_map_ws/src
cd semantic_map_ws/src
git clone git@github.com:g-ch/mask_kpts_msgs.git
git clone --recursive git@github.com:tud-amr/semantic_dsp_map.git
cd ..
catkin build

Quick Test Guidance

Test with Real-world Data Captured by ZED2 Camera

Download a ros data bag The bag contains downsampled depth image, camera pose and a message in "mask_kpts_msgs" form containing segmentation and transformation estimation results. The data is collected in Delft with a ZED2 camera.

Launch the test by

roslaunch semantic_dsp_map zed2.launch
rosbag play zed2_clip.bag  # Add -r 2 to accelerate playing

Test with Data from Virtual KITTI2 Dataset

Download a ros data bag. The bag contains depth image, rgb image, and camera pose from Virtual Kitti 2 dataset, and a message in mask_kpts_msgs form containing segmentation and transformation estimation results.

NOTE: Modify include/settings/settings.h by changing #define SETTING 3 to #define SETTING 2 and recompile by running catkin build to use camera intrinsics, etc. of Virtual Kitti 2 dataset.

Launch the test by

roslaunch semantic_dsp_map virtual_kitti2.launch
rosbag play clip1.bag  

Additional Information

• Backup ros bag Baidu Yun download link. Code: 86if.

• To Visualize the input segmentation and instance color image, particles image and show cout messages, Modify include/settings/settings.h by changing VERBOSE_MODE from 0 to 1, and then recompile by running catkin build. This image shows the segmentation and instance image, particle weight image, and particle number image from left to right.

User Instructions

Overview

Our map handles noise from semantic and instance (Panoptic) segmentation image, depth image, pose, and object transfromation estimation to build an instance-aware semantic map. There are three modes that can be choosen: ZED2 Mode (recommended), Superpoint Mode, and Static Mode. Below shows the instructions of these mode. Before you use one of them, please read this to know the input data format shared by all three modes.

ZED2 Mode (Recommended)

SDK of getting depth image, 3D BBOX, instance masks, and pose from ZED2 camera is used to realize real-time mapping. An additional semantic segmentation network can also be used to give semantic labels to all objects.

Check ZED2 Mode Instructions for details.

Superpoint Mode

This mode uses Superpoints and Superglue to match feature points from adjacent images. Matched feature points on each object (like cars) and the depth image are used to esimate a rough transformation matrix instead of using 3D BBOXes.

Check Superpoint Mode Instructions for details.

Static Mode

This mode doesn't consider instances or dynamic objects. It receives depth image, semantic segmentation image and pose to make a local and global semantic map.

Check Static Mode Instructions for details.

Citation

If you found our work useful, please cite the following.

@misc{chen2024particlebasedinstanceawaresemanticoccupancy,
title={Particle-based Instance-aware Semantic Occupancy Mapping in Dynamic Environments},
author={Gang Chen and Zhaoying Wang and Wei Dong and Javier Alonso-Mora},
year={2024},
eprint={2409.11975},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2409.11975},
}

@article{chen2023continuous,
  title={Continuous occupancy mapping in dynamic environments using particles},
  author={Chen, Gang and Dong, Wei and Peng, Peng and Alonso-Mora, Javier and Zhu, Xiangyang},
  journal={IEEE Transactions on Robotics},
  year={2023},
  publisher={IEEE}
}

Trouble Shooting

Check here

Liciense

Apache-2.0