ROS2 Foxy node for computer vision tasks inference on Versal VCK190 together with Husky Rover and OAK-D Lite depth camera
The goal of Work Package 300 is to develop a framework that enables performing deep learning inference together with Robot Operating System (ROS2) directly on the Xilinx Versal VCK190 developer board. Additionally, the framework can grab color and depth images from a commonly available sensor (in this case it is Luxonis OAK-D Lite) and publish them to ROS2 topics, and also can control the Husky robot using ROS2 messages.
Below you can find the list of hardware and software that we use for this project.
- Vitis 2022.1
- Vitis HLS 2022.1
- Vivado 2022.1
- Petalinux 2022.1
- Host PC with Ubuntu 20.04.4 LTS.
- Xilinx Versal VCK190 developer board.
- Luxonis OAK-D Lite depth camera.
- Husky robot with Ouster OS1-64 lidar sensor.
- IMU sensor.
- VS Code - good IDE for remote development.
- Tmux - terminal multiplexer, allows to detach and reattach to a running session.
- ROS2 Foxy - host installation enables the generation of ROS2 messages and services.
- Picocom - serial communication terminal.
The main part of this step is based on the VCK190 Base TRD document. It contains all the necessary information about the VCK190 board and how to create a project for it with working DPU inference. Follow 1 to 4.4 steps from the document and then continue with the steps below to generate the Petalinux image and copy it to an SD card.
Note, that
$WORKSPACE
variable is the path to the petalinux project directory generated insidevck190-base-trd
. In our case it is/home/put/Projects/vck190-base-trd/petalinux/xilinx-vck190-base-trd/
.
# COPY DEDICATED VCK190 FILES TO THE PROJECT
cp $WORKSPACE/../../overlays/xvdpu/kernels/binary_container_1.xsa $WORKSPACE/project-spec/hw-description/system.xsa
cp $WORKSPACE/../../overlays/xvdpu/kernels/binary_container_1.xclbin $WORKSPACE/images/linux/
cp -r $WORKSPACE/../../overlays/xvdpu/apps/* $WORKSPACE/project-spec/meta-base-trd/recipes-apps
# BUILD PROJECT
petalinux-build
# GENERATE BOOT FILES
petalinux-package --boot --u-boot --dtb $WORKSPACE/images/linux/system.dtb --qemu-rootfs no --force
# GENERATE BOOT IMAGE
petalinux-package --wic --bootfiles "binary_container_1.xclbin ramdisk.cpio.gz.u-boot boot.scr Image BOOT.BIN"
# COPY BOOT IMAGE TO SD CARD
sudo dd if=$WORKSPACE/images/linux/petalinux-sdimage.wic of=<DEVICE> status=progress conv=sync
Changes in the Petalinux configuration are required to enable some features. You can edit it using petalinux-config
tool.
-
Set MAC address for the ethernet interface.
Subsystem AUTO Hardware Settings ---> Ethernet Settings ---> Disable: Randomise MAC address Set: Ethernet MAC address Enable: Obtain IP address automatically
-
Enable startup from SD card.
Image Packaging Configuration ---> Root filesystem type (INITRD) ---> Enable: INITRD
-
Set hostname.
Firmware Version Configuration ---> Set: Host name Set: Product name Set: Firmware version
Changes in the rootfs are required to configure some features. You can edit it using petalinux-config -c rootfs
tool.
-
Ensure dropbear SSH server is enabled.
Image Features ---> Enable: ssh-server-dropbear
-
Enable debug tweaks and auto-login to enable remote development.
Image Features ---> Enable: debug-tweaks Enable: auto-login
Some changes in kernel modules are required to enable some USB features. You can edit it using petalinux-config -c kernel
tool.
Changes:
-
Enable
CONFIG_USB_SERIAL_PL2303
module to enable Prolific USB-UART converter. -
Enable
USB_SERIAL_XSENS_MT
module to enable Xsens MTi USB-UART converter. -
Enable
INPUT_JOYSTICK
andINPUT_JOYDEV
modules to activate joystick support.
This part is mostly based on the ROS 2 Humble Hawksbill with Yocto and PetaLinux article.
Steps:
-
Clone modified
meta-ros
package:git clone https://github.com/PUTvision/put-meta-ros $WORKSPACE/project-spec/meta-ros
-
Edit configuration file:
$WORKSPACE/build/conf/bblayers.conf
and add parts:-
add headers:
# define the ROS 2 Yocto target release ROS_OE_RELEASE_SERIES = "honister" # define ROS 2 distro ROS_DISTRO = "foxy"
-
append sources:
${SDKBASEMETAPATH}/../../project-spec/meta-ros/meta-ros2-foxy \ ${SDKBASEMETAPATH}/../../project-spec/meta-ros/meta-ros2 \ ${SDKBASEMETAPATH}/../../project-spec/meta-ros/meta-ros-common \
-
-
Create destination image directory:
mkdir -p $WORKSPACE/project-spec/meta-user/recipes-images/images
and copy our configuration file:
cp $WORKSPACE/project-spec/meta-ros/put_sources/petalinux-image-minimal.bbappend $WORKSPACE/project-spec/meta-user/recipes-images/images/`
-
Each layer below clone into the
$WORKSPACE/project-spec/meta-user/recipes-apps/
directory. -
Our application layers:
- put-meta-base-config - Base configuration containing common things like bashrc, etc.
- put-meta-depthai-core - DepthAI core package. Based on depthai-core sources. It enables the usage of DepthAI API in C++ and Python.
- put-meta-depthai-ros - ROS bindings for DepthAI. Based on depthai-ros/foxy sources. It provides ROS2 nodes for DepthAI API.
- put-meta-husky - Husky mobile robot package for Petalinux usage. Based on husky/foxy-devel package.
- put-meta-xsens - modified Xsens package. Based od bluespace_ai_xsens_ros_mti_driver.
-
Connect via USB using serial console (115200 baud rate, 8N1, no flow control).
sudo picocom -b 115200 /dev/ttyUSB1
Note: You have to use USB1 port.
Then, set bash as default shell (.bashrc and .bash_profile are used by default)
chsh -s /bin/bash root
Call
bash
command to start bash shell. You should see welcome message, together with some information about the system: ETHERNET IP ADDRESS and ROS_DOMAIN_ID.After that, you can login via SSH using
root
user androot
password. -
Create ROS2 inference engine node
mkdir -p ~/ros2_ws/src cd ~/ros2_ws/src git clone https://github.com/PUTvision/ros2_fpga_inference_node.git inference_node/ cd ~/ros2_ws colcon build source install/setup.bash
Note: Note that actually the repository is private. You have to contact with the author to get access to it.
# run inference engine
ros2 run inference_node inference_engine
Scripts below are added to Systemd services and you don't need to run it.
# run imu node
ros2 launch bluespace_ai_xsens_mti_driver xsens_mti_node.launch.py
# run husky node
ros2 launch husky_base put_base.launch.py
# run camera node
ros2 launch depthai_cognition rgb_depth_publisher.launch.py
-
Booting process is performed 3 times. It is caused by the fact that the system is not able to find the rootfs partition.
-
Sometimes, during a boot process, the system hangs with a message like: "No found devices in /dev...". In this case, you have to reboot the system.
-
When booting without Internet connection, it is required to set datatime manually. Otherwise, the ROS system will hang. Because we set
date -s "2023-01-01 12:00:00"
as systemd service, it is required to update the date after 2023. -
Boot time is very long. From switching on the power to begin of communication with the Husky robot, it takes about 2 minutes.
This work was funded by European Space Agency OSIP, PO number: 4000138073, COGNITION project.