Immersive Embodied Telemanipulation System with a Velocity Controller

0. Demo

1. Description of Files

• mr_teleop_lite6_mvc.py: Cartesian velocity version.

• lite6mr.exe: Unity user client (PCVR). The source code of the Unity client might be released in a separate repo.

2. Hardware

• UFactory Lite 6 Robotic Manipulator (require velocity controller)
• Meta Quest 3 or Meta Quest Pro

3. Install

• Install Ubuntu 20.04 and ROS Noetic (recommend to use Windows+WSL2 [Ubuntu 20.04])
• Install xarm_ros: link
• Install ROS-TCP-Endpoint: link
• Install dependencies: sm, rtb, r

• Download Oculus App: link
  • Version: Oculus App 60.0.0.162.352 (60.0.0.162.352)
  • Connect your MR device to your PC, normally Windows, using Quest Link (recommended) or AirLink
  • Enable developer mode: link
  • Oculus App -> Setting -> General ->Turn on "unknown source"
  • Oculus App -> Setting -> Beta -> Turn on "Runtime", "Oculus Link Passthrough (required)"
  • Launch Quest Link : link
• Download lite6mr.exe from release page [link]

4. Bring-up

• Open xArm Studio through browser: http://192.168.1.<your-robot-ip>:18333

• Plug-in Quest Link, then run lite6mr.exe. Reset initial view as desired.

• Open Ubuntu (WSL2), then run following nodes in separate terminals:

  ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1
  roslaunch ros_tcp_endpoint endpoint.launch
  roslaunch xarm_bringup lite6_server.launch robot_ip:=192.168.1.<your-robot-ip>
  rosrun xarm_bringup mr_teleop_lite6.py

• Enjoy!

5. Additional Setup for Agile Motion (Experimental, Optional)

May cause inevitable damage! Be very cautious! Be aware of safety!

• Turn on reduced mode for Lite 6
• Set joint speed limit to 180 deg/s
• Increase p_gain

6. Notes for adapting for other platforms

Let's build this codebase for diverse robots.

7. Citations

Immersive Embodied Telemanipulation System with a Velocity Controller paper

@INPROCEEDINGS{10718831,
  author={Wu, Ziniu and Chu, Yijie and Zhu, Xiaohui and Sun, Qilei and Yue, Yong},
  booktitle={2024 29th International Conference on Automation and Computing (ICAC)}, 
  title={Immersive Embodied Telemanipulation System with a Velocity Controller}, 
  year={2024},
  volume={},
  number={},
  pages={1-6},
  keywords={Automation;Operating systems;Mixed reality;User experience;Human in the loop;6-DOF;Motion control;Robots;Augmented reality;Graphical user interfaces;Telerobotics;Teleoperation;Mixed Reality;Digital Twin;Manipulation},
  doi={10.1109/ICAC61394.2024.10718831}}

8. Acknowledgments

The author would like to express his sincere gratitude to people from the Virtual Engineering Centre (VEC), XJTLU who contributed to the development and experiments of this research.

9. References

• Unity Robotics Hub
• Robotics Toolbox for Python

10. Future Roadmap

• Diverse support for robotic platforms
• Enhanced UI
• Bimanual manipulation