LeapMotion-Telemanipulation

About

This project is used to control a PANDA arm using a Leap motion controller. This package includes the files for either one or two Leap Motion devices to be used, as well as example code for both changing how multiple devices work, as well as how to expand to any amount of Leap Motion devices, these are found in the Example directory. The instructions of how to set up the devices for both 1 or 2 Leap Motions is described below, as well as in different situations (physical or virtual machines).
This repo also contains a launch scripts directory the launch scripts that I used to run all the bash commands were included, these will need to be modified to the catkin workspace that you use (mine was called "ws"), I used these for conveniance and they are labeled based on what they do.

Info

This was done using ubuntu 18.04lts however should be recreatable on other linux versions with little or no modification.
This repo contains two packages, if using a single Leap Motion, you do not require the VM package
Using multiple Leap Motions through a virtual machine is optional, also a seperate computer may be used instead

Installing with one Leap Motion:

1. Install ROS
2. Install Leap motion sdk driver
3. Later versions of ubuntu* require this fix
4. Install leap motion for ros
5. Install moveit for ros including panda_moveit_config
6. Install tkinter for python `sudo apt-get install python-tk`
7. Install 3dtransform for python 'pip install transforms3d'
8. Copy over leap_panda_telemanipulation into your catkin workspace src folder

Launching with one Leap Motion:

1. Plug in the first Leap Motion
2. Launch the leapd service, ros leap motion, and the leap_to_panda packages on the host machine

Installing with two Leap Motions:

1. Install ROS
2. Install Leap motion sdk driver
3. Later versions of ubuntu* require this fix
4. Install leap motion for ros
5. Install moveit for ros including panda_moveit_config
6. Install tkinter for python `sudo apt-get install python-tk`
7. Install 3dtransform for python 'pip install transforms3d'
8. Copy over leap_panda_telemanipulation and vm_leap into your catkin workspace src folder
9. If you are using a virtual machine download your prefered virtual machine software and set up an os, I used vmware player running ubuntu 18lts
10. Complete steps 1 to 4 again on either your new virtual machine, or your secondary computer
11. Copy over vm_leap into your second/virtual machines catkin workspace src folder
12. Set up the ros network on both machines so that the host pc is the master

Launching with two Leap Motions on a VM:

1. Plug the first Leap Motion into a "less dominant usb socket"**
2. Pass that usb device into the virtual machine
3. Start ros core on the host machine
4. Launch the leapd service, ros leap motion, and the vm_leap packages on the virtual machine
5. Plug in second Leap Motion
6. Launch the leapd service, ros leap motion, and the leap_to_panda packages on the host machine

Launching with two Leap Motions on a seperate machine:

1. Plug in your Leap Motions to each machine
2. Start ros core on the host machine
3. Launch the leapd service, ros leap motion, and the vm leap packages on the second machine
4. Launch the leapd service, ros leap motion, moveit, and the the leap panda telemanipulation packages on the host machine

Configuring the leap panda telemanipulation package

• To specify the usage of either one or two Leap Motion devices, goto `LeaptoPanda.cpp` and set the bool `DualLeaps` to true if not ensure that it is set to false
• In order to change the distance between multiple leap devices, goto `LeaptoPanda.cpp` and set the double `CalibrationDistance` to the distance between the center of the two leap devices in meters. You also need to set self.Calibration` in LeapMoveGroup.py to the same value, if you are only using a single leap device, please ensure that `self.Calibration` is set to 0

How To Use

• The panda arm by default will follow the users right hand, as long as it is in reach
• The panda arms gripper will open and close based off of the users right hand pinching (thumb and index finger)
• The left hand is used to move the workspace around but only whist recognising a closed fist gesture

Key Points

• When using two Leap Motion devices, the host Leap device should be on the left hand side by default
• The final position is calculated (offset + Leap position) * scale
• Offset variables are based off of the raw Leap Motion positions and will not be affected by scale
• When using two Leap Motion devices, if a hand is visible on both devices the one with the highest gesture confidence is taken, if these are the same then the left device is always prefered
• When using two Leap Motion devices, if no hands are detected, the ui will show -calibration/2 for the x positions, this will not affect the data in any way, this is just what is returned when no hand is detected

Example Code

• Provided is example code that was uses the alternate method of working out where then hand is when seen by multiple devices, this uses averaging rather than confidence score
• Provided is example code that demonstrates the scalability of the project by giving a theoretical example of how 4 leap motions in a square would work, and what code would needed to be changed in order to make that happen

*I believe versions after ubuntu 14, maybe, I know for a fact 16lts and 18lts require this

**essentially the first leap motion must be pluged in to a usb socket that the host will ignore, basically we want the 2nd leap to be plugged into a socket that the host machine leap service will discover first so that it will connect to that one and not the one we are using for the VM. This will be an issue until vmware usb virtualisation is imporved

Videos

Dual Leap setup and configuration:

Launch script overview:

Full artefact run through:

Example Code: