This is the central repo that holds the all the software for GELLO. See the website for the paper and other resources for GELLO https://wuphilipp.github.io/gello_site/ See the GELLO hardware repo for the STL files and hardware instructions for building your own GELLO https://github.com/wuphilipp/gello_mechanical
git clone https://github.com/wuphilipp/gello_software.git
cd gello_software
git submodule init
git submodule update
pip install -r requirements.txt
pip install -e .
pip install -e third_party/DynamixelSDK/python
First install docker
following this link on your host machine.
Then you can clone the repo and build the corresponding docker environment
Build the docker image and tag it as gello:latest. If you are going to name it differently, you need to change the launch.py image name
docker build . -t gello:latest
We have provided an entry point into the docker container
python scripts/launch.py
Now that you have downloaded the code, there is some additional preparation work to properly configure the Dynamixels and GELLO. These instructions will guide you on how to update the motor ids of the Dynamixels and then how to extract the joint offsets to configure your GELLO.
Install the dynamixel_wizard. By default, each motor has the ID 1. In order for multiple dynamixels to be controlled by the same U2D2 controller board, each dynamixel must have a unique ID. This process must be done one motor at a time. Connect each motor, starting from the base motor, and assign them in increasing order until you reach the gripper.
Steps:
- Connect a single motor to the controller and connect the controller to the computer.
- Open the dynamixel wizard
- Click scan (found at the top left corner), this should detect the dynamixel. Connect to the motor
- Look for the ID address and change the ID to the appropriate number.
- Repeat for each motor
After the motor ID's are set, we can now connect to the GELLO controller device. However each motor has its own joint offset, which will result in a joint offset between GELLO and your actual robot arm.
Dynamixels have a symmetric 4 hole pattern which means there the joint offset is a multiple of pi/2.
The GelloAgent
class accepts a DynamixelRobotConfig
(found in gello/agents/gello_agent.py
). The Dynamixel config specifies the parameters you need to find to operate your GELLO. Look at the documentation for more details.
We have created a simple script to automatically detect the joint offset:
- set GELLO into a known configuration, where you know what the corresponding joint angles should be. For example, we set out GELLO in this configuration, where we know the desired ground truth joints. (0, -90, 90, -90, -90, 0)
- run
python scripts/gello_get_offset.py \
--start-joints 0 -1.57 1.57 -1.57 -1.57 0 \ # in radians
--joint-signs 1 1 -1 1 1 1 \
--port /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT7WBG6
# replace values with your own
- Use the known starting joints for
start-joints
. - Use the
joint-signs
for your own robot (see below). - Use your serial port for
port
. You can find the port id of your U2D2 Dynamixel device by runningls /dev/serial/by-id
and looking for the path that starts withusb-FTDI_USB__-__Serial_Converter
(on Ubuntu). On Mac, look in /dev/ and the device that starts withcu.usbserial
joint-signs
for each robot type:
- UR:
1 1 -1 1 1 1
- Panda:
1 -1 1 1 1 -1 1
- xArm:
1 1 1 1 1 1 1
The script prints out a list of joint offsets. Go to gello/agents/gello_agent.py
and add a DynamixelRobotConfig to the PORT_CONFIG_MAP. You are now ready to run your GELLO!
The code provided here is simple and only relies on python packages. The code does NOT use ROS, but a ROS wrapper can easily be adapted from this code. For multiprocessing, we leverage ZMQ
First test your GELLO with a simulated robot to make sure that the joint angles match as expected. In one terminal run
python experiments/launch_nodes.py --robot <sim_ur, sim_panda, or sim_xarm>
This launched the robot node. A simulated robot using the mujoco viewer should appear.
Then, launch your GELLO (the controller node).
python experiments/run_env.py --agent=gello
You should be able to use GELLO to control the simulated robot!
Once you have verified that your GELLO is properly configured, you can test it on a real robot!
Before you run with the real robot, you will have to install a robot specific python package.
The supported robots are in gello/robots
.
- UR: ur_rtde
- panda: polymetis. If you use a different framework to control the panda, the code is easy to adpot. See/Modify
gello/robots/panda.py
- xArm: xArm python SDK
# Launch all of the node
python experiments/launch_nodes.py --robot=<your robot>
# run the enviroment loop
python experiments/run_env.py --agent=gello
Ideally you can start your GELLO near a known configuration each time. If this is possible, you can set the --start-joint
flag with GELLO's known starting configuration. This also enables the robot to reset before you begin teleoperation.
We have provided a simple example for collecting data with gello.
To save trajectories with the keyboard, add the following flag --use-save-interface
Data can then be processed using the demo_to_gdict script.
python gello/data_utils/demo_to_gdict.py --source-dir=<source dir location>
GELLO also be used in bimanual configurations.
For an example, see the bimanual_ur
robot in launch_nodes.py
and --bimanual
flag in the run_env.py
script.
Due to the use of multiprocessing, sometimes python process are not killed properly. We have provided the kill_nodes script which will kill the python processes.
./kill_nodes.sh
If you want to use a new robot you need a GELLO that is compatible. If the kiniamtics are close enough, you may directly use an existing GELLO. Otherwise you will have to design your own.
To add a new robot, simply implement the Robot
protocol found in gello/robots/robot
. See gello/robots/panda.py
, gello/robots/ur.py
, gello/robots/xarm_robot.py
for examples.
Please make a PR if you would like to contribute! The goal of this project is to enable more accessible and higher quality teleoperation devices and we would love your input!
You can optionally install some dev packages.
pip install -r requirements_dev.txt
The code is organized as follows:
scripts
: contains some helpful pythonscripts
experiments
: contains entrypoints into the gello codegello
: contains all of thegello
python package codeagents
: teleoperation agentscameras
: code to interface with camera hardwaredata_utils
: data processing utils. used for imitation learningdm_control_tasks
: dm_control utils to build a simple dm_control enviroment. used for demosdynamixel
: code to interface with the dynamixel hardwarerobots
: robot specific interfaceszmq_core
: zmq utilities for enabling a multi node system
This code base uses isort
and black
for code formatting.
pre-commits hooks are great. This will automatically do some checking/formatting. To use the pre-commit hooks, run the following:
pip install pre-commit
pre-commit install
@misc{wu2023gello,
title={GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators},
author={Philipp Wu and Yide Shentu and Zhongke Yi and Xingyu Lin and Pieter Abbeel},
year={2023},
}
This source code is licensed under the MIT license found in the LICENSE file. in the root directory of this source tree.
This project builds on top of or utilizes the following third party dependencies.
- google-deepmind/mujoco_menagerie: Prebuilt robot models for mujoco
- brentyi/tyro: Argument parsing and configuration
- ZMQ: Enables easy create of node like processes in python.