Xuxin Cheng*
·
Jialong Li*
·
Shiqi Yang
Ge Yang
·
Xiaolong Wang
This code contains implementation for teleoperation and imitation learning of Open-TeleVision.
conda create -n tv python=3.8
conda activate tv
pip install -r requirements.txt
cd act/detr && pip install -e .
Install ZED sdk: https://www.stereolabs.com/developers/release/
Install ZED Python API:
cd /usr/local/zed/ && python get_python_api.py
If you want to try teleoperation example in a simulated environment (teleop_hand.py):
Install Isaac Gym: https://developer.nvidia.com/isaac-gym/
For Quest local streaming, follow this issue.
Apple does not allow WebXR on non-https connections. To test the application locally, we need to create a self-signed certificate and install it on the client. You need a ubuntu machine and a router. Connect the VisionPro and the ubuntu machine to the same router.
- install mkcert: https://github.com/FiloSottile/mkcert
- check local ip address:
ifconfig | grep inet
Suppose the local ip address of the ubuntu machine is 192.168.8.102
.
- create certificate:
mkcert -install && mkcert -cert-file cert.pem -key-file key.pem 192.168.8.102 localhost 127.0.0.1
ps. place the generated cert.pem
and key.pem
files in teleop
.
- open firewall on server
sudo iptables -A INPUT -p tcp --dport 8012 -j ACCEPT
sudo iptables-save
sudo iptables -L
or can be done with ufw
:
sudo ufw allow 8012
tv = OpenTeleVision(self.resolution_cropped, shm.name, image_queue, toggle_streaming, ngrok=False)
- install ca-certificates on VisionPro
mkcert -CAROOT
Copy the rootCA.pem via AirDrop to VisionPro and install it.
Settings > General > About > Certificate Trust Settings. Under "Enable full trust for root certificates", turn on trust for the certificate.
settings > Apps > Safari > Advanced > Feature Flags > Enable WebXR Related Features
-
open the browser on Safari on VisionPro and go to
https://192.168.8.102:8012?ws=wss://192.168.8.102:8012
-
Click
Enter VR
andAllow
to start the VR session.
For Meta Quest3, installation of the certificate is not trivial. We need to use a network streaming solution. We use ngrok
to create a secure tunnel to the server. This method will work for both VisionPro and Meta Quest3.
- Install ngrok: https://ngrok.com/download
- Run ngrok
ngrok http 8012
- Copy the https address and open the browser on Meta Quest3 and go to the address.
ps. When using ngrok for network streaming, remember to call OpenTeleVision
with:
self.tv = OpenTeleVision(self.resolution_cropped, self.shm.name, image_queue, toggle_streaming, ngrok=True)
- After setup up streaming with either local or network streaming following the above instructions, you can try teleoperating two robot hands in Issac Gym:
cd teleop && python teleop_hand.py
-
Go to your vuer site on VisionPro, click
Enter VR
andAllow
to enter immersive environment. -
See your hands in 3D!
-
Download dataset from https://drive.google.com/drive/folders/11WO96mUMjmxRo9Hpvm4ADz7THuuGNEMY?usp=sharing.
-
Place the downloaded dataset in
data/recordings/
. -
Process the specified dataset for training using
scripts/post_process.py
. -
You can verify the image and action sequences of a specific episode in the dataset using
scripts/replay_demo.py
. -
To train ACT, run:
python imitate_episodes.py --policy_class ACT --kl_weight 10 --chunk_size 60 --hidden_dim 512 --batch_size 45 --dim_feedforward 3200 --num_epochs 50000 --lr 5e-5 --seed 0 --taskid 00 --exptid 01-sample-expt
- After training, save jit for the desired checkpoint:
python imitate_episodes.py --policy_class ACT --kl_weight 10 --chunk_size 60 --hidden_dim 512 --batch_size 45 --dim_feedforward 3200 --num_epochs 50000 --lr 5e-5 --seed 0 --taskid 00 --exptid 01-sample-expt\
--save_jit --resume_ckpt 25000
- You can visualize the trained policy with inputs from dataset using
scripts/deploy_sim.py
, example usage:
python deploy_sim.py --taskid 00 --exptid 01 --resume_ckpt 25000
@article{cheng2024tv,
title={Open-TeleVision: Teleoperation with Immersive Active Visual Feedback},
author={Cheng, Xuxin and Li, Jialong and Yang, Shiqi and Yang, Ge and Wang, Xiaolong},
journal={arXiv preprint arXiv:2407.01512},
year={2024}
}