Question about the vision module

[zichunxx]

Hi! Dear developer team!

I'm very interested in the repo after reading the introduction. However, I'm confused about whether I can achieve my goal with this repo and would like to get an answer.

I want to construct a pick-and-place task, in which the vision module can recognize the pose (position and rotation) information of some objects. For now, I just want to realize this with only one object on the table.

I saw the vision module is integrated into this repo. But if the vision module can only finish the object segmentation?

I'm new to this area, so please forgive my ignorance.

Thanks in advance!

[michalvavrecka]

Hi Stay,

you are at the right place. Our simulator is capable of pick and place tasks. The implementation of 6DOF vision (position and rotation) is WIP, we do have sematic segmentation and unsupervised VAE. You can extend both (multiple camera segments or latent space intrerpolation) to the 3D vision.
If you would like to train the PnP task with 6DOF vision, you will face the time problems, as the most advanced 3D bounding box vision algorithms are capable to process about 2 frames per second. It means that your training will take 500000 seconds per 1milion steps = 6 days.
I will reccomend you to train without vision using objects recognizable by 6DOF vision and test it with data from camera (the speed during testing is not cruicial)