GPT-4 Visual Inputs for Robotics

[yuan-alex]

Is anyone else excited to see how we can use GPT-4's visual inputs for perception-action loops? Perhaps we will no longer need to describe the scene to the drone or robotic arm, and instead they will be able to identify their surroundings and nearby objects on their own.

[vbandi]

It's probably going to be very slow and expensive, but definitely an exciting prospect that a single model can do all this.

[Prof-pengyin]

Is there any why we can speed up the procedure of using other 3rd-party tools to help ChatGPT understand the visual inputs for the Robotics application?

Max

[vbandi]

Sure. I think this repo is full of examples. You can use existing computer vision models, such as YOLO and others.