Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion

Tomas Jakab*, Ruining Li*, Shangzhe Wu, Christian Rupprecht, Andrea Vedaldi. University of Oxford. (* equal contribution)

In 3DV 2024.

Farm3D learns an articulated object category entirely from "free" virtual supervision from a 2D diffusion-based image generator. We propose a framework that employs an image generator, such as Stable Diffusion, to produce training data for learning a reconstruction network from the ground up. Additionally, the diffusion model is incorporated as a scoring mechanism to further improve learning. Our method yields a monocular reconstruction network capable of generating controllable 3D assets from a single input image, whether real or generated, in a matter of seconds.

Code and 3D Articulated Animals Benchmark

Coming soon!