PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DOF Object Pose Dataset Generation
We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. PEGASUS allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that captures images from both object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.
我们介绍了一种用于生成六自由度（6DOF）物体姿态数据集的多功能数据集生成器——物理增强高斯溅射模拟系统（PEGASUS），它基于三维高斯溅射技术。使用普通相机就能轻松获得环境和物体的表示，并通过高斯溅射重建。PEGASUS允许通过合并环境与一个或多个物体各自的高斯溅射点云，来组成新场景。利用物理引擎可以模拟自然物体在场景中的放置，通过物体与环境提取的网格之间的相互作用来实现。因此，通过结合不同的环境和物体，可以创建大量新场景——无论是静态的还是动态的。通过从各种视角渲染场景，可以提取出多样的数据点，如RGB图像、深度图、语义掩膜和6DoF物体姿态。我们的研究表明，通过在PEGASUS生成的数据上进行训练，姿态估计网络能够成功地从合成数据转移到现实世界数据。此外，我们还介绍了拉面数据集，包括30种日式杯装面条项目。该数据集包括球形扫描，能够捕获物体半球和高斯溅射重建的图像，使其与PEGASUS兼容。