Skip to content

Bachelors Thesis: Photorealistic Rendering of Training Data for Object Detection and Pose Estimation with a Physics Engine

Notifications You must be signed in to change notification settings

dralois/PR-Rendering

Repository files navigation

PRRendering Dataset Generator

Required Libraries & Setup

Most of the required libraries are automatically downloaded, compiled and installed correctly, however the requirements listed bellow need to be installed manually. On Linux, Clang-9 is needed to compile PhysX. For everything else GCC-9 is fine. On Windows, Visual Studio 19 has been tested and works, although earlier versions should be fine, too.

  • Download & install at least version 3.15, preferrably 3.17
  • Although 3.15 is supported, only 3.17 has been tested
  • Download & install according to the docs
  • Setup the build pipeline accordingly for Windows or Linux
  • Blender has to be built as described here, since there is no official release for the python module version
  • An unofficial prebuilt version is provided & installed automatically
  • The 2.82 folder needs to be put into the python install directory
  • Blenderseed just needs to be downloaded, as it is a Blender plugin
  • Linux is currently not offically supported and needs to be built manually
  • Appleseed is included in the Blenderseed plugin
  • An unofficial prebuilt version is provided & installed automatically
  • Most Linux distributions include boost
  • On Windows it is necessary to install / build boost
  • Boost is statically linked, build / install accordingly
  • The required modules are System, Thread, Filesystem, Python (3.7, may have to be built manually!)

Boost build instructions

  • Build bjam / b2 using bootstrap
  • Create a user-config.jam in the same directory
  • Adjust and add the following:
using python : 3.7 
  : Path-to-Python-dir\\python.exe
  : Path-to-Python-dir\\include
  : Path-to-Python-dir\\libs ;
  • Execute b2 two times:
./b2 --with-python --with-filesystem --with-thread --with-system stage
./b2 --with-python --with-filesystem --with-thread --with-system install
  • Potentially add and set BOOST_ROOT entry in cmake
  • Download script is also provided in data/3Rscan. It requires python2 to run smoothly
  • To prepare the dataset, run the unpack.sh script and set path to the downloaded data
  • Set path of 3RScan in the config.json
  • The 3D objects can be downloaded through the provided link
  • Unpack the zip file into a folder and setup the config file
  • Other meshes may be used instead, supported are Wavefront and glTF.

Building

Windows

  • Create a new folder (e.g. build) and setup cmake
  • Click generate & wait until all dependencies have been built
  • Build the project using Visual Studio

Linux

  • Make sure both Clang 9 & GCC 9 are installed
  • Make sure boost & boost::python37 are available
  • Make sure python 3.7 dev is available
  • Create a new folder (e.g. build)
  • Generate the project from root directory
cmake -S ./ -B ./build -DCMAKE_BUILD_TYPE=Release
  • Compile the project in build folder
cd build && make -j 16
  • Copy all PhysX libraries & the appleseed library from the build folder to /lib/

Configuration & Options

  • The config.json file contains options & settings
  • General settings are in the first block, available memory needs to be set
  • The paths in the second block need to be set to folders & files
  • Blurry image detection can be adjusted in the third block
  • Simulation & render output can be controlled in the fourth block
  • Object physics can be adjusted in the fifth block
  • Optionally, custom intrinsics can be set in the sixth block
  • Lastly, the objects that will be used in the simulation have to be defined

Scene & Meshes Layout

Meshes

  • Meshes need to be of .obj, .ply or glTF 2.0 (.glb / .gltf) format
  • Used meshes need to be listed in the corresponding array in the config file (render_objs)
  • Each mesh has three properties but only the path is required
    • Each file path (mesh_path) can be either relative to the base directory (meshes_path) or an absolute path
    • The optional diffuse albedo texture (mesh_texture) needs to be in the same directory as the mesh
    • The classification (mesh_class) can be any arbitrary string
    • The scaling factor (mesh_unit) needs to be set so 1 unit = 1 meter
  • If no texture path is given, the program attempts to use a default diffuse albedo texture (meshdir\objectname_color.png)
  • The meshes are read with the assumption that Y=forward, Z=up

Scenes

  • Each scene needs to have a rgbd folder and a mesh in .obj format
  • The scene mesh needs to be named mesh.refined.obj, it may be scaled using the config file (scene_unit)
    • The mesh is read with the assumption that Y=forward, Z=up
    • If present, the diffuse albedo texture needs to be named mesh.refined_0.png
  • The rgbd folder must contain a _info.txt, containing details about the used camera intrinsics
    • The camera width & height in pixels are named m_colorWidth = x & m_colorHeight = x
    • The intrinsics are a 4x4, rhs matrix named m_calibrationColorIntrinsic = x x ... (16 values)
    • Alternatively, the intrinsics may be set manually in the config file instead
  • The frames have to end with framenum.color.jpg with corresponding framenum.pose.txt
  • The pose file has to contain only the 4x4, rhs extrinsics matrix, one row per line, with Y=forward, Z=up