Optimize mesh of a single building using custom images

[jialechen7]

Hi,
Thank you for your great work! I'm using this project to reconstruct a single building mesh using my custom real-world images(about 300 images) and camera poses. However, it doesn't work well.
I generated mask images from the existing pose and ref_mesh(it is accurate) , processed using scale_images.py in nerd in the project, and then I copied nerd_ehead.json and only modified the ref_mesh and out_dir parameters to optimize mesh.
I want to know if I did anything wrong, or if there is any way to improve it.
My results are as follows.
My config:

My ref_mesh:

My nvdiffrec result:

[jmunkberg]

Hello @jialechen7,

I would suggest setting the option "camera_space_light" to false if your dataset is captured using one single lighting configuration and many camera views. The camera_space_light flag is a WAR for the case of a turntable setup (like the nerf_ehead dataset), when the object is rotating and the camera and light are static.

For improving quality, here are some recommendations, but it is not always possible for real-world datasets.

• Ensure that the foreground segmentation masks and camera poses are as accurate as possible.
• We assume a single lighting configuration for all images in the dataset, so a day-to-night setup is not ideal.
• Check the mesh_scale parameter so that the initial grid tightly covers the bounding box of the model.
• You can play with increasing the dmtet_grid size to 128 for more details (given that the masks and camera poses are highly accurate).

I hope this helps.

[jialechen7]

Thank you for your response. After adjusting the mesh_scale parameter to better cover my model, I achieved somewhat better results. However, I'm wondering how the program can adaptively adjust the mesh_scale parameter to ensure that the tetrahedra cover my model when the ref_mesh is unknown, especially for inputting nerd data. It seems that I encountered some issues where the details were not well reconstructed due to the mesh_scale being too large.

[jmunkberg]

There is no automatic adjustments of the mesh scale in the current code base unfortunately. You could implement some space carving based on the foreground segmentation masks to determine a reasonable initial bbox if you want it automated. A brute-force way is just to look at the early output images, and adjust the mesh_scale parameter until the initial triangle cloud tightly fits the space covering your model.

[jialechen7]

Thank you. I adjusted the mesh_scale to a suitable value and checked the output images. If I understand correctly, the tetrahedra should currently be able to cover my model. However, I'm confused as to why, in the following output images, the mesh in the red boxed area seems to have improved, but in the final output dmtet_mesh, the resulting mesh doesn't resemble that area at all. I'm not sure what could be causing this discrepancy. Could you please provide some insight?
My some output images:


My output dmtet mesh: