Optimize Mesh of an Indoor Scene

[spr-serena]

Hi,

thank you for your great work! However, I'm wondering if this project can be used to optimize meshes that reconstructs indoor scenes? I'm currently optimizing my mesh based on a set of RGB images and camera poses, but it doesn't work well.

My goal: optimize RGB color and position for each vertex, ignore PBR materials and environment light.

Since I'm reconstructing an indoor scene, so I apply alpha = 255 throughout the images for masking.

Here's my json file. I read through other discussions after the training, so I know the texture_res might be too high, but the texture besides the noise still seems to be incorrect.

{
    "base_mesh": "scannet.obj",
    "ref_mesh": "data/nerf_synthetic/scannet",
    "random_textures": true,
    "iter": 5000,
    "save_interval": 100,
    "texture_res": [ 2048, 2048 ],
    "train_res": [468, 624],
    "batch": 1,
    "learning_rate": [0.03, 0.01],
    "ks_min" : [0, 0.25, 0.0],
    "laplace_scale" : 3000,
    "display": [{"latlong" : true}, {"bsdf" : "kd"}, {"bsdf" : "ks"}, {"bsdf" : "normal"}],
    "background" : "white",
    "out_dir": "scannet"
}

I'm using images from Scannet like this:

Here are my results:

Any suggestions would be helpful. Thank you so much!

[JHnvidia]

Hi Serena,

Unfortunately one of the limitations of the method is that you need a clear foreground (object) / background separation with the mask. Density based methods, (nerf/neus/neuralangelo) perform better on full scenes. If you can supervise with a reasonably accurate depth estimate, it might be possible to make training converge, but I don't expect it to work without mask.