How to handle DTU dataset?

[Pyn-hust]

Great work! The paper says that since the DTU dataset does not have static lighting, images with inconsistent lighting environments are manually deleted. Here are my questions. Looking forward to your answer.
(1) How exactly do you distinguish an image with an inconsistent lighting environment?
(2) Each scene of The original DTU dataset have images under different lighting conditions, which is numbered by 0 to 6. Are the images with inconsistent light environments different within those images which have the same suffix, e.g. _3_r5000.png ? Or does it refer to that light environments of images with the same light number are consistent, and those with different light number are inconsistent?

[liuyuan-pal]

Hi, I use the DTU dataset from here: https://www.dropbox.com/sh/w0y8bbdmxzik3uk/AAAaZffBiJevxQzRskoOYcyja?dl=0
I just manually rule out the images with inconsistent lights. The main reason is that the camera occludes the lights so some images are much darker. I checked the images containing the shadows of the cameras or not to rule out these images.

[Pyn-hust]

Thank you for your answer, very helpful!

[xdobetter]

Hi, great work!
I have a problem with training the DTU dataset, I used the method above to train, but exist bug.

the bug as shown:

• near > far

• density is negative value
  

• weights and color exit inf

• other log info

I loaded the DTU dataset code as shown below, the config is similar NeRO synthetic dataset config, the rest of NeRO's code was not modified.

class VolSDFSyntheticDatabase(BaseDatabase): 
    def __init__(self, database_name, dataset_dir, testskip=8):
        super().__init__(database_name)
        _, model_name = database_name.split('/')
        RENDER_ROOT = dataset_dir
        print("[I] Use VolSDFSyntheticDatabase!")
        print("[I] RENDER_ROOT", RENDER_ROOT) # data/VolSDF
        self.root = f'{RENDER_ROOT}/{model_name}'
        print("[I] self.root", self.root) # data/volsdf/scan24
        self.scale_factor = 1.0
        
        image_paths = sorted(glob_imgs(f'{self.root}/image')) 
        self.img_num = len(image_paths) 
        self.img_ids = [str(k) for k in range(self.img_num)] 
        self.cam_file = f'{self.root}/cameras.npz'
        camera_dict = np.load(self.cam_file)
        scale_mats = [camera_dict['scale_mat_%d' % idx].astype(np.float32) for idx in range(self.img_num)]
        world_mats = [camera_dict['world_mat_%d' % idx].astype(np.float32) for idx in range(self.img_num)]
        self.intrinsics_all = []
        self.pose_all = []
        for scale_mat,world_mat in zip(scale_mats,world_mats):
            P = world_mat @ scale_mat
            P = P[:3,:4]
            intrinsics,pose = load_K_Rt_from_P(None,P) # 解内参和外参
            self.intrinsics_all.append(np.array(intrinsics[:3,:3]).astype(np.float32))
            self.pose_all.append(np.array(pose[:3,...]).astype(np.float32))
        self.rgb_images = []
         
        for path in image_paths:
            rgb = imread(path)[..., :3]
            self.rgb_images.append(rgb)
        self.imgs = np.array(self.rgb_images).astype(np.float32)  # [img_num,1200,1600,3]
        self.poses = np.array(self.pose_all).astype(np.float32) # [img_num,3,4]
        self.ks = np.array(self.intrinsics_all).astype(np.float32) # [img_num,3,3]
        self.resolution = self.imgs[0].shape[:2] # [1200,1600]
    
    def get_image(self, img_id):
        return self.imgs[int(img_id)].copy().astype(np.float32)

    def get_K(self, img_id):
        return self.ks[int(img_id)].copy().astype(np.float32)
    
    def get_pose(self, img_id):
        return self.poses[int(img_id)].copy().astype(np.float32)
    
    def get_img_ids(self):
        return self.img_ids

    def get_depth(self, img_id):
        assert (self.scale_factor == 1.0)
        depth = self.imgs[int(img_id)][..., -1] 
        return depth

    def get_mask(self, img_id):
        raise NotImplementedError

the config as shown below(similar NeRO synthetic dataset config)

name: scan37_shape

####network#####
network: shape
database_name: volsdf/scan37
apply_occ_loss: true
occ_loss_step: 20000
clip_sample_variance: false

######loss######
loss: ['nerf_render','eikonal','std','init_sdf_reg','occ']
val_metric: ['shape_render']
key_metric_name: psnr
eikonal_weight: 0.1
freeze_inv_s_step: 15000

####dataset#####
train_dataset_type: dummy
dataset_dir: data/DTU
train_dataset_cfg:
  database_name: volsdf/scan37
val_set_list:
  -
    name: val
    type: dummy
    cfg:
      database_name: volsdf/scan37

####trainier####
optimizer_type: adam
lr_type: warm_up_cos
lr_cfg: {}
total_step: 300000
val_interval: 5000
save_interval: 1000
train_log_step: 20