down right back to right up back conversion in llff.py

[hturki]

Describe the bug
I'm trying to make sure that I understand the purpose of https://github.com/kwea123/nerf_pl/blob/dev/datasets/llff.py#L198. In particular, we're changing the coordinate system of the rotation matrix R but not of the translation vector, which remains in "down right back" coordinates - is that intentional?

Also bmild/nerf#34 suggests that this is mainly intended for forward facing scenes where we're using NDC, but if I'm interpreting the code correctly this transformation still happens for 360 scenes. Is that intentional and desired behavior?

Which branch you use
dev

[kwea123]

This is not specific to NDC, but for all scenes constructed using COLMAP, which is desired behavior. Actually this conversion has relation with this part:

nerf_pl/datasets/ray_utils.py

Lines 5 to 24 in 20d1670

	def get_ray_directions(H, W, focal):
	"""
	Get ray directions for all pixels in camera coordinate.
	Reference: https://www.scratchapixel.com/lessons/3d-basic-rendering/
	ray-tracing-generating-camera-rays/standard-coordinate-systems
	Inputs:
	H, W, focal: image height, width and focal length
	Outputs:
	directions: (H, W, 3), the direction of the rays in camera coordinate
	"""
	grid = create_meshgrid(H, W, normalized_coordinates=False)[0]
	i, j = grid.unbind(-1)
	# the direction here is without +0.5 pixel centering as calibration is not so accurate
	# see https://github.com/bmild/nerf/issues/24
	directions = \
	torch.stack([(i-W/2)/focal, -(j-H/2)/focal, -torch.ones_like(i)], -1) # (H, W, 3)
	return directions

When we generate camera rays, the current code assumes "right up back" camera coordinate. Take the upper left (0, 0) pixel for example, its ray will have direction in (-x, +y, -z) under this code.

That means you can omit the coordinate conversion, but if you do so you have to rewrite the get_ray_directions function to correctly generate rays in your coordinate system. Btw, the translation vector has nothing to do here, since it's the camera's position in world coordinate, so it won't change. Remember that we are changing how is the camera rotated w.r.t. the world only.

The author mentioned NDC because this conversion is indispensable for the reasons he stated in that post. 360 scenes do not necessary require this conversion, but having the same conversion leads to same code and more conciseness, so I adopted this conversion for both settings.

[hturki]

Thanks for the clarification. Just to confirm, this repo doesn't actually do the alternate pose conversion for spheric poses detailed in https://github.com/bmild/nerf/blob/55d8b00244d7b5178f4d003526ab6667683c9da9/load_llff.py#L184 (and just calls the same center_poses method for both forward-facing and 360 scenes?)