Confusing coordinate system. #153

mcourteaux · 2022-01-21T17:07:03Z

mcourteaux
Jan 21, 2022

I'm trying to wrap my head around the coordinate system, and it looks unlike anything I've seen before. I setup a simple test transforms.json:

{
    "camera_angle_x": 2.1273956954128375,
    "scale": 1.0,
    "offset": [
        0.0,
        0.0,
        0.0
    ],
    "frames": [
        {
            "file_path": "png/view_f0000f",
            "transform_matrix": [
                [
                    1.0,
                    0.0,
                    0.0,
                    0.8
                ],
                [
                    0.0,
                    1.0,
                    0.0,
                    1.0
                ],
                [
                    0.0,
                    0.0,
                    1.0,
                    1.2
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ]
        }
    ]
}

Notice how the camera I set up is having identity rotational part, and translation vector [0.8, 1.0, 1.2]. The result looks like this:

Confusing things:

The camera and the unit cube have different orientations for X, Y, and Z (i.e.: red, green, blue axis). (see first image)
The position of the camera relative to the unit cube seems wrong. The the camera seems to be at x=1, and z=0.8, whereas I specified x to be 0.8, and z to be 1.2. (see second image).
The camera is at y=1.2, whereas it should be 1.0. (see third image).

So, position in the testbed is (1, 1.2, 0.8), but it should be (0.8, 1.0, 1.2). This shuffles around ALL axis. After long searching, I found this snippet:

instant-ngp/include/neural-graphics-primitives/nerf_loader.h

Lines 50 to 70 in 409613a

    
           auto nerf_matrix_to_ngp(const Eigen::Matrix<float, 3, 4>& nerf_matrix) { 
        
           	Eigen::Matrix<float, 3, 4> result; 
        
           	int X=0,Y=1,Z=2; 
        
           	result.col(0) = Eigen::Vector3f{ nerf_matrix(X,0),  nerf_matrix(Y,0),  nerf_matrix(Z,0)}; 
        
           	result.col(1) = Eigen::Vector3f{-nerf_matrix(X,1), -nerf_matrix(Y,1), -nerf_matrix(Z,1)}; 
        
           	result.col(2) = Eigen::Vector3f{-nerf_matrix(X,2), -nerf_matrix(Y,2), -nerf_matrix(Z,2)}; 
        
           	result.col(3) = Eigen::Vector3f{ nerf_matrix(X,3),  nerf_matrix(Y,3),  nerf_matrix(Z,3)} * scale + offset; 
        
           	if (from_mitsuba) { 
        
           		result.col(0) *= -1; 
        
           		result.col(2) *= -1; 
        
           	} else { 
        
           		// Cycle axes xyz->yzx 
        
           		Eigen::Vector4f tmp = result.row(0); 
        
           		result.row(0) = (Eigen::Vector4f)result.row(1); 
        
           		result.row(1) = (Eigen::Vector4f)result.row(2); 
        
           		result.row(2) = tmp; 
        
           	} 
        
           	return result; 
        
           }

So, my question kinda reduces to: how to think about this coordinate system, as I still haven't figured out how to convert my dataset transformations to yours.

mmalex · 2022-01-21T17:30:42Z

mmalex
Jan 21, 2022

sorry for the shift in coordinate system! internally, ngp uses entirely 0-1 bounding box, as displayed in the gui, with cameras looking down positive z. that's just the convention we chose early on. however, we wanted to be compatible specifically with the original nerf datasets, which place the origin at 0, scale the cameras to be around 3 units from the origin, and have a different convention for 'up', camera up, etc. so the snippet you found could be described as 'converting from the original nerf paper conventions to ngp's conventions'. over time, the 'original nerf' format of transforms.json became the dominant (only practical) way to get data into ngp; so, it comes across as confusing as we don't explicitly talk about the mapping. In the end, I am afraid it is what it is. you can adjust the scale and offset components of the transformation by adding extra parameters to the json, see nerf_loader.cu that looks for keys 'offset', 'scale' and 'aabb'. however the coordinate flipping is hard coded for the time being.

…

On Fri, Jan 21, 2022 at 5:07 PM Martijn Courteaux ***@***.***> wrote: I'm trying to wrap my head around the coordinate system, and it looks unlike anything I've seen before. I setup a simple test transforms.json: { "camera_angle_x": 2.1273956954128375, "scale": 1.0, "offset": [ 0.0, 0.0, 0.0 ], "frames": [ { "file_path": "png/view_f0000f", "transform_matrix": [ [ 1.0, 0.0, 0.0, 0.8 ], [ 0.0, 1.0, 0.0, 1.0 ], [ 0.0, 0.0, 1.0, 1.2 ], [ 0.0, 0.0, 0.0, 1.0 ] ] } ] } Notice how the camera I set up is having identity rotational part, and translation vector [0.8, 1.0, 1.2]. The result looks like this: [image: image] <https://user-images.githubusercontent.com/845012/150566804-e932cdc4-b977-4a0c-9d4f-3ecaae490d93.png> [image: image] <https://user-images.githubusercontent.com/845012/150567363-a5ecf787-111a-46ca-9db7-1b499c115acd.png> [image: image] <https://user-images.githubusercontent.com/845012/150567796-e4dec845-4aea-463e-b456-751db1adf200.png> Confusing things: - The camera and the unit cube have different orientations for X, Y, and Z (i.e.: red, green, blue axis). (see first image) - The position of the camera relative to the unit cube seems wrong. The the camera seems to be at x=1, and z=0.8, whereas I specified x to be 0.8, and z to be 1.2. (see second image). - The camera is at y=1.2, whereas it should be 1.0. (see third image). So, position in the testbed is (1, 1.2, 0.8), but it should be (0.8, 1.0, 1.2). This shuffles around ALL axis. After long searching, I found this snippet: https://github.com/NVlabs/instant-ngp/blob/409613afdc08f69342a9269b9e674604229d183f/include/neural-graphics-primitives/nerf_loader.h#L50-L70 So, my question kinda reduces to: how to think about this coordinate system, as I still haven't figured out how to convert my dataset transformations to yours. — Reply to this email directly, view it on GitHub <#72>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOYTZYYMZ3QX7H22NUVDTUXGHEHANCNFSM5MQA7I5Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

0 replies

mcourteaux · 2022-01-21T17:54:09Z

mcourteaux
Jan 21, 2022
Author

Looking at the fox dataset, it seems that the -y is the camera up axis. I'm still puzzling, haha...

(a few minutes later...)

This was key!! The camera orientation was flipped (y-down). Adding that 180 degree rotation upon converting makes it work.

Now, still rotate everything into the normal upward rotation in your coordinate system.

I looked through the code, and my intuition is that the up setting doesn't do anything for nerf.

1 reply

Yanall-Boutros Jul 2, 2022

i remember feeling infuriated when i found out unreal engine uses a left handed coordinate system.

mcourteaux · 2022-01-25T14:19:00Z

mcourteaux
Jan 25, 2022
Author

For reference for others. This is what I have right now, based on a position and euler rotations for a Blender-like camera:

def generate_transform_matrix(pos, rot):
    def Rx(theta):
      return np.matrix([[ 1, 0            , 0            ],
                        [ 0, np.cos(theta),-np.sin(theta)],
                        [ 0, np.sin(theta), np.cos(theta)]])
    def Ry(theta):
      return np.matrix([[ np.cos(theta), 0, np.sin(theta)],
                        [ 0            , 1, 0            ],
                        [-np.sin(theta), 0, np.cos(theta)]])
    def Rz(theta):
      return np.matrix([[ np.cos(theta), -np.sin(theta), 0 ],
                        [ np.sin(theta), np.cos(theta) , 0 ],
                        [ 0            , 0             , 1 ]])

    R = Rz(rot[2]) * Ry(rot[1]) * Rx(rot[0])
    xf_rot = np.eye(4)
    xf_rot[:3,:3] = R

    xf_pos = np.eye(4)
    xf_pos[:3,3] = pos - average_position

    # barbershop_mirros_hd_dense:
    # - camera plane is y+z plane, meaning: constant x-values
    # - cameras look to +x

    # Don't ask me...
    extra_xf = np.matrix([
        [-1, 0, 0, 0],
        [ 0, 0, 1, 0],
        [ 0, 1, 0, 0],
        [ 0, 0, 0, 1]])
    # NerF will cycle forward, so lets cycle backward.
    shift_coords = np.matrix([
        [0, 0, 1, 0],
        [1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 0, 1]])
    xf = shift_coords @ extra_xf @ xf_pos
    assert np.abs(np.linalg.det(xf) - 1.0) < 1e-4
    xf = xf @ xf_rot
    return xf

5 replies

FlyFish-space May 19, 2022

What is the relationship between the coordinate axis before the conversion and the coordinate axis after your conversion?
(x,y,z -> y -x z ?)

mcourteaux May 25, 2022
Author

Well, I quickly ran the code, and the combined matrix that transforms the axis is this one:

M = 
[[ 0  1  0  0]
 [-1  0  0  0]
 [ 0  0  1  0]
 [ 0  0  0  1]]

When using this as: ngp = M * blender, I read this as: (x, y, z)_ngp = (y, -x, z)_blender. So yes, what you had seems about right. Note that just applying this axis reshuffling doesn't suffice, as it also rotates.

mcourteaux Sep 13, 2022
Author

I think this transforms too much. It seems to rotate 180° around the up (y-axis in NeRF) too much. I'm getting upset again.

AndresPiola Apr 6, 2023

average_position

wha is average_position?

mcourteaux Apr 7, 2023
Author

average_position
wha is average_position?

For me that was the mean position of all cameras, which I used to subtract from all camera positions to get the scene to be close to the center of the bounding volume of the NeRF.

mcourteaux · 2022-01-25T15:06:14Z

mcourteaux
Jan 25, 2022
Author

Absolutely beautiful!!

nerf_barbershop_spherical_2.small.mp4

21 replies

mcourteaux Apr 1, 2022
Author

For NeRF, you need to put the Minimum Distance to zero in the nerf training options. Once you do that, you might want to reset training.

qhdqhd Apr 1, 2022

Thank you, may I trouble you to elaborate a little bit? How is Minimum Distance put to zero?

mcourteaux Apr 1, 2022
Author

It's the "Near Distance", under the "NeRF Training Options":
https://github.com/NVlabs/instant-ngp/blob/master/src/testbed.cu#L399

qhdqhd Apr 2, 2022

Thanks a lot for your advice, it's very useful. I got the same result as you.

But I still found some problems:

The 256x256 image looks good, but it does lack some texture detail.

Does it mean that a larger number of small resolution images and a small number of large resolution images can achieve the same effect? I wonder why are you using such a small resolution?
2. The cameras are distributed on the surface of a sphere or cube. When the virtual viewpoint moves within the sphere or cube, the rendering result is good, but when the virtual viewpoint deviates from the training viewpoint, the rendering result becomes worse.

I would like to ask, why distribute the cameras on a sphere or a cube? Evenly distributing multiple viewpoints in the scene should give a freer virtual viewpoint.

mcourteaux Apr 2, 2022
Author

Amazing! I'm glad other people managed to recreate the results!

I chose those small textures as a quick test to not have to select a subset of images from the full dataset. But yes, generally, due to the aliasing, the results can be really good from very small aliased pictures, but a lot of them.
My field of research is light fields where data comes from a more regularly spaced camera setup. In the paper, I elaborate in greater detail what exactly I'm looking for that I provide in this dataset.

mmalex · 2022-01-25T15:21:39Z

mmalex
Jan 25, 2022

thankyou for this! I am tempted to make a pullrequest for you that disables as much of the coordinate transformation as possible - dont worry the default wont change so your dataset will continue to work - I feel guilty about the trouble we caused :)

0 replies

mcourteaux · 2022-01-25T15:36:59Z

mcourteaux
Jan 25, 2022
Author

I think I wouldn't be the only one that would appreciate a non-weirdly-behaving coordinate system 😋 I must admit, I got to this point by brute forcing transformation matrices and a little bit of iterative educated guessing. I'm confused as the camera upside-down thing we discussed earlier is not clearly in my transformation code anymore.

https://twitter.com/ID_AA_Carmack/status/432298265254449152

0 replies

jc211 · 2022-01-27T02:51:38Z

jc211
Jan 27, 2022

To remove transform nastiness from the nerf code, the only areas I found that need to be changed are in nerf_loader.h

Eigen::Vector3f nerf_direction_to_ngp(const Eigen::Vector3f& nerf_dir) {
  Eigen::Vector3f result = nerf_dir;
  return result;
}

Eigen::Matrix<float, 3, 4> nerf_matrix_to_ngp(const Eigen::Matrix<float, 3, 4>& nerf_matrix) {
  Eigen::Matrix<float, 3, 4> result = nerf_matrix;
  result.col(3) = result.col(3) * scale + offset;
  return result;
}

Eigen::Matrix<float, 3, 4> ngp_matrix_to_nerf(const Eigen::Matrix<float, 3, 4>& ngp_matrix) {
  Eigen::Matrix<float, 3, 4> result = ngp_matrix;
  result.col(3) = (result.col(3) - offset) / scale;
  return result;
}

void nerf_ray_to_ngp(Ray& ray, bool scale_direction = false) {
  ray.o = ray.o * scale + offset;
  if (scale_direction)
	  ray.d *= scale;
}

If you change this and compile, the transforms are expected to be in the opencv convention (and in this case the ngp convention) which is

Otherwise, the cameras in transforms.json are actually already in the blender convention (camera looking down -z and up is +y). You just have to make sure the intrinsic matrix is all positive.

0 replies

milliondabeat · 2022-06-30T18:27:48Z

milliondabeat
Jun 30, 2022

hey does anyone know how i could import a camera i tracked from a 3d software like blender into instant ngp as a camera path?

2 replies

mcourteaux Sep 14, 2022
Author

You generate a path JSON file. To see an example, just play with the path editor in the testbed; export the path, and look at the file format. It's simple, but you'll need quaternions. Note that this does NOT do all of this transformation garbage; this just uses regular coordinates like how the internal engine works. Just keep in mind that positive-Y axis of the camera points down, just like @jc211 mentioned above.

allAxisM May 3, 2023

'To see an example, just play with the path editor in the testbed; export the path'.

Any chance you could explain how to export the path? I can't seem to work this out. Also, how would you import a JSON file with a camera path?

Thanks :)

TheLordSaiyan · 2023-03-15T18:07:34Z

TheLordSaiyan
Mar 15, 2023

I think you guys could help me with my issue, I can't find the problem. The coordinates of the camera are off. #1286

0 replies

SergioMOrozco · 2023-05-25T18:34:45Z

SergioMOrozco
May 25, 2023

Hello, I am dealing with an issue that I feel is related. If anybody could help me, I would greatly appreciate it!

You can find my issue here #1360

0 replies

WillowProduction · 2023-11-15T11:49:35Z

WillowProduction
Nov 15, 2023

Hello every one,
Is there anyone who tried to import the saved preset camera path into Blender? the JSON file is really simple and works with kwyframes, but the interpolation inside ngp is so strange. Can't model this behaviour inside Blender

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Confusing coordinate system. #153

{{title}}

Replies: 11 comments 29 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Confusing coordinate system. #153

Replies: 11 comments · 29 replies

mcourteaux Jan 21, 2022 Author

mcourteaux Jan 25, 2022 Author

mcourteaux May 25, 2022 Author

mcourteaux Sep 13, 2022 Author

mcourteaux Apr 7, 2023 Author

mcourteaux Jan 25, 2022 Author

mcourteaux Apr 1, 2022 Author

mcourteaux Apr 1, 2022 Author

mcourteaux Apr 2, 2022 Author

mcourteaux Jan 25, 2022 Author

mcourteaux Sep 14, 2022 Author

Replies: 11 comments 29 replies

mcourteaux
Jan 21, 2022
Author

mcourteaux
Jan 25, 2022
Author

mcourteaux May 25, 2022
Author

mcourteaux Sep 13, 2022
Author

mcourteaux Apr 7, 2023
Author

mcourteaux
Jan 25, 2022
Author

mcourteaux Apr 1, 2022
Author

mcourteaux Apr 1, 2022
Author

mcourteaux Apr 2, 2022
Author

mcourteaux
Jan 25, 2022
Author

mcourteaux Sep 14, 2022
Author