ViewCoordinates are not working properly for camera frustums

[emilk]

A user reported that FLU doesn't work as expected, resulting in camera frustums pointing the wrong way in 3D space.

This is for setting the view coordinates of e.g. world/camera (NOT world/camera/image, which is covered by #1387)

[abey79]

Goal is to make #2512 work with nice and intuitive code :)

[jleibs]

Can we capture more details here regarding what the user was trying to do, what the expected behavior is, and what the actual behavior is with a minimal repro? Very hard to assess whether #2553 actually fixes the behavior without this context.

[jleibs]

Something like this:

import rerun as rr
import numpy as np

rr.init("coord", spawn=True)

img = np.zeros((100, 100, 3))
img[49:50, 10:50] = [255, 0, 0]
img[40:60, 10:12] = [255, 0, 0]
img[10:50, 49:50] = [0, 0, 255]
img[10:12, 40:60] = [0, 0, 255]

rr.log_image("world/image", img)
rr.log_pinhole("world/image", child_from_parent=[[100, 0, 50], [0, 100, 50], [0, 0, 1]], width=100, height=100)
rr.log_view_coordinates("world", xyz="FLU")

Looks totally fine to me. What's the issue?

[emilk]

Something like this:
…
Looks totally fine to me. What's the issue?

The issue is that the forward of the camera is not along X, but along Z. xyz=FLY has no effect on anything:

[emilk]

We have a user that is using a quadcopter to capture images, and the quadcopter uses FLU coordinates, i.e. X is the forward axis, Y is left, and Z is up. The camera has a known resolution and focal length, but no explicit pinhole projection matrix.

It makes sense to expect Rerun to easily support this use case in a way that looking at the camera in Rerun, the frustum is along the X+ axis in the cameras local coordinate system.

There is a strong argument that having anything except Z+ or Z- as the forwards axis is mathematically wrong when you have a standard pinhole matrix where Z is always the projected axis. Correct mathematics matters because we want to be able to help users understand when they are doing something wrong. However, our user above doesn't really want to set a specific pinhole matrix, just point a camera and set a focal length (field-of-view). Currently we don't support setting just the focal length of a pinhole camera, but we should (as an alternative to setting the full 3x3 matrix). I think this is actually the problem: that we force users to set a 3x3 pinhole matrix with Z=forward, when some users don't want Z forward.

I suggest therefore that we merge #2553 - I think it is the best solution proposed so far, especially if we then proceed to work on supporting pinhole transforms without an explicit 3x3 matrix.

We can perhaps come up with better solutions in the future, but I think any solution would need to be able to handle this use case of wanting to see a camera frustum pointing along the local X+ axis. If we say that only Z can be forward, then we need to create helpers for transforming between coordinate systems, and come up with a UI for that. We also need to add some of warning or error if the user sets view coordinates that doesn't not conform to that. I think this is a much bigger design task, and not worth it right now.

[jleibs]

I still pretty strongly disagree that this is a good idea and anticipate a new number of confused users exceeding the number of users that you think that PR will help. But, if you insist, then at a minimum please update the relevant documentation for view coordinates to indicate that it is no longer just adding semantics but will in fact add hidden transforms into the transform chain.

[jleibs]

xyz=FLU has no effect on anything:

Not true, xyz=FLU in that example correctly sets the world coordinates to FLU and does change the default scene layout and eye-camera behavior.

There was no world-to-camera transform logged, so I expect my world-to-camera transform to be Identity. As such, I would expect the camera projection axis (Z) to be pointed up, which is exactly what it's doing. Plenty of roboticists will agree with me on this being totally expected behavior for that example snippet.

This is perhaps more clear using FRD:

rr.log_image("world/image", img)
rr.log_pinhole("world/image", child_from_parent=[[100, 0, 50], [0, 100, 50], [0, 0, 1]], width=100, height=100)
rr.log_view_coordinates("world", xyz="FRD")

In this case I would now expect the optical axis of the camera (Z) to point down under the identity transform, and again, this behaves as expected.

[jleibs]

Not sure how I closed this 😕 bad click on comment?

[emilk]

Your example is quite confusing though, since you hang a pinhole transform in world space, i.e. you have no camera extrinsics.

I 100% agree that the world coordinate system should not affect the camera view coordinates.

This is a more realistic test:

rr.log_image("world/camera/image", img)
rr.log_pinhole("world/camera/image", child_from_parent=[[100, 0, 50], [0, 100, 50], [0, 0, 1]], width=100, height=100)
rr.log_view_coordinates("world/camera", xyz="FLU")

I created a new issue:

• Specify Pinhole without an explicit matrix #2589

Once resolved, this could then become:

rr.log_image("world/camera/image", img)
rr.log_pinhole("world/camera/image", focal_length_px=50, width=100, height=100)
rr.log_view_coordinates("world/camera", xyz="FLU")

There would here be no hidden transform or any confusion. The user states that they want X to be Forward, and so when we visualize the camera frustum we point it along X.

[emilk]

Since we allow putting the extrinsics (Transform) and intrinsics (Pinhole) on the same entity, we should probably use the ViewCoordinates of the pinhole entity to decide the direction in which to point the camera frustum (in contradiction with what I said earlier in this thread).

We should also consider (in the future) to split ViewCoordinates into multiple components doing different things:

• Setting 3D up-axis, and maybe compass directions (X=East)
• Setting the camera view coordinates (frustum orientation)
• Setting the 2D view coordinates of an image (XY=RU, XY=UL etc)

This should significantly reduce the risk of misunderstanding and mistakes

[jleibs]

There would here be no hidden transform or any confusion

(1) Which axes do image-x and image-y map to? Do we negate X when mapping to L? Do we negate Y when mapping to U? What if we specify UFL? Is U always "camera-y"? Plenty of ambiguity and room for confusion.

(2) First I add this to my code because I don't have a calibration and I want to "just see things work". Now I run almost any camera calibration code and it provides me with camera exstrinsics which almost certainly describe a rigid transform between world and the standard "z-forward" image. I log these extrinsics:

rr.log_transform3d("world/camera", my_exstrinsics)

But now my camera is pointing the wrong direction (because yes, there is a hidden transform). How do I make sense of what's happening? How do I know that I actually have to remove view coordinates to use these exstrinsics? What's the correct way to adjust my exstrinsics so the right thing happens when I specify view coordinates?

[emilk]

I'm starting to come down to this as a (short-term) plan:

log_pinhole(…) is given an optional xyz argument for orienting the camera frustum. This is equivalent to setting the view-coordinates for the same entity. For instance: rr.log_pinhole("world/camera/image", focal_length=50, width=100, height=200, xyz="RUB"). This is now the preferred way of setting the frustum orientation. That way the coordinate system of the parent (which could be world) does not affect it at all.

Yes, there is still confusion to be found if a users sets something other than RUB or RDF (e.g. X in image is Z in view-space), but such confusion would be explicitly self-inflicted, and often not a problem (e.g. the common FLU for drones, where you just want to show an image in front of the drone).

In the long run we try to split the view coordinates into different components as outlined in #2244 (comment)