Programmatic control of height & width of the 3D viewer's viewport & control what camera tracks from blueprint

[javrtg]

Hi! Thank you for this amazing library. I am currently finding it really helpful for my work. I have a feature request:

Having the ability to control both the intrinsics and extrinsics of the 3D viewer can unlock several useful applications. In addition to debugging purposes, users could:

• visit the POV of a camera (requires programmatic control over extrinsics, intrinsics, height and width),
• animate camera trajectories (requires control over extrinsics).

Some visual examples of this feature can be (image data taken from Aachen dataset):

reference image	POV in 3D viewer	camera trajectory

Just to showcase an example of how this can be achieved with other library (although lacking several of rerun's features), the previous visualizations use the open3d viewer. Concretely, they use its methods get_view_control and convert_to_pinhole_camera_parameters.

[Here's a toy example of its use]

import open3d as o3d
import numpy as np

# extrinsics and intrinsics.
pose = np.eye(4)
width, height, fx, fy, cx, cy = 640, 480, 400, 400, 320, 240

# launch viewer.
vis = o3d.visualization.Visualizer()
vis.create_window(height=width, width=height)

# set extrinsics and intrinsics.
cam = vis.get_view_control().convert_to_pinhole_camera_parameters()
cam.extrinsic = pose
cam.intrinsic.set_intrinsics(width, height, fx, fy, cx, cy)
vis.get_view_control().convert_from_pinhole_camera_parameters(cam, allow_arbitrary=True)
vis.run()

I believe it would be great to be able to do this within rerun since it allows easy and fast exploration of specific timestamps.

I think one possible way of doing this without breaking functionalities would be to check whether a pose/intrinsics have been set for certain timestamps.

If so, a checkbox in the GUI could be used to allow arbitrary exploration with the mouse when unchecked, and when checked, snap back the extrinsics and instrincs to the ones specified programmatically by the user. However, I am sure this is something that is much easier said than done.

Related feature request (for extrinsics): #2305

[Wumpf]

@javrtg thanks for filing this issue and even providing an example :)
I think what you're describing is already possible, just not yet programmatically and a bit hidden: How it works right now in Rerun is that you duplicate the 3D space view and in one of them double click the logged camera. That will then track the camera while in the other 3D space view you'd still have manual control over it.
Or maybe I'm missing something? 🤔

[emilk]

Adding to this: you can use rr.log_transform3d to log extrinsics and rr.log_pinhole to log intrinsics. See for instance the objectron example: https://github.com/rerun-io/rerun/blob/0.7.0/examples/python/objectron/main.py#L126-L156
Run it and double-click the camera frustum to track the camera as it moves around.

[javrtg]

@Wumpf @emilk thank you both so much! I didn't know that achieving this manually was possible. This is going to be really useful for me.

It's possible that I may have overlooked it, but with this approach, the only drawback seems to be that the user can't resize the 3D viewer to an exact width and height.

On the other hand, I have observed that a portion of the frustum is projected into the POV, which occludes a small section of the logged geometry. Here is an example:

Is there any way to avoid this?

If you are considering adding programmatic control over the 3D viewer's POV, I will keep the issue open. Otherwise please feel free to close it.

[Wumpf]

Created a separate issue for the frustum lines flickering. Again thanks for putting in the extra effort to add a video snippet! 🤯

Yeah let's keep this ticket open to track to make it possible to do this from the API - this way at least you also get a ping when we got that in :)

[Wumpf]

The exact width & height thing is quite interesting: Personall,y I thought so far only that this is an issue for our 2D views, but you're right we also need to expose the "viewport" for 3D views.

[SummerIcequeen]

Hello, I see this issue is still open. Does it mean rerun still does not support configure the pose of the rendering camera in 3D? If rerun supports this, can someone give me a pointer? IMO this is an important feature if to visualize dynamic large 3D scene in a controlled manner (e.g., visualize radar data from autonomous driving, SLAM etc) and a useful feature to generate render fly through. It seems straightforward to implement. Maybe I'm simplifying a bit, but I imagine the API can expose the scene rendering camera as a Pinhole with a special entity_path, where the pose can be set by Transform3D.

[SummerIcequeen]

I found a walk around by using rr.Transform3D. However it seems inefficient to transform the entire scene geometry rather than the rendering camera. Since I'm fairly new to rerun, maybe I missed something.

[Wumpf]

@SummerIcequeen setting the pose of a camera via Transform3D from code is supported and also what a lot of the examples do, that's the idea of how to do it :). By double clicking a camera in the 3D view you can also track it.

Things that are mentioned in this ticket that aren't supported

• specific width/height on 3D view (both in the Viewer and from Blueprint)
• set a 3D view to track a logged camera from Blueprint (very related to Allow tracking arbitrary entities in spatial space views #4812)

This ticket title is a bit misleading so changing it now.

---

There's also this meta ticket tracking missing features on other things that can't be configured from code (with a whole bunch of camera related stuff):

• [tracking] More configurable view properties #6826