Vertical camera - rotate depth frames?

[keithahern]

| Camera Model | {D400 |
| Firmware Version | V2.31.0 |
| Operating System & Version | Linux / MacOS |
| Kernel Version (Linux Only) | (e.g. 4.14.13) |
| Platform | PC |
| SDK Version | { 2.X } |
| Language | {python} |
| Segment | {/others } |

Issue Description

I have a developed application in python and some tests indicate the performance of the camera would be better if rotated 90 degrees. It would be extremely time consuming (and confusing!) to rewrite and change all the code which assumes x is horizontal and y is vertical. We use a lot of librealsense API calls. Is there an easer to to rotate the camera and still have x, y behave correctly? e.g. in 480x640 resolution?

[MartyG-RealSense]

I believe that you wish to rotate the camera 90 degrees but rotate the image 90 degrees in the opposite direction so that it appears in its normal orientation, please?

[keithahern]

no, thats not correct.

I want to turn the camera 90 degrees and still have X axis be horizontal, Y is vertical. So, for example in landscape the camera is 640x480 and in portrait it is 480x640.

If the camera is NOT rotated the follow code is correct
dist = depth_frame.get_distance(x, y)

if the camera IS rotated then the code needs to be
dist = depth_frame.get_distance(y, x)

which will get confusing very quickly, every x must swapped for y.

A solution could be done at the frame level
depth_frame = depth_frame.rotate(rs2.90_DEGREES_CLOCKWISE)
dist = depth_frame.get_distance(x, y)

but would probably be better at the device level so the color and IR frames also do the 'right thing'
config = rs.config()
config.device_orientation(rs2.90_DEGREES_CLOCKWISE)
config.enable_stream(rs.stream.depth, 480, 640, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 480, 640, rs.format.bgr8, 30)

Or something like that. Does that help?

[keithahern]

I am measuring a tall vertical groove at close range. When the camera is horizontal the cameras can not see deeply into the groove. When the camera is vertical it can.

…

On 11 Mar 2020, at 12:36, MartyG-RealSense ***@***.***> wrote: I considered your question carefully. It sounds as though it would be much easier to keep the camera in its normal orientation and seek to improve the output results by other means. Could you tell me please what problems you have with the camera in its normal orientation that seem to be better when it is rotated 90 degrees please? Thanks very much! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMO7CS5R4KDC2VKWEUANLRG6AWTANCNFSM4LFTPTIA>.

[dorodnic]

Hi @keithahern
We have such capability on the upcoming L500 camera, and we may be able to port this to D400 cameras in one of the future releases. I agree it's not as easy as rotating the image, all calibration data and other parameters need to be updated, and we do want to enable it in the SDK at some point

[keithahern]

Great. Thank you Sergey and Marty.

…

On 11 Mar 2020, at 12:55, Sergey Dorodnicov ***@***.***> wrote: Hi @keithahern <https://github.com/keithahern> We have such capability on the upcoming L500 camera, and we may be able to port this to D400 cameras in one of the future releases. I agree it's not as easy as rotating the image, all calibration data and other parameters need to be updated, and we do want to enable it in the SDK at some point — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMO7HC7VI45JNXKZ7TR5DRG6C5BANCNFSM4LFTPTIA>.

[RMichaelPickering]

@dorodnic @MartyG-RealSense I would like to upvote this request, as we would like to use it in our application, which involves detecting human faces. The point is that having the largest field of view in the vertical (i.e. a portrait orientation rather than landscape) helps to allow for a wider range of subject heights, potentially including people sitting in wheelchairs, etc. It would be most helpful to have the requested feature for our application, too!!

[MartyG-RealSense]

@RMichaelPickering Thanks so much for your supportive input!

You could also increase the vertical field of view by having two cameras stacked vertically with their fields of view overlapping, and this should avoid the image rotation implications.

[Kevin-Delnoije]

@MartyG-RealSense @dorodnic any update on this? a vertical orientation parameter or a function that can rotate realsense frames would be very helpfull!

[keithahern]

My solution has been to just use RealSense for initial depth and color frame acquisition and then move to the (also Intel backed) Open3D library and do my processing from there (i.e. I'm not using any RealSense projection/deprojection APIs. In Open3d it's much easier to merge and manipulate point clouds, render to 2D, create cross platform UIs etc. The documentation is lacking in places but it's allowed me to move forward.

See below on how to get an rs frameset into an open3d PointCloud and transform it - its very fast.

`
depth_frame = frameset.get_depth_frame()
color_frame = frameset.get_color_frame()

depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())

img_depth = o3d.geometry.Image(depth_image)
img_color = o3d.geometry.Image(color_image)

img_rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(img_color,img_depth)
intrinsics = profile.as_video_stream_profile().get_intrinsics()
pinhole_camera_intrinsic = o3d.camera.PinholeCameraIntrinsic(intrinsics.width, intrinsics.height, intrinsics.fx, intrinsics.fy, intrinsics.ppx, intrinsics.ppy)

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(img_rgbd, pinhole_camera_intrinsic)
pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])

`

[Kevin-Delnoije]

My solution has been to just used RealSense for initial depth and color frame acquisition and then move to the (also Intel backed) Open3D library and do my processing from there (i.e. I'm not using any RealSense projection/deprojection APIs. In Open3d it's much easier to merge and manipulate point clouds, render to 2D, create cross platform UIs etc. The documentation is lacking in places but it's allowed me to move forward.

See below on how to get an rs frameset into an open3d PointCloud and transform it - its very fast.

`
depth_frame = frameset.get_depth_frame()
color_frame = frameset.get_color_frame()

depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())

img_depth = o3d.geometry.Image(depth_image)
img_color = o3d.geometry.Image(color_image)

img_rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(img_color,img_depth)
intrinsics = profile.as_video_stream_profile().get_intrinsics()
pinhole_camera_intrinsic = o3d.camera.PinholeCameraIntrinsic(intrinsics.width, intrinsics.height, intrinsics.fx, intrinsics.fy, intrinsics.ppx, intrinsics.ppy)

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(img_rgbd, pinhole_camera_intrinsic)
pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])

`

Thanks this solves the problem!

[qualityland27]

Hi @keithahern We have such capability on the upcoming L500 camera, and we may be able to port this to D400 cameras in one of the future releases. I agree it's not as easy as rotating the image, all calibration data and other parameters need to be updated, and we do want to enable it in the SDK at some point

@dorodnic I am working with the L515. Could you kindly tell me how to do it in Python? I cant find it ...

[RMichaelPickering]

FWIW, we've implemented some frame rotation code using OpenCV but it would of course be far better to do as much of this as possible in the camera! If the capability from the L500 could be ported to the D4xx cameras and fully supported in the SDK that would be excellent!! Mike R. Michael Pickering CBIP President & CEO CloudConstable Incorporated 10271 Yonge Street, Suite 321 Richmond Hill, ON L4C 3B5 Canada www.cloudconstable.com ***@***.*** Toronto: 416-721-1826 Orlando: 321-337-2824

…

On Fri, 3 Dec 2021 at 08:45, Qualityland ***@***.***> wrote: Hi @keithahern <https://github.com/keithahern> We have such capability on the upcoming L500 camera, and we may be able to port this to D400 cameras in one of the future releases. I agree it's not as easy as rotating the image, all calibration data and other parameters need to be updated, and we do want to enable it in the SDK at some point @dorodnic <https://github.com/dorodnic> I am working with the L515. Could you kindly tell me how to do it in Python? I cant find it ... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEABETJN34AXB2B37JF6QI3UPDCX7ANCNFSM4LFTPTIA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[FeiSong123]

Hi, I'm using L515 and I also need to rotate camera 180 degree for my project. This is very helpful to me, so can you tell me more about the usage ?

[MartyG-RealSense]

Hi @FeiSong123 If you rotate the L515 camera 180 degrees in the Y direction then although the image would be upside down, you would not need to make any changes to the image as its values should still be correct

If you needed to rotate / flip the image after rotating the camera 180 degrees so that the image was the 'right way up' then you could do that using the methods in this discussion or at #4395 (comment) and #9088

[FeiSong123]

I also prefer to rotate at the camera level, and I see dorodinc commented that L515 has such capability. What should I do with C++(I can't find this function device_orientation with config).

[MartyG-RealSense]

The comment was made before the L515 was first released on sale, and not every planned feature is released for various reasons (for example, if the feature did not work well). As far as I am aware there is not a dedicated image rotation feature for the L515. I apologize for the inconvenience.

[FeiSong123]

That's a pity. Nevertheless, I still want to thank you for your guidance.

[maciejandrzejewski-digica]

@dorodnic @MartyG-RealSense According to what @dorodnic has written: "I agree it's not as easy as rotating the image, all calibration data and other parameters need to be updated, and we do want to enable it in the SDK at some point".

I would like to know if we can expect same depth/image accuracy, quality, performance, reponse, etc. if the camera is rotated 90* left/right? For some reason calibration data is mentioned. So rotating those frames manually using some other software like OpenCV is to be considered sub-optimal because camera is not positioned up-right?

[MartyG-RealSense]

Hi @maciejandrzejewski-digica There are no performance issues with physically rotating only the camera 90 degrees. There would be a problem with calibration values if the frame was then rotated with software such as OpenCV to make the image display in the normal orientation.

[maciejandrzejewski-digica]

@MartyG-RealSense In the project we would like to rotate the camera to better suite the position in the device case. But for AI inference we need to rotate the images back to the up-right position. Can you elaborate more what does it mean "problem with calibration values" how that affects depth accuracy?

[MartyG-RealSense]

Depending on the maximum depth range required, the RealSense D405 model (or its D401 PCB board version) may be an easy fit for a device case as it is a small square rather than wide horizontal shape. Its ideal depth range is 0.7 cm to 50 cm, meaning that it is suited to very close range depth sensing applications rather than multiple-meter distances.

https://store.intelrealsense.com/buy-intel-realsense-depth-camera-d405.html

If a longer-distance camera model is required, a RealSense user earlier in this discussion at #6023 (comment) shared a software workaround for frame rotation.

There is unfortunately not any further information available about the implications to camera parameters of frame rotation other than dorodnic's statement at #6023 (comment)

[Nir-Az]

Hey guys,
Checkout PR #13499 created by @noacoohen.
We think this fulfill the need.

Note: This rotation is CPU implementation based.

[MartyG-RealSense]

Hi everyone, have any of you been able to test the new rotation post-processing filter implemented on the development branch of librealsense, please? Thanks!

https://github.com/IntelRealSense/librealsense/tree/development

[mewescott]

I would like to test it but I haven't figured out how it even works. Do you know how to use it? My install is using the ROS 2 wrapper which he said it is not supported yet but I would like to know how to use it without the wrapper too. Maybe I can figure out a way to make it work that way.

[Nir-Az]

I would like to test it but I haven't figured out how it even works. Do you know how to use it? My install is using the ROS 2 wrapper which he said it is not supported yet but I would like to know how to use it without the wrapper too. Maybe I can figure out a way to make it work that way.

You can use this code reference as a usage example:

https://github.com/IntelRealSense/librealsense/blob/development/examples/post-processing/rs-post-processing.cpp

[Nir-Az]

Thanks to @noacoohen now also available on RS ROS Node.
See PR #3274

[mewescott]

Thanks for the heads up. Where can I find instructions for install/use? Which repo branch of realsense library and wrapper do I need?

[Nir-Az]

Please build from source librealsense development branch: https://github.com/IntelRealSense/librealsense/tree/development
and ros wrapper ros2-development branch: https://github.com/IntelRealSense/realsense-ros/tree/ros2-development

Both have install instructions when building from source

[mewescott]

Thank you. Can you tell me how to use the rotate filter in ROS after installation. I just installed the development branch from both on Thursday. Do I need to do it again or is there a way to just update them?

[Nir-Az]

PR 3274 was merged on 12/12
Depends when you took it.
But you can git pull your forks to latest and it will be OK

[mewescott]

I went through the instructions in both repo's and I can't figure out how to use the rotation filter with ROS. Where can I find instructions? I need to turn my images 90 degrees as my cameras are mounted narrow side down. I plan to use Aruco markers and human detection so I would like to get the orientation of the image right first.

…

On Sun, Dec 15, 2024 at 1:58 PM Nir Azkiel ***@***.***> wrote: PR 3274 <IntelRealSense/realsense-ros#3274> was merged on 12/12 Depends when you took it. But you can git pull your forks to latest and it will be OK — Reply to this email directly, view it on GitHub <#6023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPBK3B2S3KILDAOZ2ZQT3T2FXI2HAVCNFSM6AAAAABR7OLXCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBUGAYTAOJUHE> . You are receiving this because you commented.Message ID: ***@***.***>

[noacoohen]

You can find instructions for using the rotation filter with ROS in this launch file:
rs_launch.py on the ros2-development branch

To enable and configure the rotation filter, you can use the following parameters:

1. rotation_filter.enable:

   • Default: false
   • Enable or disable the rotation filter.

2. rotation_filter.rotation:

   • Default: 0.0
   • Set the rotation angle. Supported values are 0.0, 90.0, -90.0, and 180.0.

You can configure these parameters to rotate the images as needed for your setup.

[mewescott]

I just tested this and I am only seeing rotation in the depth image. The color image does not change. Is it supposed to rotate both? Here is how I launched it:

ros2 launch realsense2_camera rs_launch.py rotation_filter.enable:=true rotation_filter.rotation:=90.0

[Nir-Az]

I just tested this and I am only seeing rotation in the depth image. The color image does not change. Is it supposed to rotate both? Here is how I launched it:

ros2 launch realsense2_camera rs_launch.py rotation_filter.enable:=true rotation_filter.rotation:=90.0

Hey @mewescott ,
Glad to see you managed to get the depth frames rotated!
Correct, original request and this thread title is Vertical camera - rotate depth frames? and that's what we implemented at phase 1.
Color stream rotation is more complex as the SDK support multiple color formats (YUYV, RGB, M420,...).
We will open another enhancement request for the color stream.

Stay tuned :)