Announcement: RealSense D457 GMSL/FAKRA stereo depth camera now in stock at the RealSense Store

[MartyG-RealSense]

Hi everyone,

Intel's new RealSense D457 stereo depth camera model is now listed as in stock at the official online RealSense Store.

https://store.intelrealsense.com/buy-intel-realsense-depth-camera-d457.html

A range of information resources about this camera can be found at #10984

Please note that the D457 model is designed for use with a GMSL/FAKRA connection and cannot be used as a "plug and play" USB device.

[sam598]

@MartyG-RealSense Is this camera capable of synchronized color and depth exposures? Or is it the same as all of the other RealSense cameras where "RGB and depth frames have a temporal offset bounded by a period of one frame"?

If it is the latter, it is very surprising that Intel would sell another camera without fixing this critical issue for 3D scanning. It cannot be understated how much this issue has prevented the success of RealSense hardware over the years.

[MartyG-RealSense]

@sam598 I will discuss your D457 sync question with my Intel RealSense colleagues. Thank you.

[MartyG-RealSense]

Hi again @sam598 My colleagues informed me that the depth and RGB are synced in D457 as they are on the same PDB, unlike D435.

They also emphasize that genlock sync was an experimental and unvalidated proof of concept. The mature and validated hardware sync system is the original Inter Cam Sync Mode 1 and 2, which is depth only.

https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration

[sam598]

Hi @MartyG-RealSense , to be absolutely clear I am not talking about Genlock. I am talking about color and depth synchronization on a single camera.

With your comment here: #10850 (comment) and this discussion here: #10926 (comment) it is clear that the other D400 cameras are not capable of Color and Depth sync.

"RGB and depth frames have a temporal offset bounded by a period of one frame" is not exposure synchronization. Any camera pair is capable of temporal alignment. For 3D scanning it is a requirement that the color and depth images be exposed at the exact same moment in time.

Is this a feature that has been added to the D457 that was not present in the D455?

[MartyG-RealSense]

The genlock reference in the provided response is because I highlighted your interest in genlock sync to my colleagues whilst discussing your D457 query.

I will seek clarification about whether the D457 advice relates to depth and RGB on the same camera.

[MartyG-RealSense]

Hi @sam598 My colleagues confirmed that it is referring to sync between the depth and RGB sensors on the same D457 camera.
RGB and depth are on the same PCB.

[sam598]

Thanks. Before I purchase dozens more of these cameras I need to ask that you can verify these two statements are true:

1. The D415, D435, D435i, and D455 cameras are not capable of color and depth sync because the cameras are printed on separate PCB. Instead the SDK matches frames based on their hardware time stamps, and there is a temporal offset of one frame or less.

2. The D457 camera has its color and depth cameras printed on the same PCB, and is able to synchronously capture color and depth frames at the exact same moment in time. There is no offset in the hardware time stamps, because they are being internally triggered at the same time.

[sam598]

I don’t understand how the D455 and D457 cameras can both use the same D450 camera module and not both suffer the same issue.

[MartyG-RealSense]

I have referred your latest questions to my colleagues. Thanks again for your patience.

[sam598]

@MartyG-RealSense I may have found an answer.

In the D457 datasheet on section 7.13:
Intel® RealSense™ D400 Series supports hardware sync signal for multi-camera configuration. Multiple cameras are hardware synchronized as to capture at identical times and frame rates. For multiple D457s, all of them can work in slave mode; the external trigger is applied to the camera through the GMSL cable from the host side. There are two triggers, one for depth camera and one for RGB camera. The depth and RGB cameras can be triggered at both the same and different supported frame rates.

In the D400 family datasheet on section 7.13:
Intel® RealSense™ D400 Series supports a hardware sync signal for multi-camera configuration. For multiple cameras to be hardware synchronized so as to capture data at identical times and frame rates, pins 5 (SYNC) and pins 9 (Ground) on the external sensor sync connector will need to be connected. The external sensor sync connector is located on the Vision Processor D4 V1 and V3 board and is accessible on the Depth Cameras. D405 does not support the hardware sync signal for multi-camera configuration.

It does indeed seem like this is a new feature that has been added to the D457. I have additional question, mainly which pin on the D457 connector can be used to trigger the color camera. I can open up a new issue if that makes sense instead?

[MartyG-RealSense]

Thanks very much @sam598 for the update. Let's stay on this issue. I have added your third question to my communication to my colleagues.

[sam598]

Any update on which pin can be used to trigger RGB?

[MartyG-RealSense]

I will follow up with my colleagues regarding this question.

[MartyG-RealSense]

I received a response from my colleagues about your question about which pin on the D457 connector can be used to trigger the color camera. The separate triggers referred to in your D457 data sheet quote are from the GMSL cable, not the 9-pin connector. This may therefore make the feature irrelevant for your needs if you plan to make use of the 9-pin connector.

Although the D457 performs like a D455 under USB, the D457 is designed for GMSL applications and so the information below focuses on GMSL.

D457 Depth and RGB have their own trigger pin, so they can be triggered independently via two different trigger sources. However, those trigger pins are embedded in the GMSL link (not on the 9-pin AUX connector), so users need to apply the trigger on host side via Deserializer(Des) (MAX9296A) which is the pairing chip of the Serializer(Ser) (MAX9295A) built inside of D457.

On D457 the Serializer uses MFP1 for the Depth trigger and MFP0 for the RGB trigger (MFP means the GPIO of the Serializer).

When the D457 is operating in either the USB mode or GMSL mode, the IR and RGB are not using the same ISP, so their auto exposure control are not the same. As a result, the exposure time could be different even when they’re synced. However, while being synced under the same frame rate, the “End of Exposure” time is same but the “Start of Exposure” time varies. If users want the same exposure time, they have to use fixed exposure time for both IR and RGB.

Below is the typical timing chart when Depth and RGB are synced.

[khuongkhoa1999]

Hi,
Our product in development has been currently using D415/D435 for a while now and we would like to have a camera with an IP rating and secure connectors since ours is an outdoor agricultural application.
Everything has been written with Realsense SDK. GMSL/FAKRA is a new concept for us but we hope to integrate the camera quickly.
Can we keep using RealSense SDK with the GMSL interface? Is there any guide for the GMSL interface? We are currently using Jetson NX as our host platform but that may change in the future.

[MartyG-RealSense]

Hi @khuongkhoa1999 Yes, you can continue to use the SDK. There is a pre-release test version of the RealSense SDK (2.52.1) with GMSL support.

https://github.com/IntelRealSense/librealsense/releases/tag/v2.52.1

More information about cabling is can be found at #10984 (comment) and the comments beneath it. Because the camera uses the GMSL protocol, any GMSL cable can work with it.

In addition to the D457 camera, you will need a Maxim Deserializer component on the host side to decode the streams. You could initially test the D457 quickly without needing a deserializer though by using it with Connect Tech's products, and then later switch to using the deserializer component when you need to finally integrate the D457 into your outdoor weatherproofed system.

https://connecttech.com/connect-tech-offers-built-in-support-on-the-newly-announced-intel-gmsl-realsense-depth-camera-d457/

The data sheet document for D457 provides the following summary descriptions about GMSL and FAKRA.

---

GMSL - Gigabit Multimedia Serial Link. GMSL serializer and de-serializer (SerDes) are high-speed communication ICs that fully support the high bandwidth, complex interconnect, and data integrity requirements needed to support evolving automotive infotainment and advanced driver assistance systems (ADAS).

FAKRA (Fachkreis Automobil, a German standard) connector is a modified SMB connector with a keyed and color-coded plastic housing and latch, used in the automotive industry. They are available in 14 different mechanical layouts which are color coded for easy identification and mis-matching prevention.
D457 has a C-Coding (blue) connector.

[khuongkhoa1999]

Thanks @MartyG-RealSense

I see that you have added MIPI support, can I just buy an off-the-shelf deserialize board like LI-GMSL2-IPX-DESER, which has MAX9296A Deserializer, uses the MIPI output and have the same plug-and-play capabilities like with USB?
https://www.leopardimaging.com/product/accessories/adapters-carrier-boards/li-gmsl2-ipx-deser/

One more question, we were led to believe Intel is slowly shutting down production and support for its 3D camera business but since the release of this camera, we want to give this one more chance before exploring different alternatives. What are the near-future plans for the D400 series in 10-year time?

[MartyG-RealSense]

I will consult with my Intel RealSense colleagues regarding your deserializer question.

The reports about the future of RealSense were a miscommunication by news websites. Whilst some RealSense camera models were declared End Of Life (retired), the 400 Series stereo camera models have continued. Intel's official statement is below.

https://github.com/IntelRealSense/librealsense#update-on-recent-changes-to-the-realsense-product-line

[MartyG-RealSense]

Hi again @khuongkhoa1999 My colleagues confirmed that you can use this hardware that you linked to..

However, D457 is not a plug and play camera like the other 400 Series USB models are. You will need to port the driver to the host, as Intel only have an Xavier driver available.

[MartyG-RealSense]

Hi @khuongkhoa1999 Do you require further assistance, please? Thanks!

[MartyG-RealSense]

Does anyone have further questions on this discussion about D457, please? Thanks!

[haitaoQ]

Hi @MartyG-RealSense,

Is there a detailed guide for using D457 on AGX Xavier?
I have D457 & 9296 De-serializer Board & Nvidia AGX Xavier. Install SDK based on pre-release version v2.52.1. But rs-enumerate-devices doesn't detect any cameras.

Looking forward to your reply. Thanks!

[MartyG-RealSense]

Hi @haitaoQ I have referred your question to my Intel RealSense colleagues. Thanks very much for your patience!

[RealSenseCustomerSupport]

---

Hi @haitaoQ, there is a reference driver package designed for D457 to work with Intel De-serializer board on the AGX Xavier. Please kindly submit a request through Zendesk support portal for us to provide the driver under NDA. Thank you.

[AndreV84]

@RealSenseCustomerSupport How to contact you through zendesk via NDA? Could you extend, please?

[AndreV84]

@haitaoQ @khuongkhoa1999
Were you able to get the d457 with LI-GMSL2-IPX-DESER working on Jetson AGX as plug and play?
did it require to "port the driver" ?

[jan-krueger]

In addition to the D457 camera, you will need a Maxim Deserializer component on the host side to decode the streams. You could initially test the D457 quickly without needing a deserializer though by using it with Connect Tech's products, and then later switch to using the deserializer component when you need to finally integrate the D457 into your outdoor weatherproofed system.

Good morning, I would like to confirm a couple of things. I have been reading this thread but I am still a bit confused.

Assuming that the AGX Xavier carrier/board I choose has the Maxim Deserializer included is this basically just plug-and-play/does Intel offer the necessary Linux drivers to make the D457 work through librealsense? Why is the deserializer necessary when it is possible to test the camera without it, or is the testing just possible through a different port that is not GMSL/FAKRA?

If the driver for the D457 on the AGX Xavier exists where would I find it?

Thank you for your time.

[MartyG-RealSense]

Hi @jan-krueger The D457 is not plug and play like the other RealSense 400 Series models are. It requires a 'reference driver'. Intel provide a driver for use with AGX Xavier at the link below.

https://github.com/IntelRealSense/realsense_mipi_platform_driver

This driver is designed for use with Intel's own de-serializer board product.

https://store.intelrealsense.com/buy-intel-realsense-des457.html

The D457 has a USB port in addition to the GMSL / FAKRA interface, though it is intended for use for debugging purposes or production line testing rather than normal everyday use. The D457 is designed to be used with a GMSL / FAKRA connection.

The de-serializer component decodes the streams at host-side. If the D457-compatible product by Connect Tech is used then it is not necessary to purchase a separate de-serializer as it is built into Connect Tech's hardware.

https://connecttech.com/connect-tech-offers-built-in-support-on-the-newly-announced-intel-gmsl-realsense-depth-camera-d457/

[AndreV84]

I am not certain why would not it work as plug and play given the OS of the nvidia Jetson has the driver for the de-serializer
https://docs.nvidia.com/jetson/l4t-multimedia/group__max9295.html
from that point of view it is not clear what exactly needs to be done with the reference driver in order to get the camera working e.g. with intel de-serializer, in my opinion

[MartyG-RealSense]

There are instructions at the link below for installing the D457 driver to AGX Xavier.

https://github.com/IntelRealSense/realsense_mipi_platform_driver#install-kernel-and-d457-driver-to-jetson-agx-xavier

The driver was designed for Intel's own de-seriaizer product though rather than de-serializers from other suppliers.

[MartyG-RealSense]

Regarding using the camera with USB, advice has been posted by an Intel Customer Support team member in regard to intermittent issues with USB connection that a RealSense user was experiencing.

https://support.intelrealsense.com/hc/en-us/community/posts/10293079498387/comments/13946561028243

[jan-krueger]

@MartyG-RealSense Thank you for your swift response. In one of your previous comments you said "In addition to the D457 camera, you will need a Maxim Deserializer component on the host side to decode the streams.". As far as I can tell Maxim belongs to Analog Devies and is not an Intel subsidiary/related. If I understand it correctly that comment implies that the Intel driver works with the Maxim Deserializer and the Deserializer that Intel offers?

[MartyG-RealSense]

It works with the Intel de-serializer, and products that contain Maxim de-serializer technology. An example of such products that has been discussed is those by a company called Leopard Imaging.

https://www.leopardimaging.com/product/accessories/adapters-carrier-boards/li-gmsl2-ipx-deser/

My Intel RealSense colleagues confirmed that the product in the above link can be used with D457.

My understanding is that de-serializer products containing MAX9296A technology are compatible, as discussed at #10984 (comment)

The Leopard Imaging board is now listed as In Stock at the time of writing this.

[MartyG-RealSense]

Hi everyone, I have received information from my Intel RealSense colleagues that it is fine to use the D457 only in USB mode and it has the same USB performance and the same feature-set as the D455, with the added IP65 protection.  So it is absolutely valid to purchase the D457 for the protection and use it as a USB camera instead of in GMSL / FAKRA mode.

To enable USB mode you need to set the mode switch to USB mode, as the D457's default is MIPI mode. The switch is under a small cap on the camera's casing.

[masip85]

Hello

IntelRealSense/realsense_mipi_platform_driver#221 (comment)

Is possible to have this project working for orin nano dev kit? I've tried to use it directly, or adapting it modifying files , but I really don't know how to do this. I'd really need some help and guidance. Why is orin nano/nx outside the scope of this project? Does it need many changes?

[MartyG-RealSense]

Hi @masip85 The MIPI driver is designed for AGX Xavier and Intel's own brand of deserializer board.

https://store.intelrealsense.com/buy-intel-realsense-des457.html

When using D457 with a different model of Jetson or a different deserializer board, the customer needs to 'port' the driver for use with the particular equipment that they have. Intel are unable to assist customers with this porting process, unfortunately.

It looks as though the MIPI driver's installation instructions for Orin are expecting JetPack 6 to be installed. Is your Orin Nano using JP6, please?

https://github.com/IntelRealSense/realsense_mipi_platform_driver?tab=readme-ov-file#install-kernel-and-d457-driver-to-jetson-orin

Nvidia advise in a discussion at the link below that if JetPack 5.1 is used on Orin then a manual fix has to be applied in order for the “Configure Jetson AGX CSI Connector” option to work. They included the fix in JP versions after 5.1.

https://forums.developer.nvidia.com/t/r35-2-1-jetson-io-configure-jetson-agx-csi-connector/241893