HoloLens 2 PV camera's frame timestamps reference

[emaschino]

Hi,

I want to locate with xrLocateSpace HoloLens 2 PV camera at given frame timestamps. But I don't know how to properly interop between camera timestamps and OpenXR runtime timestamps, such as predicted display times.

For HoloLens 2 PV camera, I'm using MediaFrameReference::SystemRelativeTime timestamps, defined as 100ns interval QPC time, so a fraction of a QPC value if I understand it correctly. Thanks to xrConvertWin32PerformanceCounterToTimeKHR from XR_KHR_win32_convert_performance_counter_time extension, I was hoping to convert the camera timestamps to OpenXR runtime XrTime timestamps for use with xrLocateSpace. But this doesn't work, i.e. I'm getting an invalid location.

Below are the logs of a running session where I'm outputting the timestamps of interest:

• MediaFrameReference time is the timestamp returned as-is by MediaFrameReference::SystemRelativeTime in MediaFrameReader::FrameArrived event;
• System QPC time is the timestamp returned by a call to QueryPerformanceCounter in MediaFrameReader::FrameArrived event;
• System XrTime is the above timestamp converted to XrTime with xrConvertWin32PerformanceCounterToTimeKHR;
• OpenXR predictedDisplayTime is OpenXR runtime XrTime timestamp.

MediaFrameReference time = 541897815709
System QPC time = 1040444504957
System XrTime = 54189817966510
OpenXR predictedDisplayTime = 54189878473400
OpenXR predictedDisplayTime = 54189895134200
MediaFrameReference time = 541898148908
System QPC time = 1040445144667
System XrTime = 54189851284739
OpenXR predictedDisplayTime = 54189911812200
OpenXR predictedDisplayTime = 54189928482900
MediaFrameReference time = 541898482108
System QPC time = 1040445778477
System XrTime = 54189884295677
OpenXR predictedDisplayTime = 54189945160400
OpenXR predictedDisplayTime = 54189961798100
MediaFrameReference time = 541898815307
System QPC time = 1040446429967
System XrTime = 54189918227447
OpenXR predictedDisplayTime = 54189978474100
OpenXR predictedDisplayTime = 54189995152100
MediaFrameReference time = 541899148507
System QPC time = 1040447063824
System XrTime = 54189951240833
OpenXR predictedDisplayTime = 54190011809100
OpenXR predictedDisplayTime = 54190028465800

If I pass to xrLocateSpace the timestamp returned by MediaFrameReference::SystemRelativeTime, the PV camera isn't located. Expected as it's a QPC time value.

Not shown in the logs, if I pass to xrLocateSpace the above timestamp converted to XrTime with xrConvertWin32PerformanceCounterToTimeKHR, the PV camera isn't located. The resulting XrTime timestamp appears to be based on a very different time reference than OpenXR predicted display time.

If I pass to xrLocateSpace the timestamp returned by the call to QueryPerformanceCounter in MediaFrameReader::FrameArrived event, the PV camera isn't located. Expected as it's a QPC time value.

But if I pass to xrLocateSpace the above timestamp converted to XrTime with xrConvertWin32PerformanceCounterToTimeKHR, the PV camera is successfully located.

I wish I could use the camera timestamp directly (using the appropriate adjustment and/or conversion, of course) rather than having to perform a call to QueryPerformanceCounter to generate a timestamp suitable for use with xrLocateSpace (once converted to XrTime). Any guidance on this or is calling QueryPerformanceCounter in MediaFrameReader::FrameArrived event the right approach? Otherwise, what am I doing wrong with the camera timestamps handling?

If you carefully look at the timestamps returned by MediaFrameReference::SystemRelativeTime, you'll notice there are roughly 1/100th of OpenXR runtime XrTime timestamps. And indeed, if I pass to xrLocateSpace the timestamps returned by MediaFrameReference::SystemRelativeTime multiplied by 100, the PV camera is successfully located. Is this 100 multiplier in any way correlated to the 100ns interval QPC time in MediaFrameReference::SystemRelativeTime documentation? In short, is it expected that I simply have to multiply by 100 the timestamps returned by MediaFrameReference::SystemRelativeTime to pass to xrLocateSpace without having to rely upon xrConvertWin32PerformanceCounterToTimeKHR at all? Does it mean that HoloLens 2 PV camera gives already-suitable-for-use-with-OpenXR timestamps? If yes, is this guaranteed to always be the case for all the WMR devices? What's then the purpose of XR_KHR_win32_convert_performance_counter_time extension?

Thanks.

[MarkVabulas]

As a side question, how are you actually locating the camera pose in OpenXR? Through the scene graph bridge? It's so poorly documented, what sequence of calls did you perform to make it give you the position in OpenXR space? I do sucessfully have the GUID and Mixed Reality Spatial pose from the camera frame...

[emaschino]

Yes, with XR_MSFT_spatial_graph_bridge extension. You get the dynamic node ID of the PV camera from the calibrated transform retrieved from the MediaReferenceFrame properties (lookup on MFSampleExtension_CameraExtrinsics). You then create a reference space for this dynamic node (of type XR_SPATIAL_GRAPH_NODE_TYPE_DYNAMIC_MSFT). This can be later located with xrLocateSpace, thus giving you the location of the PV camera if using the correct timestamp (hence my above question).