HMD Position -> SlimeVR Server -> Tracker Position is likely always >= 1 frame behind

[kitlith]

This issue is exacerbated by any additional latency in the server itself, which is being tracked here: SlimeVR/SlimeVR-Server#511

I think we have a couple routes here:

• If the server sends us the HMD position it's using, we can cut this latency loop by correcting with the current HMD position.
  • VRChat basically does this with OSC, though the reasoning is different. (potentially different tracking spaces, NOT latency)
  • This partially fixes the specific latency route mentioned in the title of the issue. However, it doesn't fix HMD rotation possibly having other affects on the skeleton.
    • Could we also tell the driver about head length/offset and have it correct for that?
  • It also doesn't help anything else if it means that all of our tracker information is >= 1 frame behind.
    • This seems like something that we could measure. driver -> server -> driver latency versus tracker -> server -> driver latency.
    • (This is basically why I objected to this idea in the past when iirc @TheButlah brought it up)
  • We might be able to make it correct for controller/vive tracker positions if we send enough information to the driver.
    • honestly this feels like it's getting closer and closer to implementing the skeletal model in the driver and running it there.
• Once the server has low enough latency, we could have an integrated loop between the driver and server where the driver sends HMD information for this frame, server sends back the most up to date tracker information in response, and driver waits for that response before continuing.
  • This might require changes in how server works. Possibly an independent thread, or maybe a way for the server to register "I need to send out tracker updates now" in response to a network request.
    • The thought of an independent thread here goes almost counter to how I was going to suggest helping improve things in the server in comparison to OSC though, heh.
  • This allows us to keep complications out of the driver and in the server, but it would mean that if the server stalls, the driver stalls too.

[TheButlah]

@Louka3000

[kitlith]

The more I think about this the more I go back to my original stance that "correct the HMD position in the driver" does not technically result in correct results. Problem is:

• VRChat's OSC basically does exactly this (unintentionally, because they want to align playspaces)
• We're not setup to do it the other way right now.

For now, I'm going to take the stance of getting us as close to as on-par with OSC as we can, and then leaving this issue open so we can do the more correct thing in the future.

[TheButlah]

For now, I'm going to take the stance of getting us as close to as on-par with OSC as we can,

I agree that merely doing "adjust translation to be based on current headset position instead of old one" is not a complete fix. But its low hanging fruit, imo. Should just be a new field on the protobuf, and a quick subtraction of translation.

We should still also work on optimizing the actual round trip latency, but the translation adjustment will help perceptually quite a lot, especially in the chest and waist.

[kitlith]

I happened to look at the driver sample included with openvr today and noticed this:

void RunFrame() {
    // In a real driver, this should happen from some pose tracking thread.
    // The RunFrame interval is unspecified and can be very irregular if some other
    // driver blocks it for some periodic task.
    /* <snip> */
}

which answers the question I was thinking of asking about threads vs waiting in RunFrame.

so at the very least receiving positions from the server should be handled in another thread, if not other things. So, #33 is relevant... and apparently I was requested to review it a couple weeks ago. >_>

as-is it doesn't solve this issue, but it could be solved by handling position messages in the bridge thread introduced in that PR immediately instead of pushing into a queue.

There's probably an argument for not waiting for a bridge frame to send HMD positions as well, but shrug

[ButterscotchV]

Has this been resolved now? There have been a lot of changes to attempt to mitigate this.

[Erimelowo]

It’s not solved as @0forks ’s fix had some problems.
You can see the delay in my devkit video due to using lock-all. It’s quite obvious.