Added NVRVirtualHand and NVRVirtualInputDevice to use with NPC and remote users

[jbroadway]

Attach NVRVirtualHand to the hands of NPCs and remote users to replay remote interactions over multiplayer. You can control the hand by calling one of three new methods on NVRVirtualHand:

• Hold() begins holding the closest interactable
• Release() releases the current interactable
• Use() triggers UseButtonDown() on the current interactable

[jbroadway]

I added two new events to NVRHand, OnBeginUseInteraction and OnEndUseInteraction, which can be used to capture the trigger down/up and broadcast them. Using these, we now have NewtonVR working over multiplayer, with grab and release and the trigger working, which is very exciting!

[SmartCarrion]

I haven't inspected your code yet, but judging by the description I'm worried about design for lag. Shouldn't 'Hold' take the interactable to be acted upon as an argument? If it just grabs the closest thing, and lag or physics makes positioning slightly different on the clients, then they could be out of sync and each client shows the player grabbing a different object.

[jbroadway]

@SmartCarrion in short, you're right. The Hold() method mainly just wraps PickupClosest() which is what gets called when you try to grab something locally already, so it's basically the same on both sides in that sense. But the On*Interaction events on NVRHand do provide an NVRInteractable so you can send the object that was grabbed and do correction based on that.

Lag is definitely a design challenge in multiplayer VR, but getting NewtonVR to where a virtual hand can grab and use objects is step 1. From here, additional logic will be needed to do prediction and synchronization. It may work to build that into NewtonVR as a next step, but it may be something that ends up needing fine tuning on a case-by-case basis, depending on the type of interaction and gameplay.

[jbroadway]

We added a few new improvements which help improve replay accuracy a little bit, namely a Hold(withObjectID) method to ensure the virtual hand picks up the correct object and not just the closest. This doesn't replace the need for client-side prediction, but those can be built on top of this as next steps.

[jbroadway]

I should add that setting an interaction point on the objects makes a big difference to how well this feels :)

[jbroadway]

Any news on when this might get incorporated, or on changes needed to do so?

We're planning additional multiplayer-related work on our end, so it would be great to get the basics in sync with the main NewtonVR project before getting too far along in our forked repo.

[Proktologe]

Hi, i just downloaded the last version from the Asset Store. Wehre can i find the scripts for use it in a Mutliplayer?

[jbroadway]

@Proktologe to start implementing multiplayer, you'll need to incorporate the modifications from this pull request into your project, and then you can attach virtual hands (the NVRVirtualHand component) to the objects that represent your remote players' hands in the scene.

From there, you would send the transform data of each user's hands and use it to update the position of each virtual hand, as well as capturing the various OnBeginUseInteraction events on the regular NVRHand components to know when to call Hold(), Release() and Use() on the virtual hands.

This of course won't recreate the interactions perfectly, due to network latency, but there are techniques for keeping things in sync. Here's a great resource on those techniques:

http://gafferongames.com/networking-for-game-programmers/what-every-programmer-needs-to-know-about-game-networking/

[jbroadway]

Added one more method, EndUse(), to virtualize releasing the use trigger.

@zite, any ETA on looking into merging this?

[jbroadway]

I had to merge two other PR's into my fork's master branch for manageability, so they're included in this one now too (an intermittent crash in builds that doesn't happen in editor, and separate interaction points for right- and left-handed use).

Any update on looking at these PR's, @zite? It's getting harder to maintain compatibility over several months with no word, and I'd like to keep contributing since we're actively using NewtonVR with multiplayer over here :)

[zite]

So you're using this for networking? Are you using a plugin or something custom? I was thinking just making a kind of dummy item would work better than trying to stub out a whole nvrplayer replacement.

At a minimum I'll need to be able to test this to make sure it works before accepting a push. But just to let you know I may end up going a different route. Can you provide any other code to help me test? If not that's cool, I can try and put something together myself. Multiplayer was on the roadmap anyway.

[jbroadway]

I'm using this for two things:

• Networked users
• Replay of recorded user actions

In both cases, I don't use NVRPlayer for them at all (only for the local player), I just attach the NVRVirtualHand to the hand objects that represent remotes and replays and on receiving incoming packets I do one of two things:

1. For positional data, I apply it to the hand's transform with a bit of lerping (I'm only sending positional data at 30 fps), which doesn't touch NVRVirtualHand at all
2. For actions like hold, release, etc. I just call the appropriate public method on NVRVirtualHand

A quick test would be:

• Create a simple mirrored "player" in the scene (you could call it Dummy and put a Head, LeftHand and RightHand under it with spheres on them)
• Add an NVRVirtualHand to each of the dummy's hands
• Mirror the NVRPlayer's positional data from the head and hands onto the dummy object on update
• Connect the events on each NVRHand to the mirrored NVRVirtualHand's Hold(), Release(), Use() and EndUse() methods

Outside of that, any networking logic stays outside of NewtonVR which could use Unity's networking, Photon, Steamworks, or anything else without requiring any changes to NewtonVR itself.

I can see about setting up an example scene like I described for you too if that would be helpful :)

[zite]

Ah gotcha! Thanks for the extra info. I was just planning on adding a separate scene to newton that showed an example of networking the example scene eventually, so figured I'd start here. Most of that wouldn't be built into "main" newton, just an extra example. Probably going to start with photon first unless you have another preference.

[zite]

I've been working on our photon implementation and I'm not seeing a direct use for this. It seems like you should be tracking the location of the items separately anyway, otherwise they don't move when you release them. I'm not rendering remote hands yet, so maybe my opinion will change when I get there. But it seems like having this done locally you're going to get some weird situations.

For replaying user action that seems like it could make sense, but with rigidbodies you're going to get into similar weird situations since things aren't deterministic / lag / other object interaction on the rigidbodies. Might as well just track the objects in the recording too.

I appreciate you contributing, and this has pushed me to implement a multiplayer example, but right now it's not looking like I'm going to merge this into main, probably best left in your fork. However, I am adding some hooks and events in that wouldn't have been there without this request, so thanks for that.

[jbroadway]

Either way it works out, I'm glad even if some of it ends up being useful :)

I'm curious how you see the remote players' actions playing out to each other. When we went to implement hands on our remote avatars, we needed a way of programmatically taking an incoming packet that said "user x grabbed object y" and applying that to the hand/interactable on the receiving end. That's where those extra methods on NVRVirtualHand came in for us, since remote users wouldn't have the NVRPlayer or NVRHand that the local player does.

We've also implemented a couple different methods for replaying object movement, depending on the type of object. Some we don't care if they replay exactly and we just apply the velocity, etc. on release. Others we track more precisely.

Anyway, looking forward to seeing your multiplayer example too. Cheers!

[zite]

Wellll, I'm kind of doing a 180 here. Initially I had object tracking without modifying any of the player bits. But now that I'm trying to render the players too I realize I probably should just do a full set of nvrplayer inheritance down to hand. That way each client can just check .IsAttached on objects, get the hand from the player, things like that. I'm skipping over nvrinputdevice right now because it seems kind of redundant, but we'll see how things go as I progress. Just thought I'd give you an update to let you know I'm still working on it.

[jbroadway]

Since our fork has evolved a bit beyond this PR (adding separate left and right hand interaction points), but is being actively maintained for our project, I've added a wiki page here that describes what we've added. Hope that helps in case these changes are useful for others.