Joint Solver Incorrectly Bends Certain Characters (Bug Bounty)

[TheNexusAvenger]

A long-standing flaw of Nexus VR Character Model since version 1 is how limbs are calculated. Version 1 worked well for the default package, but not for custom packages. In the case of version 2, this is the opposite. It works mostly fine for custom packages:

But with the default character, it causes the arms to bend at the elbows:

The effects in V1 of the math not being optimal resulted in no graphical issues but the position of the hands was off. In V2, this leads to the default character looking ugly for everyone. Those who want to tinker with this can do so with the following test file:
ArmMathDemo.zip (place files not supported for upload on GitHub)

I have spent a lot of time on this issue with no success on improving the situation. The current code works but leads to graphical issues, which has resulted in a lot of players reporting this bug. However, this bug has existed since the start of V2, which was released over a year ago. I want to get this resolved, and I am putting a 100,000 Robux bounty on a Pull Request that gets this resolved. The provided test file can be used as a base with final changes put in the Appendage.lua file. For the bug bounty, I am only looking for a successful demo with complete code or a pull request with the working code. Ideas can be discussed but won't be accepted unless someone acts on it.

[Anaminus]

My first observation is that the problem is more or less apparent with all limbs. My second observation is that, with the default character limb, the angle between the initial shoulder-elbow vector and the elbow-wrist vector is a lot wider than that of the other limbs. My guess is that this angle has an effect on the final result.

In the solver function, adjusting Z-angle of the PlaneCFrame variable controls the angle of the elbow.

My solution is to offset the elbow angle with this initial shoulder angle. This way, limbs with shallower shoulder angles are corrected less, while wider shoulder angles are corrected more. Applying the full angle appears to correct it too much, so a separate factor is included to reduce the impact. A factor of 0.5 seems to work best.

Here is an example, and an updated demo:

ArmMathDemoCorrected.zip

The lower row is the original solver, and the upper row is the new solver with the elbow correction applied.

[TheNexusAvenger]

That solution is fairly similar to another one that I was given privately. What makes me hesitant to accepting either of them is a new artifact in the example above: the twisting of the lower limb. It may look better, but it isn't the complete end-all solution I am looking for. It also causes strange behavior if a player were to have there controllers facing down at their waist.

[Anaminus]

The obvious problem is that the elbow has too many degrees of freedom. It's meant to be a hinge, and the shoulder and wrist are basically ball&sockets. It also doesn't take into account the relative orientations of the joints. Doing this properly would require a more advanced solver that simulates proper joints with limits.

Thinking about it this way, it occurs to me that implementing these things would be redundant. Why do this when Roblox already has a working IK solver with all the needed constraints? In fact, Roblox's animation plugin utilizes the solver by temporarily connecting preconfigured constraint objects to the model. These constraints could be ripped right out and applied to the VR model.

Unfortunately, the problem then is that the solver is limited to plugins only:
https://developer.roblox.com/en-us/api-reference/function/WorldRoot/IKMoveTo

The other option is to try to run them with actual physics. This seems like it could be possible without too many artifacts. I don't have VR, so I can't know how well it works in practice. Regardless here's a demonstration, using the constraints provided by the animation plugin:

physical-limbs.zip

Driving the character's hands directly with the hand controllers is pretty much a disaster. Instead, separate "control" parts are driven directly by the hand controllers, then the character's hand's are approximated to these using AlignPosition and AlignOrientation. OneAttachment mode could be used here instead of parts, where the hand controller's CFrame directly sets the corresponding align constraints.

[TheNexusAvenger]

Relying on physics with constraints is an interesting idea but risks breaking compatibility with certain games. Having a character model go from Motor6Ds to several different physics constraints probably would break certain things that games rely on existing even if the values can't be relied on. That is not a compromise I want to make.

An interesting hacky workaround would be simulating the limbs on a ghost model and replicating the changes to the character. Without trying it out, it seems like it would be 1 frame behind for rendering though.

[cl1ents]

Hey!

After tinkering around for a few hours, I came up with a solution that might just be the one.
The idea is that it goes though n amount of interations (4 here) to get LowerLimbEndAttachment and LimbEndAttachment to align properly using the original solver, and to use the previous iteration to refine the precision of said alignment.
I honestly don't know how else to describe it, I don't know how well it'd actually work in the long run, but here it is!

ArmMathDemo.zip

I added the argument Arm to the solver because I honestly don't know how well it'd work for legs IK.

[TheNexusAvenger]

Interesting approach. The picture you posted is much better than what I currently have, but only with that picture. 1 of the demo arms completely falls apart with the approach for some reason.

RobloxStudioBeta_51GR8FuMv8.mp4

Moving the arm to test isn't required to have this happen. Moving all the test end points up 1 stud can result in this.

[cl1ents]

Interesting approach. The picture you posted is much better than what I currently have, but only with that picture. 1 of the demo arms completely falls apart with the approach for some reason.

I honestly do not know why that occurs, but an issue like that was bound to happen.
My solution seems to not be as viable as I thought, but I'll see what I can do in the future!

[cl1ents]

Would this be a proper solution? I made it revert to the old solver if LowerLimbEndAttachment and LimbEndAttachment are too far away:

Peek.2022-07-04.10-27.mp4

It will snap to it which is kind of a downside, but this is the best I can do off the top of my head right now

ArmMathDemo.zip

[TheNexusAvenger]

It is better but I don't think I want to accept the solution if it results in that for that body type.

[mgmchenry]

Nexus, do you still want this solved? I've been considering looking at the arm IK because I'm uncomfortable with some of the things it does, especially the offset from an actual quest controller position.

I meticulously calibrated the size, position, and orientation of hand models (here on right) and quest controllers (here on left)

And it's far off the mark in terms of matching up with where the hands end up. I was also going to look into getting the joints into more natural positions more of the time.

But if the bounty is still out, I'll see if I can get the general IK problem as well.

[TheNexusAvenger]

The bounty is in limbo now with the IKController being announced at RDC and expected to enter beta this year. It should provide a general solution that does more than the arms and works on any character. I'll accept and pay for a solution if it works as a general solution with many different arm models, but it may get replaced fairly quickly with a native solution from Roblox.

[cl1ents]

Not sure if it's of any use but, here's a decent lead: I don't think a proper solution is possible through a single iteration like the one that's currently in NexusVR - most IK systems in other leading engines, and general 3D software (Unity & Blender are good examples) also uses iteration-based solutions.
The closest we've gotten to that is my solution, and even then, it's FAR from perfect, and I'll probably go back to it sometime.

[mgmchenry]

with the IKController being announced at RDC

Do you figure that's related to the new SimHumanoidPhysics FFlag, and Humanoid seemingly getting a ControllerManger in the future? I'm optimistic about what this could bring
https://robloxapi.github.io/ref/class/ControllerManager.html

I've been eagerly watching changes to PlayerModule scripts waiting for some clues on this. Yesterday I was trying to work out some camera issues integrating Nexus VR Character Model with a modified EgoMoose wallstick controller, and noticed CameraModule.VRCamera got a bunch of changes tied to FFlagUserFlagEnableVRUpdate3 I'm hoping might help with VR BillboardGui orientation later. (FFlagUserFlagEnableVRUpdate3 still = false, so will worry about that another time)

The new API has Instance classes for AirController, ClimbController, GroundController, and SwimController. I've been working on a game that really requires local gravity vectors and magnitude as well as client-specific camera UpVector.
(1 million thanks to Animinus for keeping these insights up to date, btw. much praise. incredible tool!)

It's been a frustration that the underlying character movement behavior is a total blackbox behind Player.Move/Humanoid.Move forcing us to work around hard coded side effects like

• Detection of climbable surfaces does not take character or camera or move vector orientation into account.
• Magic air friction is applied that kills the character momentum. Since the air friction parameters could change in the future without notice, one of the currently better solutions is using a PID to counteract it, which feels smelly to me.

I'm happy the recent roloxapi changes show the new AirController has a CancelAirMomentum bool which solves one of those problems. The ControllerManager has a Vec3 FacingDirection and MovingDirection which could solve the rest?

I'm not stoked that the old black box C function is being replaced by more complicated hard coded behavior that can't be replaced by lua. I'm fully expecting roblox to ignore the facing vector for detection of climbable surfaces.
Still worth the pain of updating custom character controllers if we get performant IK for foot and hand placement, though.

The new classes were not browsable in studio yesterday, but after I updated studio today, they are. (ControllerManager, GroundController, etc)
I can absolutely instantiate a GroundController, instantiate a ControllerManager and parent the GroundConroller to it, then parent the ControllerManager to a humanoid, but ControllerManager:GetControllers() returns {}, ActiveController is nil and read only, and there doesn't seem to be a way to set it. Not good signs for being to interact with the controller behavior, but at least the classes aren't marked RobloxScriptSecurity.

If anyone has any insight on where this and the character IK is heading, I'd love to know!

[TheNexusAvenger]

... and Humanoid seemingly getting a ControllerManger in the future? I'm optimistic about what this could bring

From what I heard from several engineers, this sounds correct, but not much is public yet and only the IKController seems relevant to the project for now. I don't intend to try to make Nexus VR Character Model work with the IKController until the official beta launches. Doing it now risks behavior or API changes that throws away any work that becomes irrelevant. However, if anyone gets it working ahead of time and makes a pull request to use it, I will see if I can merge it when the time comes.

[TheNexusAvenger]

Turns out I misspoke before - it is IKControl, not IKController. A limited version of IKControl is now out as a beta and expected to be out of beta either last Q4 2022 or sometime in Q1 2023. Once some of the initial issues are gone and IKControl.Pole is re-enabled, I'll try it out for the next release. The bug bounty reward for this is suspended.

https://devforum.roblox.com/t/inverse-kinematics-control-instance-for-animation-is-now-in-beta/1999010/

[TheNexusAvenger]

IKControl is out of beta. The bug bounty is tentatively closed once I try to integrate it next week.
https://devforum.roblox.com/t/ik-controls-are-now-out-of-beta/2156480

The Issue will be closed once a new release is created with IKControl.

[cl1ents]

IKControl is out of beta. The bug bounty is tentatively closed once I try to integrate it next week.
https://devforum.roblox.com/t/ik-controls-are-now-out-of-beta/2156480

The Issue will be closed once a new release is created with IKControl.

Single question, have you found a solution to make the elbows bend properly so far using IKControls?

[TheNexusAvenger]

Single question, have you found a solution to make the elbows bend properly so far using IKControls?

From my tests on the latest (NOT second-latest) release, it works just fine. Issues you notice should be posted on the announcement thread.

[TheNexusAvenger]

Heads up: I am pulling the bug bounty on February 28th, 2023 if no one claims it.
https://thenexusavenger.io/article/28/ending-my-contributions-to-nexus-vr-character-model

[TheNexusAvenger]

The new IKControl constraints beta seems like it may be the solution to this. It takes some time to tune constraints and they still aren't quite right. We'll see if this can replace the current math when it exits the beta.
https://devforum.roblox.com/t/ikcontrol-joint-constraint-beta/2227050/12

RobloxStudioBeta_MnPbBwz2GS.online-video-cutter.com.1.mp4

[TheNexusAvenger]

Roblox's new IKControl with constraints is no longer a beta...
...but requires games to opt into the feature by changing IKControlConstraintSupport to Enabled. Until it is a safe assumption that games will have this, the current changes still can't be shipped. I have no clue what to expect for it to be enabled in all games. It could be weeks. It could be months.

[TheNexusAvenger]

Version 2.7.0 is up with IKControl. Because of the constraints being a toggle in Workspace that isn't readable by scripts, the setting Extra.TEMPORARY_UseIKControl has been added to toggle this.