-
-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement SkeletonRetarget
and overhaul some animation features
#56902
Implement SkeletonRetarget
and overhaul some animation features
#56902
Conversation
SkeletonRetarget
and overhaul some animation features
Is retargeting limited to Skeleton3D, or does it also work for Skeleton2D? The class reference should make this clear. |
@Calinou It's 3D only, do I need to rename the class to SkeletonRetarget3D in addition to putting the note in the SkeletonRetarget document (class reference)? |
I'm not sure if the class should be renamed – I'll let others give their opinion on this 🙂 In Godot, 3D-only classes such as VoxelGI, ReflectionProbe and Decal generally don't have a 3D suffix. |
c256b84
to
cd68419
Compare
Updated the document for now, clarifying that "SkeletonRetarget is for Skeleton3D" in the class reference. I don't have much knowledge about Skeleton2D. If it has the concept of rest/pose, then I think a similar implementation of SkeletonRetarget is possible. However, it may not possible to implement a retargeting track for Skeleton2D in AnimationPlayer. The reason is that Skeleton2D does not have its own track type for transform. In Skeleton3D, only Pos/Ros/Scl3DTracks are allowed for bone transform. But, Skeleton2D bone transforms are shared with normal properties, and there are two types of tracks: Bezier Tracks and Property Tracks. This makes retarget tracks for Skeleton2D difficult to implement. Although, I think 2D assets don't have much concept of stance, unlike 3D assets, so I guess there is not much demand for implement them for Skeleton2D. In other words, there are few models that have the "same stance" and different bone naming conventions and rest as an asset; It's means, there are few animation packages. So I guess it is rare case that need to retarget for 2D, since multiple models in one product may be created by the same person. I also wait for others' opinions. |
cd68419
to
403b02b
Compare
2952860
to
1534ab6
Compare
a9af0da
to
a72ecb3
Compare
We're struggling with @reduz on whether to record The fundamental difference between my approach and reduz's is the point that "animation requires or does not require a Skeleton from which the animation is generated". The premise is that animation data can have information about the original Skeleton through pre-calculation. However, this is destructive and the information of the original Skeleton cannot be retrieved from the animation. For example, let's say the bone rest value is
In this case, a value of What is being debated is where to record the "information as to whether the value is a pre-calculated value or not". I believe it should be stored in the same place as the value. In other words, Reduz thinks it should be selected as a preset when importing the model and store the I consider the reduz's method to be problematic in two main points. 1) If a preset is changed after imported animations, all animation values must be recalculated Also, the original skeleton which is source of imported animations is required for recalculation. It means that they will be broken if the original skeleton has been removed and does not exist, or if there are duplicates or newly created animation that are not tied to the original skeleton. 2) It is to be impossible to blend LocalAnimation and GlobalAnimation This is clear if we assume that presets are to be used, but if you store the LocalAnimation and GlobalAnimation cannot be blended directly because they are recorded in different coordinate spaces by pre-calculation. AbsoluteAnimation (glTFAnimation, Godot4 Default)
LocalAnimation (Blender, Godot3 Default) (a.k.a RestAnimation)
GlobalAnimation
In order to blend them, their values must be restored to the target skeleton bone's coordinate space, but to do so, it is necessary to know whether the animation was "pre-calculated as LocalAnimation" or "pre-calculated as GlobalAnimation". There is no need to think too hard, it is obvious that if you have an animation extracted as GlobalAnimation, its playback will be broken if LocalAnimation mode is applied to the target skeleton bone. We are discussing hard about this case of blending LocalAnimation and GlobalAnimation because reduz claims that it is a very niche case. For blending LocalAnimation and GlobalAnimation:For example, I have Model A and Model B, with rest optimized for my project. These models have different rest values, but the rules for the joints match, such as +X rotation bends, and parent to child is Y-axis and roll. The basic animation is created by Model A and shared with Model B as LocalAnimation. It is important to understand that LocalAnimation is less likely to cause bone breaks if the models match the joint rules. This is very useful in cases where different models have different orientations, such as the thumb. If you have struggled with finger control in Unity models, this is probably easy to understand. However, I want to add more animations, so I want to import animations from Model M, where Rest rules do not match. If the animation of Model M can be imported as If there are two ways. Import the animation of model M as GlobalAnimation. However, re-import the animation of model A and model B as GlobalAnimation instead of LocalAnimation. At this time, the advantage of LocalAnimation's bone breaking resistance is lost, and the playback animation of the fingers will be dirty. This means that if there is even one model in the project with a different rest rule, only GlobalAnimation can be used. Alternatively, the rest rule of Model M could be matched to Model A and Model B by somehow, but the process of validating rest rule is complicated. Also, if you do simple re-rests, it produce messier animations. TL;DR
We need others' opinions as it is difficult for us to resolve the discussion on our end. Please give us your opinion if possible: @fire @lyuma @SaracenOne @AndreaCatania @TwistedTwigleg @nonunknown If any other developers are interested in character animation or retargeting, please join the discussion. |
Hm. Looking at my character models, who have different bone names and rest poses despite coming from the same author and pack, I think I would mostly use GLOBAL mode. However, for some things, e.g bending arms to wave, or legs to sit down, I think LOCAL would be more intuitive (just bend this by 90 degrees, this other by 90 degrees, done). So yes, +1 to something that allows mixing GLOBAL and LOCAL. |
984509f
to
974e153
Compare
44b64c0
to
13fe177
Compare
Yes, it has been proven in Godot 3.x (which uses LocalAnimation as default) that it is not sufficient for retargeting. For the time being, GlobalAnimation should be the default for retargeting to avoid confusion for light users. However, for advance users, for parts with delicate joints such as fingers, it is more appropriate to use LocalAnimation (RestAnimation) with matching RestRule. MMD (MikuMikuDance), a dance creation software, basically only performs GlobalAnimation, but there are plug-ins that adjust fingers' Rest for editing, which proves that there is a demand for this method. For about AbsoluteAnimation, the only use case I can think of for transferring AbsoluteAnimation to another model is when you want to animate a bone which is child of the root of the IK joint as an IK target, regardless of the model's figure. It does not seem to be completely useless.
This is in principle the same as making an A-pose into a T-pose; doing it automatically is not sufficient. There must always be manual confirmation, or the joint may bend in a strange direction. For reference, some MMD plug-ins will align the RestRule to some degree, but correct looks requires manual adjustment finally. However, even if the rest rule matches, GlobalAnimation should be used for Hips and Spine. The reason is that they do not require as much rigorous deformation because the joints are not clearly visible, and because they are closer to the root, there are more errors that depend on the model. For example, when sharing "an animation of a horizontal waist twist created with a model that does not have an S-shaped waist", using LocalAnimation may cause the arms to turn at an angle on a model with an S-shaped waist. In the same way, Neck-Head and Shoulders should basically use GlobalAnimation, but in some cases such angled rotation may be desired, so blending with LocalAnimation should be possible.
Here is a more common case. As an assumption, the quality of retargeting for fingers and limbs can be ordered as follows: [Highest Quality]
[Lowest Quality] There is a game which player collect character units. The idle animations of the characters are created and shared using the During missions, characters are rendered smaller, but instead require many animations. Therefore, animations are imported from various assets and shared using However, idle animations are also used during the mission, so If we adopt redus's method, we need to convert some animations; If the idle animation is Blending
Yes, I believe that LocalAnimation and GlobalAnimation need to be blended. And to do so, it is necessary to mark the PlaybackMode into AnimationTrack with pre-multiplying (solution 2), as shown in the diagram above. Alternatively, there are solutions that store the original skeleton in the AnimationTrack without pre-multiplying (solution 1), but this implementation should be avoided for performance and redundancy reasons. Regardless of which solution is used, marking is required on the Also, in order to absorb height differences in the root motion, adjustments using the hips height of the model will be necessary, and I think that this information needs to be included in the AnimationTrack as well. I don't know if it should be stored directly as |
13fe177
to
5fcca45
Compare
5fcca45
to
9ec9b9c
Compare
I wanted to keep RetargetMode in the AnimationLibrary implemented in #59980 and be able to blend different RetargetModes, but @reduz rejected that. Part of the discussion: Tokage:
reduz:
Tokage:
reduz:
Tokage:
reduz:
Tokage:
reduz:
Tokage:
reduz:
Also, this PR is starting to get old, so I am abandoning it for now. I think the GUI for mapping and the functions to extract transforms for retargeting are salvageable for future implementations of Retarget. I think more user voices are needed on this. Unfortunately, I feel that not most people who use Godot are familiar with the 3D character workflow since Godot is mainly used for 2D Game. And Godot's Animation team alone cannot make a fair enough decision. If anyone interested in 3D character workflow and Retarget read the log above, it would help us a bit if you could leave a comment. Or we would like to have a specific discussion on Contributors Chat. |
As a prospective user (I have several character assets, one of them is not from the packs the others are, and even the others don't share skeletons even though they share the author) I would like even a basic retargeting system to make it in. More advanced stuff like what you seem to be discussing with reduz above can go in in a further PR IF it turns out to be needed. |
Here's how I would expect animation retargetting to work, my opinion in a grain of salt, as an artist and game developer I can work on my own animations for anything I need, however what I can't do is to customize/mass produce animations to each different character I want with different proportions, that would take too much work, unreal show a use-case with it's animation retargeting that describes exactly what I want as a developer, to re-use animations on a somewhat similar or identical humanoid skeleton with different proportions. Here's the link to unreal doc: My idea of skeleton retargeting is not to be able to gain animation assets from different skeletons from some rig on the internet, but to increase the possibilities with my animations to adopt new proportions. Here are some photos of what I described gathered from the unreal docs. And here's a definition from unity's "Retargeting of Humanoid animations" section: I could only dream of a animationblendtree node filtered on which animation I want to be retargetted on which bones. Edit: So having re-read some the posts above, if I understood it correctly, my use-case would lean on global animations, doing local animations is great because it opens a whole world to do procedual animations, however I also believe it's also going to turn into a can of worms and introduce too much complexity because you need to account for unwanted rotations, alignment issues, bone twisting, etc. I would not want to edit skeleton animations inside godot, for that I would use another package to work on, like blender, I mainly want to direct animations in godot, you go there and you start from here, at the maximum on modifying animations, experimenting with animation blendings with different segmented animations in the blendtree is what I would want from the engine. |
Implement
SkeletonRetarget
based on reduz's proposal (godotengine/godot-proposals#3379).This PR includes follows:
RefactoredSuperseded by Implementbone_pose_override
inSkeleton3D
#55840get_bone_final_pose()
toSkeleton3D
#58863AutoVolume
option toAudioPlaybackTrack
to prevent to overrideAudioStreamPlayer
volume #55218Fixed incorrect blending in Pos/Rot/Scl Track #54205Superseded by Fix blend animation to solve TRS track bug & blend order inconsistency #57675This requires #55840 because
global_pose
had a serious problem of not being able to get IK and other skeleton modifire transforms. We also needed #54205 to test theAnimationTree
.#55218 and #49411 are bug fixes that are not directly related to retarget, but I believe they need to be fixed in the future, and they cause conflicts with this PR, so I included them in this PR to resolve the conflicts in advance.
Fixed #43848. Fixed #48526. Fixed #49431. Fixed #54407. Fixed #55838.
Implement SkeletonRetarget
SkeletonRetarget (Node)
Transfers the transform between the two skeletons. I implemented three types of resources for retarget. It corresponds to the "Retargeter" resource that reduz mentioned in the proposal, which is separated into several parts to make it easier to reuse.
RetargetProfile
provides the intermediate bones. The source skeleton and target skeleton bones are mapped to each other through intermediate bones usingRetargetBoneMap
. The transfer value is calculated differently byRetargetMode
. The default mode isRetargetMode::RETARGET_MODE_GLOBAL
. You can set it for each intermediate bone usingRetargetBoneOption
.RetargetProfile (Resource)
This resource is special since it is only used in the editor. It does not have any effect on the retargeting calculation.
The
RetargetBoneMap
is needed for each skeleton with a different bone name, but the keys must be the same. Therefore, to make it easier to generate the same keys in those,RetargetProfile
lists the keys as intermediate bones.You need a
RetargetProfile
when you register a key toRetargetBoneMap
orRetargetBoneOption
in the editor. However, once you register the keys in them, you can discard theRetargetProfile
if you don't need it. Also, you can register keys in them with the script always.RetargetRichProfile
defines textures, coordinates, and groups in addition to the intermediate bones. It provides a GUI with the buttons placed on the silhouette image to register the keys toRetargetBoneMap
andRetargetBoneOption
.If users have defined their own RetargetProfile, it is recommended that they also include this profile when sharing retarget animations to make convert easy between users.
I'm still undecided on how to implement the presets that Godot has. Should I just distribute
gd_humanoid.tres
in the Asset library, or should I provide a hard-coded uneditableGDHumanoidProfile
class that inherits fromRetargetRichProfile
? A similar implementation used to exist,default_env.tres
, but that has been replaced by an internal implementation.RetargetBoneMap (Resource)
A map of skeleton bone names with intermediate bone names as key. To register a key in the editor, a
RetargetProfile
is required. In the inspector, maps are organized byRetargetProfile
, but if there is noRetargetProfile
, they are listed in the unprofiled bones section.RetargetBoneOption (Resource)
Set the options for each intermediate bone. It has
RetargetMode
as one of the options. The properties that should be set for each intermediate bone and do not need to be set for each skeleton, will be put together inRetargetBoneOption
instead ofRetargetBoneMap
.RetargetMode::RETARGET_MODE_GLOBAL
Transfers the global transforms in the model space relative to the bone rest.
If the models are in the same stance, the animations will basically transfer well. However, if the models are in different stances, such as T-stance and A-stance, the transfer will be made looks strange.
However, the retargeter should not involved in that. Potentially, it make sense that an add-on such as T-poser makes match the stance between them.
BTW, the bones do not allow for shearing, so it will be orthogonalized implicitly if the animation includes scaling.
RetargetMode::RETARGET_MODE_LOCAL
Transfers the local transforms relative to the bone rest (same as the default animation format in Godot 3).
1.mp4
For example, for bones that are slanted even at T-stance, such as fingers, the joints may rotate in odd directions. In such a case, it is recommended to use
RETARGET_MODE_LOCAL
with a restriction in the bone rest axis such as "+X axis rotation to bend".RetargetMode::RETARGET_MODE_ABSOLUTE
Transfers the local transforms relative to the initial value of
Transform
(same as the default animation format in Godot 4).If the bone rest does not match completely, it will look bad. Possibly, it could be useful for transferring animations between different stances, such as T-stance and A-stance, if the Rest rules match, but it is basically not recommended to use it for transferring the transforms between different models.
AnimationPlayer
SkeletonRetarget
transfers the transforms from the source skeleton to the target skeleton, but this means there are always two skeleton in the scene, which can make for worse performance in some cases.Therefore, I implemented the properties for retarget in the
AnimationPlayer
to make it easier to manage the animation and replace the target model.Also, as a variation of
Pos/Rot/Scl3DTrack
, implemented retarget track which hasRetargetMode
(actually it is justPos/Rot/Scl3DTrack
which has special path like:Hips
, only subpath).To extract the retarget tracks, it need a source skeleton and a
RetargetBoneMap
. In other words, once the retarget tracks are extracted, the source skeleton is no longer needed.You can create the retarget tracks from the options.
The retarget track will transform to the skeleton assigned to the
AnimationPlayer
.3.mp4
This will make it easier to manage animation resources and distribute retargeted animation packages grouped by
RetargetProfile
in asset libraries, etc.BTW, I'm worry about that should it allow multiple retarget skeletons to be assigned to an
AnimationPlayer
, in case one animation contains multiple skeletons? In my opinion, that can be solved by splitting the animation into multiple animations and usingAnimationPlaybackTrack
.Note:
For rotation, the global retarget track will calculate the multiplication
Quaternion
3 extra times per bone, and the local retarget track will calculate the multiplication 1 extra time. As a reference, for a single model with 54 bones, using or not using the global retarget track made a difference of about 0.2ms in the process.I think this is not enough of a performance hit to be a problem for several models, but I would recommend matching the axes in the DCC software and using local retarget if possible (the delay will be 1/3). Of course, it goes without saying that if you have hundreds of enemy models, you shouldn't use global/local retarget in runtime in the first place; You can restore retarget tracks into them.
TODO in the future
To adjust position value between the models have different height
The position values are transferred regardless of the model height. For example, transferring the root motion from a short model to a tall model will cause a slip. Position values are may used in weapon throwing animations. For toe and knee stands, we may need to specify specific bones in the animation. For example, a toe stand requires the length of the Foot-Toes. Kneeling requires a length of LowerLeg-Foot. And its length varies from model to model, both require a Hips bone to move.
For now, I think the only problem is the root motion, but I don't know if I should solve it in the retarget process. It may be solved by implementing an AnimationTree with a
root_scale
property. Or, maybe we need to store RootDistance (ModelScale / HipsHeight) as a single float value in RetargetBoneMap in the future. So, I make it is out of the scope of this PR.For compressed tracks
I'm a little concerned about the conversion of compressed tracks, since it convert all keys when converting to Retarget tracks, but for compressed tracks it may need to do the same for all byte arrays. This will be worked out after the implementations around of the compressed/stream track is finalized.
Retarget track renamer
We may want a utility to match the names between extracted animations with different profiles, it can be added later.
Adjusting the transform value when the target model has fewer bones than the source model
For example, if you are transferring motion from a model with four spine bones to a model with three spine bones, you will need to add the deformations of the removed bones to one of the three in order for the looks to match.
I experimented with this a bit, but aborted because it seems that we would need to change the cache format of the animation system to support this.
One solution that can be implemented quickly is to add a virtual bone of zero length to the parent or child. This helper method can be added later.
skeleton_retarget_sample_project.zip