Articulated object skinning #2076

0mdc · 2023-04-18T20:16:47Z

Context

To enable training of collaborative agents, we need the ability to render humans in simulation.

This changeset introduces skinned mesh support to Habitat-Sim by enabling articulated objects to render a skinned mesh that follows along its articulations.

skinning.webm

How it works

Articulated objects are defined by URDF files, while skinned meshes are creating using glTF or FBX files.

Upon creating an articulated object, Habitat looks for a <asset name>.ao_config.json file, which may contain a render_asset field containing the skinned mesh path.

If a skinned mesh is found, the system instantiates both objects and string-matches glTF bone names with URDF link names. During this process, new child transform nodes are created for each articulated object link so that the skinned mesh can be driven.

For the system to work as desired, both URDF and glTF definitions need to share the same bone names, dimensions and unit conventions. Blender scripts that generates URDF files from rigged meshes will be made available for enabling quick creation of models that can satisfy these constraints.

ao_config.json

The following parameters are added in ao_config.json:

render_asset: Specifies the path to the GLTF model. Activates skinning and disables primitive rendering.
debug_render_primitivies: Flag that turns on primitive rendering on top of skinned mesh for debugging.

Left: default, Middle: render_asset, Right: render_asset + debug_render_primitivies

Current limitations

PBR shading is not implemented.
- This is a small change that will be done after the architecture is pinned down, i.e. after this PR is merged.
Gfx-replay not yet supported.
Batch rendering not yet supported.
Scaling of URDF file not yet supported.

How has this been tested

Tested locally on viewer.cpp and on Habitat-Lab prototype human interactive viewer.

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

eundersander

I'd love to see a unit test soon with sample data. I'm curious e.g. are you supposed to strip the per-link visual elements out of the URDF if you're also attaching a skinned model.

eundersander · 2023-04-19T16:42:33Z

src/esp/assets/RenderAssetInstanceCreationInfo.h

@@ -44,6 +49,7 @@ struct RenderAssetInstanceCreationInfo {
  Corrade::Containers::Optional<Magnum::Vector3> scale;
  Flags flags;
  std::string lightSetupKey;
+  std::shared_ptr<physics::ArticulatedObject> rig;


RenderAssetInstanceCreationInfo is meant to be a lightweight object that can be easily serialized (JsonEspTypes.h), so I'm a little confused to see this member here. How does it get serialized?

I agree that this is not ideal.

I put the rig there because it's a straightforward way to pass it down the resource manager with minimal changes. It does not get serialized like other fields.

I'm not sold on a solution. In the spirit of making this gfx-replayable, perhaps this could evolve into a rig ID that would be registered in the resource manager, and eventually matched with a rig representation in the batch renderer.

What do you think?

Yeah. Let's just assume this will get revisited when you integrate skinning into gfx-replay. I'm happy for this to merge as-is.

eundersander

I left a few more minor comments. I like that articulated objects generally don't know anything about skinned models, and skinned models don't know that much about articulated objects (just pointers to SceneNodes). Nice to keep these as decoupled as possible.

src/esp/gfx/GenericDrawable.cpp

eundersander · 2023-04-19T17:04:48Z

src/esp/gfx/GenericDrawable.cpp

+        lastTransform = jointTransformations[i];
+      } else {
+        // Joint not found - use last transform.
+        jointTransformations[i] = lastTransform;


Why does this failed-find happen in practice?

This happens if the URDF doesn't perfectly match the skinned model.

For example, skinned models can have a first "root" bone that goes from the origin to the pelvis. It's also somewhat common for skinned models to have extra leaf bones.

Our rig-to-URDF tool should minimize these cases.

src/esp/assets/ResourceManager.cpp

…object.

…ed mesh for debugging.

jturner65

LGTM except for the minor things with the SkinData header. This is really cool!

src/esp/gfx/SkinData.h

eundersander

I left some more comments.

eundersander · 2023-04-21T14:58:17Z

src/tests/SimTest.cpp

+  }
+  ao->setJointPositions(jointPos);
+
+  checkPinholeCameraRGBAObservation(*simulator, "SimTestSkinnedAOPose.png",


Awesome! Happy to see image-based tests for this feature.

src/esp/gfx/GenericDrawable.cpp

src/esp/gfx/SkinData.h

src/esp/assets/ResourceManager.cpp

src/esp/gfx/GenericDrawable.cpp

…ive to the file. Code clean-up.

* skip unsupported 3D primitives (#2054) * bugfix for setArticulatedObjectModelFilename causing garbage log output (#2053) * Gfx-replay polish: workaround for material-overrides; new keyframe getter (#2035) * gfx-replay polish: workaround for material-overrides; new keyframe getter API needed for a python application * --Have pre-commit use node version compatible with Ubuntu 18.04 (#2058) * --use system node install if present * --change to hook-specific node version specification * --change to nearest official release version * Fix viewer.py framebuffer size mismatch. (#2055) * Fix viewer.py framebuffer size mismatch. * Change int tuples to mn.Vector2. * Minor simplifications. * Replay renderer: add line-rendering and unproject() (#2057) * integrate DebugLineRender into replay renderers * add ClassicReplayRenderer.unproject * stub implementation for BatchReplayRenderer.unproject * lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly * bugfix: cylinder proimitive collsion shape did not use halflength (#2060) * --have disabled renderer test only execute 1 time (#2065) * --(Bugfix) Fix normal transformation calc and address backface culling in case of negative scaling/reflections (#2062) * --don't use co-factor matrix for transforming pre-calculated normals * --address backface winding in Generic and Phong drawables TODO : Still need to set this up in batch renderer. * --revert change in batch renderer The normal matrix derivation will be changed in magnum to be appropriate, so we will keep the original verbiage for clarity. We keep the changes in the default and PBR renderer calls since we need the determinant to determine if winding order must be reversed for backface culling. * --test negative scaling along 3 axis * --make copy of observation buffer for ground truth observation The buffer is shared, so without a copy subsequent observations would overwrite the gt observation buffer. * Update Magnum submodules to latest. (#2066) * Update magnum submodules. (#2073) * Add runtime perf stats for troubleshooting perf problems (#2070) * add ResourceManager::getDrawableCountAndNumVertices, Sim.get_runtime_perf_stat_values, and related helpers * --Refactor Semantic Mesh loading/flattening to remove deprecated functionality and improve efficiency (#2079) * --refactor to remove deprecated functionality * --reviewer suggestions; * --fixed inappropriate alloc * Articulated object skinning (#2076) * Add rig property to RenderAssetInstanceCreationInfo. * Add render_asset to ao_config.json. * Create render asset from ao_config.json when creating an articulated object. * Store bone names into MeshTransformNode. * Add skinning asset loading and phong rendering. * Assorted minor fixes. * Add flag to render articulated object primitives while having a skinned mesh for debugging. * Add skinned articulated object test. * Fix MetadataMediatorTest * Cache joint transformations in drawables, other minor fixes. * Fix test asset path + other minor fixes. * Fix skinned mesh caching. Make ao_config.json render_asset path relative to the file. Code clean-up. * --Add access to Scene Instance-level user defined attributes. (#2081) * --add access to user defined in scene instance configs. * --If SceneDataset or SceneInstance do not exist, return nullptr. --------- Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com> * --Don't make needless copy of scene instance attributes when retrieving user defined values (#2082) * --don't make needless copy of scene instance attributes * --add a test * gfx-replay: fix to remove reflection when converting node transform matrix to rotation/translation (#2085) * Articulated object semantics (#2086) * Parse semantic_id from ao_config.json. * Propagate semantic_id to skinned mesh. * Add skinned mesh semantics test. * Propagate semantics for non-skinned articulated object. * --Convert materials to use magnum materials. (#2083) * --add Mn::MaterialTools; WIP * --address test issue; * --convert materials to magnum materials * --address fallback material not having defaults * --fix many attribute accessor bugs. * --appropriately make new attribute name with lowercase letter * --Flat/Phong and Semantic textures work. Have to fix building PBR layers * --improve ptr attribute naming; organize custom attribute assignment Still need to build texture pointer attribute array with layers if they exist in base material * --use owning material constructors * --cleanup; clang-tidy; fix flat vs phong ambient txtr map in phong shader Flat now has ambient Texture mapped directly from source material, so no need to check baseColorTexture anymore. * --get layers working properly * --support noneRoughnessMetallic texture; handle per-vert objectID for PBR drawables * --cache all material quantities in drawable to speed up draw access * --prepopulate normal texture * --address inappropriate bitflag check; add vertexID flag to PBR shader (#2090) * enable frustum-culling for classic replay renderer (#2096) * enable recompute_navmesh when creating sim with create_renderer==false (#2097) * --[BUGFIX] Reverse directional lights before sending to uniform; (#2094) * --reverse directional lights before sending to uniform; Also correct existing light setups * --fix lighting tutorial lights * --correct orientations of sample and default light with names * Update Magnum submodules. (#2100) * --[BUGFIX] Single channel texture support (#2102) * --Address uncompressed single and double channel textures by modifying swizzles to be rrr1 and rrrg respectively. * add intertiaFromURDF option to URDF loading API (#2098) * add additional magnum importer plugins to python build dependencies * refactor to reduce redundancy --------- Co-authored-by: Eric Undersander <eundersander@fb.com> Co-authored-by: John Turner <7strbass@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com> Co-authored-by: Vladimír Vondruš <mosra@centrum.cz>

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Apr 18, 2023

eundersander reviewed Apr 19, 2023

View reviewed changes

0mdc force-pushed the articulated-object-skinning branch from 2b98b73 to e96d590 Compare April 19, 2023 18:17

jturner65 reviewed Apr 20, 2023

View reviewed changes

src/esp/assets/ResourceManager.cpp Outdated Show resolved Hide resolved

0mdc requested review from aclegg3 and mosra April 20, 2023 19:41

0mdc added 8 commits April 20, 2023 16:54

Add rig property to RenderAssetInstanceCreationInfo.

1802583

Add render_asset to ao_config.json.

1d9cd0b

Create render asset from ao_config.json when creating an articulated …

52b8e18

…object.

Store bone names into MeshTransformNode.

5370479

Add skinning asset loading and phong rendering.

6fd5d06

Assorted minor fixes.

e13aa20

Add flag to render articulated object primitives while having a skinn…

b1eb624

…ed mesh for debugging.

Add skinned articulated object test.

f34c6fb

0mdc force-pushed the articulated-object-skinning branch from ef1fb4e to f34c6fb Compare April 20, 2023 20:58

0mdc added 2 commits April 20, 2023 17:34

Fix MetadataMediatorTest

10288f3

Cache joint transformations in drawables, other minor fixes.

6115ce3

0mdc marked this pull request as ready for review April 20, 2023 22:21

Fix test asset path + other minor fixes.

51abdd7

0mdc force-pushed the articulated-object-skinning branch from 2f8a18f to 51abdd7 Compare April 21, 2023 14:45

jturner65 approved these changes Apr 21, 2023

View reviewed changes

src/esp/gfx/SkinData.h Outdated Show resolved Hide resolved

eundersander reviewed Apr 21, 2023

View reviewed changes

Fix skinned mesh caching. Make ao_config.json render_asset path relat…

e22d8a5

…ive to the file. Code clean-up.

0mdc merged commit 8282d14 into main Apr 24, 2023

0mdc deleted the articulated-object-skinning branch April 24, 2023 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Articulated object skinning #2076

Articulated object skinning #2076

0mdc commented Apr 18, 2023 •

edited

Loading

eundersander left a comment •

edited

Loading

eundersander Apr 19, 2023

0mdc Apr 20, 2023 •

edited

Loading

eundersander Apr 21, 2023

eundersander left a comment

eundersander Apr 19, 2023

0mdc Apr 19, 2023 •

edited

Loading

jturner65 left a comment

eundersander left a comment

eundersander Apr 21, 2023

Articulated object skinning #2076

Articulated object skinning #2076

Conversation

0mdc commented Apr 18, 2023 • edited Loading

Context

How it works

ao_config.json

Current limitations

How has this been tested

Types of changes

Checklist

eundersander left a comment • edited Loading

Choose a reason for hiding this comment

eundersander Apr 19, 2023

Choose a reason for hiding this comment

0mdc Apr 20, 2023 • edited Loading

Choose a reason for hiding this comment

eundersander Apr 21, 2023

Choose a reason for hiding this comment

eundersander left a comment

Choose a reason for hiding this comment

eundersander Apr 19, 2023

Choose a reason for hiding this comment

0mdc Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

jturner65 left a comment

Choose a reason for hiding this comment

eundersander left a comment

Choose a reason for hiding this comment

eundersander Apr 21, 2023

Choose a reason for hiding this comment

0mdc commented Apr 18, 2023 •

edited

Loading

eundersander left a comment •

edited

Loading

0mdc Apr 20, 2023 •

edited

Loading

0mdc Apr 19, 2023 •

edited

Loading