Skip to content

Home

Jump to bottom
Lukas Snoek edited this page May 31, 2023 · 10 revisions

Welcome to the metahuman-animator wiki! The metahuman-animator project is a proof-of-concept (POC) to leverage Unreal and the MetaHuman framework to render face/body stimuli for psychophysics research. This wiki outlines the progress made on this endeavour.

Requirements

To render MetaHuman stimuli using Unreal, you need the following software/libraries/plugins:

  • Unreal Engine (version 5.1, as the MetaHuman plugin is only available for 5.1); download via the Epic Games launcher
  • MetaHuman plugin (only necessary if you want to create MetaHumans from 3D scans/data)

Note that Unreal Engine (UE) is available cross-platform (Windows, Linux, Mac) but the MetaHuman plugin is only available for Windows! Also, be aware that UE takes up a lot of your hard disk storage (on my laptop, it was about 50GB).

Also, if you want to use Unreal's Python API (i.e., control Unreal Engine through Python commands), I recommend to do so via the VSCode editor in combination with the Unreal Engine Python plugin for VSCode (by Nils Soderman). Using this plugin is way more convenient than copy-pasting code into Unreal's Python cmd console.

Conceptual workflow

This proof-of-concept is stringing together data, tools, and software packages from different sources. The text below tries to clarify this workflow in a conceptual manner. In general, we can distinguish three elements in this work flow:

  • The input data, i.e., the 3D faces/bodies
  • The intermediate representation of the data, i.e., the transformed static (3D) or dynamic (4D) faces/bodies
  • The output data, i.e., the rendered image (3D) or video (4D) from the intermediate representation

The input data must be (textured) 3D data, which can vary substantially in terms of complexity, from simple (textured) static meshes without joints (e.g., 3D data in obj files) to more extensive representations of 3D data including joints (e.g., 3D in FBX files). These input data can be rendered with any (3D) rendering program either manually (e.g., Blender, UE, Unity) or programatically (e.g., pyrender, PyTorch3D). This proof-of-concept project uses Unreal Engine as the 3D rendering program, as it is very extensive, offers state-of-the-art rendering quality, and is well-maintained (and funded). And importantly, is integral to the MetaHuman project.

The term "MetaHuman" refers to both the UE plugin itself as well as the specific 3D data object/represention of humans this plugin uses/offers. A "MetaHuman" object in Unreal is relatively detailed (including hair) and can be animated, leading to state-of-the-art full body stimuli. The downside of using MetaHumans is that they can only be created and rendered using UE, which is relatively difficult to use. (Technically, you can export them to other 3D software programs, like Maya, but those programs are as complicated as UE.) While you can use 3D faces/bodies generated by any program (e.g., the GFG or other specialized 3D face/body tools like SMPL-X) in UE, this proof-of-concept uses MetaHumans because they are incredibly realistic and can in fact be animated with UE as well (allowing for animation and rendering using the same program, i.e., UE).

Once a MetaHuman has been configured as desired, i.e., with the desired pose and (dynamic) facial expression, it can be rendered as an image (for static representations) or videos (for dynamic representations). Assuming we're working with MetaHumans and are thus already working in UE, rendering with UE is a logical choice.

This conceptual workflow omitted a lot of details, so please find more information on the other pages of this wiki:

Clone this wiki locally