DNACalib SetVertexPositionsCommand is not efficient!

[he1chenglong]

DNACalib can change neutral joint positions, but can't change neutral mesh positions.
this code can't change mesh positions in DNA file:
new_neutral_mesh = SetVertexPositionsCommand(
mesh_index, deltas, VectorOperation_Add
)
commands = CommandSequence()
commands.add(new_neutral_mesh)
commands.run(calibrated)

only run_joints_command is efficient.

[marijavik]

Hello,
It is difficult to say from this code snippet what could be the problem but I would advise you to try to see what you have in deltas list. Those values should be the difference between new and old vertex positions. The other thing is mesh_index, make sure it has the correct value.

[he1chenglong]

Yes, I have checke the deltas list.They are difference between new and old vertex positions. I focus on mesh_index 0 , head_lod0_mesh。
I have test all examples code about SetVertexPositionsCommand: dna_viewer_grab_changes_from_scene_and_propagate_to_dna.py dnacalib_demo.py dnacalib_neutral_mesh_subtract.py .
I can't change the head mesh by SetVertexPositionsCommand. I also try build_meshes, build_rig after use SetVertexPositionsCommand.
Did I miss some steps? Is there any special step to use SetVertexPositionsCommand? Is there anyone has changed head mesh successfully by this command?

[smilicev]

Can you give us full script that you are using? So we can take a look at it.

[he1chenglong]

This is my code ,It's changed from dna_viewer_grab_changes_from_scene_and_propagate_to_dna.py.
I read some metahuman's DNA Vertex Positions and save these to new DNA files.
The face SkeletalMesh of metahuman wasn't changed after I import these new DNA files.

def load_dna_reader(path):
    stream = FileStream(path, FileStream.AccessMode_Read, FileStream.OpenMode_Binary)
    reader = BinaryStreamReader(stream, DataLayer_All)
    reader.read()
    if not Status.isOk():
        status = Status.get()
        raise RuntimeError(f"Error loading DNA: {status.message}")
    return reader


def save_dna(reader):
    stream = FileStream(
        f"{MODIFIED_CHARACTER_DNA}.dna",
        FileStream.AccessMode_Write,
        FileStream.OpenMode_Binary,
    )
    writer = BinaryStreamWriter(stream)
    writer.setFrom(reader)
    writer.write()

    if not Status.isOk():
        status = Status.get()
        raise RuntimeError(f"Error saving DNA: {status.message}")


def run_vertices_command(
    calibrated, old_vertices_positions, new_vertices_positions, mesh_index
):
    # Making deltas between old vertices positions and new one
    deltas = []
    for new_vertex, old_vertex in zip(new_vertices_positions, old_vertices_positions):
        delta = []
        for new, old in zip(new_vertex, old_vertex):
            # delta.append(new - old)
            delta.append(new)
        deltas.append(delta)

    # This is step 5 sub-step c
    new_neutral_mesh = SetVertexPositionsCommand(
        mesh_index, deltas, VectorOperation_Interpolate
    )
    commands = CommandSequence()
    # Add nex vertex position deltas (NOT ABSOLUTE VALUES) onto existing vertex positions
    commands.add(new_neutral_mesh)
    commands.run(calibrated)

    # Verify that everything went fine
    if not Status.isOk():
        status = Status.get()
        raise RuntimeError(f"Error run_vertices_command: {status.message}")


if __name__ == "__main__":
    makedirs(OUTPUT_DIR, exist_ok=True)

    namelist = ["Dax", "Chandra", "Erno", "Ettore","Glenda"]
    for name in namelist:
        reader = load_dna_reader(CHARACTER_DNA)
        CHARACTER_DNA_Additive = f"{DNA_DIR}/{name}.dna"
        MODIFIED_CHARACTER_DNA = f"{OUTPUT_DIR}/{CHARACTER_NAME}_{name}_{hour}_{minute}"
        print(CHARACTER_DNA_Additive)
        reader_additive = load_dna_reader(CHARACTER_DNA_Additive)
        calibrated = DNACalibDNAReader(reader)

        print("Start Modify...")
        # This is step 3 sub-step a
        current_vertices_positions = {}

        VertexPosition = []
        for mesh_idx in range(reader.getMeshCount()):
            # Get vertices one by one
            temp = []
            for vtx_id in range(reader.getVertexPositionCount(mesh_idx)):
                vtx = reader.getVertexPosition(mesh_idx, vtx_id)
                # print(f"Mesh {mesh_idx} - Vertex {vtx_id} : {vtx}")
                temp.append(vtx)
            VertexPosition.append(temp)

        VertexPosition_additive = []
        for mesh_idx in range(reader_additive.getMeshCount()):
            # Get vertices one by one
            temp = []
            for vtx_id in range(reader_additive.getVertexPositionCount(mesh_idx)):
                vtx = reader_additive.getVertexPosition(mesh_idx, vtx_id)
                # print(f"Mesh {mesh_idx} - Vertex {vtx_id} : {vtx}")
                temp.append(vtx)
            VertexPosition_additive.append(temp)

        for mesh_index in range(reader.getMeshCount()):
            run_vertices_command(
                calibrated, VertexPosition[mesh_index], VertexPosition_additive[mesh_index], mesh_index
            )
        print("End Modify...Start Saving")
        save_dna(calibrated)
        print(f"Save DNA Complete...Path: {MODIFIED_CHARACTER_DNA}.dna")

[he1chenglong]

We want to generate a new DNA file by modified head Vertex Positions directly . Could you gvie us a tested example code which is read Vertex Positions from a DNA file and save them to a new DNA file?

[booomji]

Hello, It is difficult to say from this code snippet what could be the problem but I would advise you to try to see what you have in deltas list. Those values should be the difference between new and old vertex positions. The other thing is mesh_index, make sure it has the correct value.

EDIT START 1 13 09 2023
Finally discovered how to do this.
discussion thread

You can see the documentation on my git

documented process
EDIT END 1 13 09 2023

Hi marijavik,
Is there a code snippet or an example file that shows the process how to use a blend shape to swap the original MH head with a new shape (same topology) and then run the set_vertices or other commands. I feel a lot of us (perhaps even he1chenglong ) are struggling to understand that workflow.

As of now the Grab_chages_from_scene script allows us to tweak a mesh using joint displacement. But most of us are looking to swap the head out with a custom one with a few tweaks. The process / methodology for that is unclear.

Please could the team elaborate on that.

Thanking you,
b

[marijavik]

Hello,

usually I generate mesh in Maya from DNA file and in that scene import another mesh containing changes I want. Then I run 'run_vertices_command', save the changes in new DNA file. To validate I generate mesh from this new DNA file, import model mesh again and compare. Something like this:

def show_meshes(dna_path, add_skinning=False, add_blend_shapes=False):
    cmds.file(force=True, new=True)

    dna = DNA(dna_path)

    # Builds and returns the created mesh paths in the scene
    config = Config(
        add_joints=True,
        add_blend_shapes=add_blend_shapes,
        add_skin_cluster=add_skinning,
        add_ctrl_attributes_on_root_joint=True,
        add_animated_map_attributes_on_root_joint=True,
        add_mesh_name_to_blend_shape_channel_name=True,
        add_key_frames=True
    )

    # Build meshes
    build_meshes(dna, config)

# Starts here
show_meshes(character_dna)
reader = read_dna(character_dna)
calibrated = DNACalibDNAReader(reader)

# Update neutral mesh LOD0 based on provided model
# Add model into scene
cmds.file(model, i=True, mergeNamespacesOnClash=True, namespace=":")

run_vertices_command(
    calibrated, get_mesh_vertex_positions_from_scene("head_lod0_mesh"),
    get_mesh_vertex_positions_from_scene("Mesh"), 0
)

# Save DNA
save_dna(calibrated, mesh_dna)
show_meshes(mesh_dna)

[marijavik]

One important note - once you re happy with your changes in Maya, do not forget to generate FBX files. It won't be enough just to replace DNA files in UE, but to reimport FBX files as well.

[booomji]

One important note - once you re happy with your changes in Maya, do not forget to generate FBX files. It won't be enough just to replace DNA files in UE, but to reimport FBX files as well.

Thanks for both answers MariJavik.
I'm glad you bought the FBX export part up.
I think there is a problem with just exporting the modified DNA scene FBX to Unreal engine.
There is bone structure mismatch. At least with the results of "grab_changes_from_scene.py" example file.

Is there some prep required before we export it ?
Some tutorials show how to do that by renaming the spine bones to DHI:root and remove a few joints from the body.
For e.g this is the metapipe tools prep process.
prepare for export

Ideally we'd like to have the "grab_changes_from_scene.py" script assemble the head rig with the correct naming and bone structure so we could send it to UE. If you could share a code snippet around that it would be golden.

I'm learning so much from you.
Thanking you abundantly
b

[marijavik]

Hi,

for FBX generation, we need some body joints and those could be found here. Normally, we would take existing MH body and delete meshes and extra joints so that we get joint chain like in those files. No need to rename anything. After body is prepared, use script for FBX files generation.

[booomji]

Hi,

for FBX generation, we need some body joints and those could be found here. Normally, we would take existing MH body and delete meshes and extra joints so that we get joint chain like in those files. No need to rename anything. After body is prepared, use script for FBX files generation.

wow !
Cant wait to try this out and make it work inside UE.

Cheers,
b

[booomji]

Hi,

for FBX generation, we need some body joints and those could be found here. Normally, we would take existing MH body and delete meshes and extra joints so that we get joint chain like in those files. No need to rename anything. After body is prepared, use script for FBX files generation.

Hello MariJ,
I followed the procedure you outlined above but my fbx is looking deformed. I disabled the skin cluster to see of the body joints used from the data/body folder are at a different height from the MH i downloaded from bridge.
deformed face
The default MH file from quixel has the head bones at a different height (a seen in the comparison pic below)
original MH vs DNA modified

Should I unbind the skin then reposition the mesh and re-bind it ? I'm just worried about the head portion not meeting the body seam in UE if I do this.
Any idea how I should be going about fixing this ?

EDIT START 1 18 09 2023
On further inspection, the offset is being caused because in the modified DNA result file the head is coming in 2 units below the normal mh file from quixel. This is because of the flip-flops which are approximately 2 units.
modified dna head coming in 2 units below the quixel mh
https://pasteboard.co/SDEz7NvmoDyB.png
EDIT END 1 18 09 2023

Thanks

[marijavik]

Hi @booomji

just make sure that you take DNA and body file from the same MetaHuman. And do prepare skeleton file to contains the same joints we have in body folder. In that case you should have corresponding height for both body and head. Do not take any of those predefined skeletons as the most likely they won't fit. Those are skeletons that fit Ada and Taro DNA files and can be used with those.

[booomji]

Hi @booomji

Do not take any of those predefined skeletons as the most likely they won't fit. Those are skeletons that fit Ada and Taro DNA files and can be used with those.

Thank you mariJ.
Just to be sure I understood you correctly.

1. Take the DNA and body mesh from the metahuman file downloaded to my quixel bridge location on hard drive
2. Prepare the skeleton file after referencing / studying the structure present in the body-folder
3. Do not literally take the body joints from the body-folder
4. "Do not take any of those predefined skeletons as the most likely they won't fit" - same as point no.3 above ? To not literally use the Ada / taro skeleton ?

Thanks for the clarification
b

[marijavik]

Yes, that is correct. In body folder we prepared bodies for Ada and Taro and I guess your is different, so you need to prepare the one you have downloaded from bridge.

[booomji]

Yes, that is correct. In body folder we prepared bodies for Ada and Taro and I guess your is different, so you need to prepare the one you have downloaded from bridge.

Thank you MariJ.

Finally, do we have the ability to add blend shapes via the DNA calib commands ?
I know there is a SetBlendShapeTargetDeltasCommand . I am assuming this will only change the current blend shape and not add another blendshape to the list.

For e.g If i want to animate a Pinocchio nose in the sequencer (or BP) I could

1. Register a blend shape in the DNA
2. Export to UE
3. Add a slider to the face board control rig in UE (if possible)
4. Trigger the long nose blend shape via the face board slider or another custom slider in the UI

Possible to do procedurally (dna toolset)?
Right now I'm adding them via the steps shown in the tutorials below but it'd be nice to have this functionality in the DNA suite of tools.
Adding blend shapes & interactively modifying them in the editor
Adding blend shapes to metahumans

Thanking you,
b

[marijavik]

Hello @booomji,

Finally, do we have the ability to add blend shapes via the DNA calib commands ?
I know there is a SetBlendShapeTargetDeltasCommand . I am assuming this will only change the current blend shape and not add another blendshape to the list.

Unfortunately, we do not have an option to add new blend shapes, only to update existing with SetBlendShapeTargetDeltasCommand. So you are right.

[booomji]

Hello @booomji,

Finally, do we have the ability to add blend shapes via the DNA calib commands ?
I know there is a SetBlendShapeTargetDeltasCommand . I am assuming this will only change the current blend shape and not add another blendshape to the list.

Unfortunately, we do not have an option to add new blend shapes, only to update existing with SetBlendShapeTargetDeltasCommand. So you are right.

Thank you Marij. You have been super helpful.
I hope Epic bring you on their channel for a deep dive on this topic for all the devs out there.
Cheers,
b

[marijavik]

If you mean at UnrealFest for the talk MetaHuman: DNA Calibration Deep Dive - I will be there :)

[booomji]

If you mean at UnrealFest for the talk MetaHuman: DNA Calibration Deep Dive - I will be there :)

WOW this is fantastic news !
I hope the talk will be recorded. It will be an invaluable resource.
Cheers,
b

[booomji]

Hello MariJ.
Was your session recorded ?
Very eager to view it.

Cheers,
b

[marijavik]

Hello @booomji,

It has been recorded and we expect it soon to be available on YT.

[booomji]

Hello @booomji,

It has been recorded and we expect it soon to be available on YT.

Hello Mari J.
The video has still not been uploaded. Please can you ask the team to up them soon.
Thanks so much.
b

[marijavik]

Hello @booomji,

The video has been uploaded today - link

[ribhav99]

@marijavik Sorry I couldn't find a better place to post this, if there is please let me know and I will.

I want to generate random metahumans programatically. I think I can do this by editing metahuman DNA files but I'm not sure how to go about this. I saw your talk at Unreal fest and read through the examples and github docs and code but couldn't find a way to do it.

My working assumption rn is that if I can find the vertices corresponding to different parts of the face, like mouth, nose, eyes, etc, I can just scale them randomly (within reason) to create an entirely new metahuman BUT I'm not sure how to find the relevant vertices for each part of the face.

Any help would be appreciated

[booomji]

Hello @booomji,

The video has been uploaded today - link

omg omg this is AMAZING !
Thank you so much.

b