landmarks condition is stronger than flow condition

[YunjieYu]

Hi,
I find that your FlowControlNet injects landmarks except flow. Through my experiments, I find that landmarks condition is actually stronger than flow condition. Below is my result:

output_flow.mp4

This is the result only using flow condition.

output_ldmk.mp4

This is the result only using landmark condition.

output_flow_ldmk.mp4

This is the result using landmark and flow condition simultaneously.

Can you explain this result? How can we get good results using only flow?

[MyNiuuu]

Hi! Thanks for your interest!

To make the model work with only motion flow inputs, you'll need to retrain it. The current model relies on both motion flow and landmarks, so it won't function correctly if either input is missing.

We have trained two versions of our landmark-based model: one that uses landmarks as conditional inputs and one that doesn't. We found that using only motion flow can already achieve control, but incorporating landmarks further stabilizes and enhances controllability, and this does not violate with the hybrid control mechanism of our entire model.

Also, we will release the training codes soon after we finished submitting the camera-ready of ECCV.

[MyNiuuu]

We have uploaded the training code.