Unable to reproduce amazing results using the inference.py script

[gohar-malik]

Hi Author @pgt4861,
Thanks for your work. I am a big fan of your paper and was really impressed by how well it uses the trimap information using prior tokens.
I was experimenting with the code but was unable to reproduce the results in terms of quality. Below are some of the examples that I ran.
In each row, from left to right, we have the original image, the trimap, the alpha mask predicted by matteformer and the final matted image.
As you can see, the output is not as good as the model can perform. Specifically in the unknown regions of the trimap, like around the fingers of the boy, the feet of the horse and the neck of the table lamp.
Can you please guide me how to improve these results?
P.S: I am generating these trimaps automatically from a segmentation mask using the Euclidean distance transform.

[hackkhai]

Hey, @gohar-malik i can help you with this contact me at khailashsanthakumar@gmail.com

[pgt4861]

Hi @gohar-malik
Thank you for you interest on our work.
I think there are some ways to improve the results on real-world applications.

First, there are some domain gaps between real world image and composited images (train data). So if the model is so overfitted to the composited (synthetic) data, it may work not well on the real-world images.
It seems that some augmentation method can be helpful to deal with the problem.

Second, the number of foreground objects in the train data is not enough to cover the diversity of real world objects.
So, I recommend you to collect foregrounds (with good ground truth) as many as possible for your target problem.

Thank you