Training the semantic head together with instance head

[abhinavagarwalla]

Thank you for open-sourcing your work.

Did you try the no-fix setting in cfg-train.py? What was the takeaway there? Is it better the complete network end-to-end or just fine-tune the regression heads as done for the released models?
I tried to run this setting, but ran into an error:
RuntimeError: Function 'SubMConvFunctionBackward' returned nan values in its 1th output.

[hongfz16]

Thank you for your interest.

In the experiments, I choose to just fine-tune the instance head only for reducing the GPU memory usage. I am not quite sure what would happen if the whole network is trained end-to-end.

As for the error, since I haven't run into this one before, it is hard for me to help you with the limited error log. But my guess is that the dynamic shifting module is not very stable. And it might produce very large or small numbers during training, which would lead to nan in the gradient.