How to train the interaction head individually

[xiaoxiaoczw]

Hi Fred,

I would like to know that, how to freeze detector weights to train the interaction head individually? Thanks!

[fredzzhang]

Hi @xiaoxiaoczw,

The detector weights are in fact frozen already. We fine-tuned the object detector as a first stage and only train the interaction head next. The corresponding code is here.

    for p in upt.detector.parameters():
        p.requires_grad = False
    param_dicts = [{
        "params": [p for n, p in upt.named_parameters()
        if "interaction_head" in n and p.requires_grad]
    }]

Fred.

[xiaoxiaoczw]

Hi Fred,

Thanks for your quick feedback!
By the way, which pretrained weights do I need to load if I want to train on my own dataset?

[fredzzhang]

To get the best performance, you would want to fine-tune the object detector on the custom dataset first, and then train the interaction head. But if the custom dataset has the same set of objects and you don't want to go through the trouble, you can just load the provided detector weights. You can choose amongst the pre-trained MS COCO weights, fine-tuned HICO-DET weights and the fine-tuned V-COCO weights. Which one works the best for you really depends on the dataset similarity. It would be hard to guess without running experiments.

Fred.

[xiaoxiaoczw]

Thanks for the suggestion!

[xiaoxiaoczw]

Hi @fredzzhang,

I have already fine-tune the object detector on the custom dataset, but when I train UPT with this fine-tuned weights, I tried various batch sizes like 4/8/16, always showing 'The HOI loss is NaN', could you please give me some advice in this case? Thanks!

[fredzzhang]

Hi @xiaoxiaoczw,

That is quite a nasty problem. I presume you have already look at #35. If none of the suggested measures worked, we will have to make a compromise. The core of the problem is the unstable statistics caused by taking the log of some spatial features. Removing the log will slightly reduce the performance, but there are probably no other solutions now.

Go to the script ops.py, navigate to line 440 and make the following change.

#        features.append(
#            torch.cat([f, torch.log(f + eps)], 1)
#        )

        features.append(f)

Then go to line 216 in interaction_head.py, and change the input dimension from 36 to 18. This should solve the problem.

Fred.

[xiaoxiaoczw]

Hi @fredzzhang,

I did try all the methods in #35, and the one you mentioned this time, but I still get 'The HOI loss is NaN', always because the spatial_head operation makes box_pair_spatial value all NaN.

[fredzzhang]

Hi @xiaoxiaoczw,

Sorry for the late reply. I was away for a few weeks.

I'm quite surprised removing the log values didn't help. If the spatial features are still returning nan values, it is likely that there might be boxes with extreme dimensions. Can you add some sanity check in the computation of spatial features (compute_spatial_encodings in ops.py) to catch nan values?

If there aren't any nan values returned by that function, it would be a much more tricky problem because that suggests this particular construct of spatial encodings will cause unstable training on your custom dataset. You would need to find the specific entries that resulted this (most likely the aspect ratios or other terms with division in it).

Fred.

[xiaoxiaoczw]

Hi @fredzzhang ,

Thanks for all the suggestions!