Weights from DETR

[ASMIftekhar]

Thanks a lot for releasing DETR weights for V-COCO. I can see the class embeds weight and bias have 81 classes. I am considering first 80 classes matched with the original V-COCO annotations. Correct me if I am wrong.

[fredzzhang]

Hi @ASMIftekhar,

Thanks for taking an interest in our work.

Yes, you are correct in assuming that the first 80 classes are object classes while the last one stands for background. This is the standard notation for DETR, which is different from the Faster R-CNN line of work, where the first class stands for background.

Cheers,
Fred.

[ASMIftekhar]

Thanks for the response, I actually tried to run inference with the weights given by you. Unfortunately, I am getting very bad results. I created a DETR model with 80 classes and directly loaded your model. So I am a bit confused about the correspondence here. Lets say, your 0th index class of class embeds, does it correspondence to the person class? Should I load the weights in a 91 class original DETR model?

[fredzzhang]

Yes, index 0 should correspond to person.

I think I implemented two options in the hicodet repo. You can either load pre-trained weights with 91 classes or fine-tuned weights with 80 classes. As a reference, here is some code (line 140-160).

...
    # Load model and loss function
    detr, criterion, postprocessors = build_model(args)
    class_embed = torch.nn.Linear(256, 81, bias=True)
    if os.path.exists(args.pretrained):
        print(f"Load pre-trained model from {args.pretrained}")
        detr.load_state_dict(torch.load(args.pretrained)['model_state_dict'])
        w, b = detr.class_embed.state_dict().values()
        keep = [
            1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21,
            22, 23, 24, 25, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
            43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61,
            62, 63, 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 84,
            85, 86, 87, 88, 89, 90, 91
        ]
        class_embed.load_state_dict(dict(
            weight=w[keep], bias=b[keep]
        ))
    detr.class_embed = class_embed
    if os.path.exists(args.resume):
        print(f"Resume from model at {args.resume}")
        detr.load_state_dict(torch.load(args.resume)['model_state_dict'])
...

Basically, the flag args.pretrained should only be used to specify the pre-trained weights (91 classes) whereas args.resume should only be used to specify the fine-tuned weights (80 classes).

Let me know if this clears things up.

Fred.

[ASMIftekhar]

Yes it clears the confusion and I got a reasonable result (shown below for resnet 50) after loading the model with 91 classes. However, I would suggest you to provide the model with 91 classes instead of 81 classes as then people can directly utilize your weights in their own DETR repo. Anyway, thanks a lot for your good work. It saved lot of computation time for me.

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.381 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.589 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.404 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.162 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.466 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.681 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.335 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.503 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.535 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.260 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.642 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.850