Custom Mask2Former model architecture?

[pieris98]

Hey again Nazir,
I'm trying to use the end-to-end RbA architecture (Mask2Former architecture with semantic segmentation heads, class/mask matcher etc.), outside of the detectron2 framework, because it's causing me some problems with video inference.
I wanted to first understand if this is possible at all.

Questions about the repo:

1. In the maskformer folder there is a maskformer.py general class, and in the modeling sub-folder I saw different files for different parts of the model which, if I understand correctly, define PyTorch Module classes and register them to detectron2?

2. I guess this is considered a custom Mask2Former implementation (modified & added layers), that's why you register it to the ModelCatalog. Does that differ significantly from the original Mask2Former architecture? Is it possible to convert it to some other format (e.g. ONNX, vanilla PyTorch) to use within other frameworks? I tried following detectron2's docs and a tutorial for specific models without success. There are several github issues about converting detectron2 models and it seems a complicated and not easily customizable process for Mask2Former-based models.

3. In the tools folder there are some scripts where you import a pretrained SWIN transformer from pytorch to detectron2. Where do these pretrained models come from?

Questions about using the Mask2Former weights outside this repo:

1. I have trained a RbA model end-to-end (using train_net.py) on my data. Is it possible to read the weights from another framework?

2. I found that MMDetection offers several Mask2Former models with different backbones. Could I use any RbA model with this framework?

Once again, thank you for your work and your helpful support in all issues.

[NazirNayal8]

Hi @pieris98

Questions about the repo:

1. For detectron2, you have to register both the backbone and the segmentation head so that you can change them easily using the config. For Mask2Former the pixel decoder and transformer decoder are both registered as part of the sem seg head.
2. All of the registered components of Mask2Former are pytorch modules, so it is straightforward to copy them and use in any other pytorch-based frame-work. Of course you need to consider the input/output format rules and detectron which might be different in other frameworks (e.g. pytorch lightning) but this can be handled easily as well after transfering the classes.
3. These scripts are provided by the original Mask2Former repo because some backbones are initialized from image-net pretrained weights. The modification of the checkpoints is simply updating the keys in the state-dict to match those in detectron2. These models can be accessed and downloaded from here: https://github.com/NazirNayal8/RbA/blob/main/MODEL_ZOO.md

Questions about using Mask2Former weights outside this repo:

1. if the state_dict of the other framework (assuming it's still pytorch) matches this of RbA model then it can be used directly. If there are slight changes in parameter and class names but the structure is the same, then the weights can be used by updating the key names in the state_dict. If the structure of the model is entirely different then it would be tricky, as you need to find the correct one-to-one mapping between the weights the RbA state dict and those in the new framework.
2. Yes, you can adapt the loss function and the training setup and you can replicate the results on any other framework.