Questions about OOD Finetuning

[pieris98]

Hey again Nazir, sorry for double posting on the Issues.
I have a couple of questions about OOD Finetuning:

1. Do outlier masks need to have a specific value for the outlier label? e.g. in Synboost I saw they assign the value 20 as the "anomaly" label (since it is >19, the number of classes in cityscapes).
2. Is there a script for fine-tuning separate than train_net.py or is OOD fine-tuning just the process of training from a checkpoint model on an OOD dataset?
3. Do I need to register the OOD dataset similar to inlier training datasets i.e. as described here? Or is there a different procedure for Outlier datasets e.g. similar to evaluation datasets (define dataloader class in datasets folder -> define config in support.py)?

[NazirNayal8]

hi @pieris98 ,

1. We handle this in the code by adding a separate outlier_mask entry for each sample in the dataset mapper here, which is a binary mask that has 1 where outlier exists and 0 otherwise. You can see here how the outlier mask is used in the loss.
2. No, train_net is the only script used for any training or fintunening. The main thing that separates training from finetuning is the config file, which specified the initial weights and the finetuning parameters (num iterations ...etc)
3. The procedure is different whether you are using the dataset for training or for evaluation. For training, you have to register the dataset if it is an entirely new dataset following the detectron2 format. You can also follow some examples that already exist in the repo. However, in the repo we only use OoD datasets for evaluation, so the coming sentence is probably of more interest to you. For evaluation, we use our own dataset class objects as you can see they are created and prepared in support.py and each of their individual implementations are present under datasets. Hence if you want to add a new OoD dataset for evaluation you are flexible to implement it the way you want without registration, as long as its behavior is similar to the ones already defined and I think the script evaluate_ood.py and all dataset files under datasets/ can provide a clear idea on how to structure your own evaluation dataset.

I hope this was helpful. If you need any more clarifications please let us know.

[pieris98]

I've already implemented a CustomDataset dataloader similar to how RoadAnomaly21 is loaded. Once I make it pretty I'll make a PR, if you're open to it.
For now, I'll try to follow the (complicated) detectron2 instructions and I'll keep this open for some days in case I need any help.

[pieris98]

Hey Nazir,
I'm now in the process of training RbA with COCO outlier supervision.

I have 2 questions:

1. I noticed that in all configs (including the ones for OOD supervision), it seems the OOD prediction head/part is frozen.

  FREEZE_BACKBONE: true
  FREEZE_PIXEL_DECODER: true
  FREEZE_TRANSFORMER_DECODER: false
  FREEZE_TRANSFORMER_DECODER_EXCEPT_MLP: true
  FREEZE_TRANSFORMER_DECODER_EXCEPT_MLP_AND_OOD_PRED: false
  FREEZE_TRANSFORMER_DECODER_EXCEPT_OBJECT_QUERIES: false

Correct me if I'm wrong, but aren't the 4 last lines mutually exclusive? i.e. only one of them can be true and the rest false. This means that given this config (which is the config of swin_b_1dl_rba_ood_coco with OOD supervision, the OOD_PRED is frozen.

2. When we're training with OOD supervision, there's a hyperparameter OOD_PROB (p_out in the paper) which dictates if a COCO mix OOD mask is cut-pasted inside the training image. So, if the probability threshold is not met i.e. we don't mix OOD masks, is there any training done at all? Is it the same as inlier training?
   I understand the RbA loss in the paper Section 3.4 and that we're only using it for the set of outlier pixels.
   However, I'm confused about what happens with the inlier pixels.
   So, to summarize: I understand that we either mix masks in the image with the OOD label (254) or don't mix depending on the random probability being >=OOD_PROB or < OOD_PROB, but I'm not sure if all of the inlier masks/classes are included in the training objective in each of these 2 cases.

Thank you again for all your help and looking forward to your help feedback!

[NazirNayal8]

Hi @pieris98

1. Here I need to clarify that FREEZE_TRANSFORMER_DECODER_EXCEPT_MLP_AND_OOD_PRED: is not related to RbA experiments, but rather it is used in the experiments where we finetuned Mask2former with the DenseHybrid method in our ablation experiments (Table 3 in the paper). That method has an OoD head and for fairness we added this option in order to make sure that the OoD head is finetuned as well. Hence this term does not hold any importance in RbA experiments. For the rest, if you check the mask2former/config.py file you would see

cfg.MODEL.FREEZE_BACKBONE = False
cfg.MODEL.FREEZE_PIXEL_DECODER = False
cfg.MODEL.FREEZE_TRANSFORMER_DECODER = False
cfg.MODEL.FREEZE_TRANSFORMER_DECODER_EXCEPT_MLP = False # This takes precedence over FREEZE_TRANSFORMER_DECODER
cfg.MODEL.FREEZE_TRANSFORMER_DECODER_EXCEPT_OBJECT_QUERIES = False # This takes precedence over FREEZE_TRANSFORMER_DECODER_EXCEPT_MLP

in which we highlight that there is some precedence defined when any of the similar options are turned on.

2. Here, OOD_PROB, is used separately for each image. This means that for each image that is loaded for finetuning, an OoD object is randomly sampled and pasted on it in a random location with probability OOD_PROB. This means that in expectation, OOD_PROB of the samples in the finetuning step will contain pasted OoDs.

[pieris98]

Thanks a lot for the detailed feedback on a weekend Nazir!
I had no clue about the EXCEPT_OOD part referring to DenseHybrid, I assumed everything was for RbA.

A follow-up about OOD_PROB:
For those samples that are not mixed, are they still trained as inlier images?
I saw that in the OOD config there is a different (weighted) loss nls but I don't understand what it's doing well in the code.

Again thank you for all your valuable support!