coco training and evaluation on custom ds

[avber]

Hi @leondgarse ,

I converted a custom ds (#78)
and run the training script (EfficientDetD0) with default parameters.

!CUDA_VISIBLE_DEVICES='0' python ./coco_train_script.py -p adamw -b 4 -i 512 --data_name data.json

Initially the following weights were loaded:
Load pretrained from: .keras/models/efficientnetv2/efficientnetv1-b0-noisy_student.h5

After unfreezing backbone 2 files were loaded

Load pretrained from: .keras/models/efficientnetv2/efficientnetv1-b0-noisy_student.h5

features: {'stack_2_block1_output': (None, 64, 64, 40), 'stack_4_block2_output': (None, 32, 32, 112), 'stack_6_block0_output': (None, 16, 16, 320)}
Load pretrained from: checkpoints/EfficientDetD0_512_adamw_data.json_batchsize_4_randaug_6_mosaic_0.5_color_random_hsv_position_rts_lr512_0.008_wd_0.02_anchors_mode_None_latest.h5

Is it normal?

After training coco_eval_script.py produced an error

!CUDA_VISIBLE_DEVICES='1' python ./coco_eval_script.py --data_name data.json -m /kecam/checkpoints/EfficientDetD0_512_adamw_data.json_batchsize_4_randaug_6_mosaic_0.5_color_random_hsv_position_rts_lr512_0.008_wd_0.02_anchors_mode_None_epoch_105_val_cls_acc_1.0000.h5

File "./coco_eval_script.py", line 122,
lambda datapoint: (*image_process(datapoint["image"]), tf.shape(datapoint["image"])[:2], datapoint["image/id"])
KeyError: 'image/id'

=======================

When trying manual inference I got

pretrained = '/kecam/checkpoints/EfficientDetD0_512_adamw_data.json_batchsize_4_randaug_6_mosaic_0.5_color_random_hsv_position_rts_lr512_0.008_wd_0.02_anchors_mode_None_epoch_105_val_cls_acc_1.0000.h5'

mm = efficientdet.EfficientDetD0(input_shape=(512, 512, 3), num_classes=1)
mm.load_weights(pretrained)

Load pretrained from: .keras/models/efficientnetv2/efficientnetv1-b0-noisy_student.h5
features: {'stack_2_block1_output': (None, 64, 64, 40), 'stack_4_block2_output': (None, 32, 32, 112), 'stack_6_block0_output': (None, 16, 16, 320)}
Load pretrained from: .keras/models/efficientdet_d0_512_coco.h5
WARNING:tensorflow:Skipping loading weights for layer #450 (named classifier_head) due to mismatch in shape for weight classifier_head/pointwise_kernel:0. Weight expects shape (1, 1, 64, 9). Received saved weight with shape (1, 1, 64, 810)
WARNING:tensorflow:Skipping loading weights for layer #450 (named classifier_head) due to mismatch in shape for weight classifier_head/bias:0. Weight expects shape (9,). Received saved weight with shape (810,)
===================

from PIL import Image
import numpy as np
import kecam

mm = efficientdet.EfficientDetD0(input_shape=(512, 512, 3), num_classes=1, pretrained=pretrained)
img_file = '..//testimages/45_gxdic.jpg'

im = Image.open(img_file)
imm2 = np.array(im)
preds = mm(mm.preprocess_input(imm2))
bboxs, lables, confidences = mm.decode_predictions(preds)[0]
kecam.coco.show_image_with_bboxes(imm2, bboxs, lables, confidences)

when loading weights this way 2 sets of weights were loaded and there were no predictions

Load pretrained from: .keras/models/efficientnetv2/efficientnetv1-b0-noisy_student.h5
features: {'stack_2_block1_output': (None, 64, 64, 40), 'stack_4_block2_output': (None, 32, 32, 112), 'stack_6_block0_output': (None, 16, 16, 320)}
Load pretrained from: /kecam/checkpoints/EfficientDetD0_512_adamw_data.json_batchsize_4_randaug_6_mosaic_0.5_color_random_hsv_position_rts_lr512_0.008_wd_0.02_anchors_mode_None_epoch_105_val_cls_acc_1.0000.h5

The problem might be the score threshold. How do I pass it when making predictions?
Also, how can I specify more than 100 ground truth objects on one image when running coco_train_script.py?

Any help is appreciated.

Edit:
score_threshold=0.1 didn't help.

bboxs, lables, confidences = mm.decode_predictions(preds, score_threshold=0.1)[0]

[leondgarse]

• The 2 weight files loading is normal, as for EfficientDet, backbone EfficientNet weights is loaded anyway, but also, this can be fixed. [TODO]
• coco_eval_script.py is using pycocotools for AP AR results. Currently not supporting custom datasets, and yes, this needs a fix... [TODO]
• When loading pretrained, the second method mm = efficientdet.EfficientDetD0(input_shape=(512, 512, 3), num_classes=1, pretrained=pretrained) is preferred.
• max_labels_per_image can be added as a new parameter for coco_train_script.py. [TODO]
• Regarding all these warnings, the final prediction should work after training. I need a test on local for checking. [TODO]

[avber]

Did another round of training with augmentations turned off.

preds = mm(mm.preprocess_input(imm2))
bboxs, lables, confidences = mm.decode_predictions(preds, score_threshold=0.01)[0]

produced 0 bboxs.

However, preds.shape is [1, 49104, 5]

dd = coco.decode_bboxes(preds[0], anchors).numpy()

Maximum score in preds is 0.00593921 (dd[:, 4:])

[leondgarse]

Just fixed some issues:

• Added a parameter indices_2_labels for coco.data.show_batch_sample. Use like show_batch_sample(..., indices_2_labels={0: "foo"}).
• The plt.show() issue displaying an image with empty box using jupyter is also fixed.
• EfficientDet with not None pretrained will not load EfficientNet backbone weights.
• coco_eval_script.py using custom dataset is now supported.
• max_labels_per_image added as a parameter for coco_train_script.py.
• Finally, this Colab kecam_coco_tiny_test.ipynb is a basic test of these full process including show / training / evaluating / predicting. The dataset coco_tiny is created from COCO dataset with only bicycle class, means this dataset is also with num_classes=1.

Anyway I think at least your prediction should work, most of these fixes are related with evaluating process. You may also try if reloading entire COCO pretrained weights works with --pretrained coco.

[avber]

Thank you!

I've found the culprit. When converting from coco I put category_id into indices_2_labels={1: "foo"}. category_ids start from 1. Setting indices_2_labels={0: "foo"} fixed the training and validation issues. Checking for such situation (when index doesn't start from 0) might help others.

3 small issues with the training script.

There might be an option to speed up validation slightly. I got multiple messages from pycocotools like

creating index...
index created!

COCO AP eval start_epoch: 21, frequency: 1
It's actually 21 + 1

When using on premises jupyter notebook, epoch losses output is messed up. It's not happening in your colab notebook.
tensorflow 2.10.0
jupyter 1.0.0
jupyter_client 7.4.2
jupyter-console 6.4.4
jupyter-core 4.11.1
jupyter-server 1.21.0
jupyterlab-pygments 0.2.2
jupyterlab-widgets 3.0.3

====================================

In case you decide to include coco format conversion in your library or if someone has similar needs, here is the working code:

`import json
import os
import pycocotools
from pycocotools.coco import COCO as _COCO
#https://github.com/ViTAE-Transformer/ViTDet/blob/main/mmdet/datasets/coco.py
class COCO(_COCO):

def __init__(self, annotation_file=None):
    super().__init__(annotation_file=annotation_file)
    self.img_ann_map = self.imgToAnns
    self.cat_img_map = self.catToImgs

def get_ann_ids(self, img_ids=[], cat_ids=[], area_rng=[], iscrowd=None):
    return self.getAnnIds(img_ids, cat_ids, area_rng, iscrowd)

def get_cat_ids(self, cat_names=[], sup_names=[], cat_ids=[]):
    return self.getCatIds(cat_names, sup_names, cat_ids)

def get_img_ids(self, img_ids=[], cat_ids=[]):
    return self.getImgIds(img_ids, cat_ids)

def load_anns(self, ids):
    return self.loadAnns(ids)

def load_cats(self, ids):
    return self.loadCats(ids)

def load_imgs(self, ids):
    return self.loadImgs(ids)

class CocoDataset():

def load_annotations(self, ann_file):
        """Load annotation from COCO style annotation file.
        Args:
            ann_file (str): Path of annotation file.
        """



        self.coco = COCO(ann_file)
        
        self.img_ids = self.coco.get_img_ids()
        data_infos = []
        total_ann_ids = []

        def convertBbox(info, bbox):
            height = info['height']
            width = info['width']
            x_min = bbox[0]/width
            x_max = (bbox[0] + bbox[2]) / width
            y_min = bbox[1] / height
            y_max = (bbox[1] + bbox[3]) / height
            return [y_min, x_min, y_max, x_max]
        
        for i in self.img_ids:
            info = self.coco.load_imgs([i])[0]
            info['image'] = info['file_name']
            
            data_infos.append(info)
            ann_ids = self.coco.get_ann_ids(img_ids=[i])
            
            ann_info = self.coco.load_anns(ann_ids)

            objects = {}
            objects['label'] = [x['category_id']-1 for x in ann_info]
            objects['bbox'] = [convertBbox( info, x['bbox'] ) for x in ann_info]
            info['objects'] = objects
            
            #cleanup
            del info['file_name']
            del info['height']
            del info['width']
            del info['id']
            
            total_ann_ids.extend(ann_ids)

        assert len(set(total_ann_ids)) == len(total_ann_ids), f"Annotation ids in '{ann_file}' are not unique!"
        return data_infos`

`ds_train = CocoDataset()
train = ds_train.load_annotations('train_d155c6d48da36cf8f5f39f5ad77b0b9d_annotations.json')

ds_validation = CocoDataset()
validation = ds_validation.load_annotations('val_d155c6d48da36cf8f5f39f5ad77b0b9d_annotations.json')

cats = ds_train.coco.dataset['categories']
indices_2_labels = {}
for cat in cats:
indices_2_labels[cat['id']-1] = cat['name']

num_classes = len(indices_2_labels)

final_data = {
'train': train,
'validation': validation,
'indices_2_labels': indices_2_labels,
'info': {'num_classes': num_classes}
}

with open('data.json', 'w') as f:
json.dump(final_data, f)`

[leondgarse]

• indices_2_labels actually should not be used during training or prediction, only takes some effect within the latest update during evaluation, but still not a required field. That's also a reason why I think at least your prediction should work.
• Jupyter log issue may be caused by print(..., end="\r"), may check it latter.
• Uh the COCO format! I also done a conversion on my local while creating coco_tiny dataset... Thanks for your sharing. :)