combining models and tflite export

[avber]

Hi @leondgarse ,

I have EffdetLite4 and MobileVit models trained on similar but not identical datasets. MobileVit model achieved more than 94% accuracy.

Could you show an example of how to replace classification head with the trained MobileVit model and export a combined model to tflite ?

The MobileVit model will need the original input image, not the preprocessed one. max_output_size, iou_or_sigma and score_threshold for bbox predictions can be hardcoded.

The intention is to use the combined model in tfjs.

if #82 is also implemented, it would be fantastic.

Thanks

[leondgarse]

Just fixed mobilevit for using as a detection backbone:

from keras_cv_attention_models import efficientdet, mobilevit, model_surgery

bb = mobilevit.MobileViT_V2_050(input_shape=(512, 512, 3), num_classes=0, pretrained=None)
mm = efficientdet.EfficientDetLite4(input_shape=(512, 512, 3), backbone=bb, pretrained=None)
# >>>> features: {'stack3_block3_output': (None, 64, 64, 128), 'stack4_block5_output': (None, 32, 32, 192), 'stack5_block4_output': (None, 16, 16, 256)}

converter = tf.lite.TFLiteConverter.from_keras_model(mm)
open(mm.name + ".tflite", "wb").write(converter.convert())
# 19078400

Converting a tflite is not hard, the hard part lies in writing your custom prediction decoder, also an issue for #82. Previously I've tried making DecodePredictions a layer, but not work. You may refer eval_func.py#L10-L25 for the most basic TF function for decoding prediction, and take some tests. I may try it also if got some time. :)

[avber]

This project has DecodePredictions as a layer. Though I'm not sure it's exportable to tflite.

https://github.com/Burf/TFDetection/blob/main/tfdet/model/postprocess/__init__.py
https://github.com/Burf/TFDetection/blob/main/tfdet/model/postprocess/retina.py

[leondgarse]

• DecodePredictions as a layer is supported. Usage has been updated in Readme#tflite-conversion. Also some tests can be found in colab kecam_test.ipynb, including saving model as saved_mode and TFLite conversion usage. The key parameter is use_static_output=True, that sets layer output shape fixed as [batch, max_output_size, 6], for TFLite seems not supporting dynamic shape on the second dimension.
• Parameters like score_threshold / iou_or_sigma / max_output_size still being custom while using model.decode_predictions(...) directly, and when converting TFLite, they can be set new value, like score_threshold usage in Readme#tflite-conversion. But after converting, I'm not sure how to set them custom, maybe by signature functions? Not familiar with that...

[avber]

I came across signature functions when testing tflite models. I don't have enough info on them either.

Thank you!