Evaluation metrics (Outcomes like Precision, Recall, mAP etc) for YOLO-NAS are not satisfactory comparing to v5 & v8.

[rexxsd]

I was trying to benchmark(ing) the models (YOLO : v5, v8 & NAS) while training them on same dataset, the evaluation metrics for v5 & v8 were satisfactory as mentioned below:

• YOLOv5: Model Summary:
  Precision: 0.914 | Recall : 0.969 | mAP@50: 0.968 | mAP90: 0.625

• YOLOv8: Model summary (fused): 168 layers, 11125971 parameters, 0 gradients, 28.4 GFLOPs
  Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.42it/s]
  all 215 230 0.966 0.948 0.98 0.781

But the outcomes for YOLO-NAS was not even close to them where I got the following outcomes:

• YOLO-NAS

SUMMARY OF EPOCH 273
├── Training
│ ├── Ppyoloeloss/loss = 1.3271
│ │ ├── Best until now = 1.3328 (↘ -0.0057)
│ │ └── Epoch N-1 = 1.3328 (↘ -0.0057)
│ ├── Ppyoloeloss/loss_cls = 0.6218
│ │ ├── Best until now = 0.6233 (↘ -0.0014)
│ │ └── Epoch N-1 = 0.6256 (↘ -0.0038)
│ ├── Ppyoloeloss/loss_dfl = 0.8454
│ │ ├── Best until now = 0.8469 (↘ -0.0016)
│ │ └── Epoch N-1 = 0.8469 (↘ -0.0016)
│ └── Ppyoloeloss/loss_iou = 0.113
│ ├── Best until now = 0.1135 (↘ -0.0004)
│ └── Epoch N-1 = 0.1135 (↘ -0.0004)
└── Validation
├── F1@0.50 = 0.0982
│ ├── Best until now = 0.1203 (↘ -0.0221)
│ └── Epoch N-1 = 0.1097 (↘ -0.0115)
├── Map@0.50 = 0.9115
│ ├── Best until now = 0.9312 (↘ -0.0197)
│ └── Epoch N-1 = 0.9246 (↘ -0.013)
├── Ppyoloeloss/loss = 1.8552
│ ├── Best until now = 1.8142 (↗ 0.0409)
│ └── Epoch N-1 = 1.8432 (↗ 0.012)
├── Ppyoloeloss/loss_cls = 0.8396
│ ├── Best until now = 0.7983 (↗ 0.0414)
│ └── Epoch N-1 = 0.8198 (↗ 0.0198)
├── Ppyoloeloss/loss_dfl = 1.1295
│ ├── Best until now = 1.0111 (↗ 0.1184)
│ └── Epoch N-1 = 1.1407 (↘ -0.0112)
├── Ppyoloeloss/loss_iou = 0.1803
│ ├── Best until now = 0.1628 (↗ 0.0175)
│ └── Epoch N-1 = 0.1812 (↘ -0.0009)
├── Precision@0.50 = 0.0516
│ ├── Best until now = 0.064 (↘ -0.0124)
│ └── Epoch N-1 = 0.058 (↘ -0.0064)
└── Recall@0.50 = 1.0
├── Best until now = 1.0 (= 0.0)
└── Epoch N-1 = 0.9957 (↗ 0.0043)

Whats the reason behind this, why these values are having such a huge differences in NAS as compare to v5 & v8, what are the factors affecting these difference.If any one can help regarding this it would be helpful .

@yeldarby @hansent @tonylampada

[SkalskiP]

Hi @rexxsd 👋🏻! That looks very interesting. Do you have a single table that would sum up your results?

[rexxsd]

No I created a separate one.

[rexxsd]

But I took all the snippets from your notebook itself.

[SkalskiP]

It is pretty hard for me to help you without more details :( Overall the subject is interesting, and I'd like to understand if YOLO-NAS is better or worst than YOLOv8. I trained YOLO-NAS on my custom dataset, and the model overall trained properly. I didn't make a proper model comparison, but in your case (~ 5% precision) model is basically useless.

[rexxsd]

Thank you for your valuable response, If you can suggest me some aspects to explore that may affect the mentioned outcomes, That would be helpful.

[SkalskiP]

Can you share your training code with me?