Skip to content

Latest commit

 

History

History
262 lines (210 loc) · 10.3 KB

Model-Results.md

File metadata and controls

262 lines (210 loc) · 10.3 KB

AI-enabled School Mapping Model Results

Table of Contents
  1. Benin (BEN)
  2. Botswana (BWA)
  3. Ghana (GHA)
  4. Kenya (KEN)
  5. Malawi (MWI)
  6. Mongolia (MNG)
  7. Mozambique (MOZ)
  8. Namibia (NAM)
  9. Rwanda (RWA)
  10. Senegal (SEN)
  11. South Sudan (SSD)
  12. Tajikistan (TJK)
  13. Zimbabwe (ZWE)

Model Results

This document provides a detailed summary of the performance metrics for models used in our AI-enabled school mapping work.

Key Metrics

  • Area Under the Precision-Recall Curve (AUPRC): Optimizes the model's precision and recall performance at all thresholds
  • F2 Score: Emphasizes recall more than precision, prioritizing the identification of all schools.

The models are trained to maximize the AUPRC, ensuring robust performance at all possible thresholds.

Using the best (non-ensemble) model, best Class Activation Mapping (CAM) method is selected based on the largest confidence drop following perturbation of the top 10% of pixels, as evaluated using the ROAD framework.

CAMs are generated only for images with probability scores exceeding the threshold that optimizes the validation set F2 score. The final F2 scores on the test set are reported below.

Benin (BEN)

Best Models Test AUPRC
swin_v2_s 0.983
vit_h_14 0.978
convnext_base 0.977
vsc-ensemble 0.998
Best CAM Method Probability Threshold F2 Score
GradCamElementWise 0.366 0.982

Botswana (BWA)

Best Models Test AUPRC
vit_l_16 0.989
convnext_large 0.985
swin_v2_t 0.984
vsc_ensemble 0.997
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.352 0.960

Ghana (GHA)

Best Models Test AUPRC
swin_v2_s 0.931
vit_h_14 0.930
convnext_small 0.928
vsc-ensemble 0.991
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.386 0.968

Kenya (KEN)

Best Models Test AUPRC
convnext_small 0.916
swin_v2_b 0.910
vit_b_16 0.906
vsc_ensemble 0.966
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.395 0.966

Malawi (MWI)

Best Models Test AUPRC
convnext_base 0.969
vit_h_14 0.967
swin_v2_s 0.962
vsc_ensemble 0.983
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.335 0.953

Mongolia (MNG)

Best Models Test AUPRC
vit_b_16 0.950
convnext_base 0.944
swin_v2_b 0.935
vsc_ensemble 0.991
Best CAM Method Prob Threshold F2 Score
Hirescam 0.570 0.938

Mozambique (MOZ)

Best Models Test AUPRC
convnext_small 0.969
swin_v2_b 0.968
vit_h_14 0.965
vsc_ensemble 0.994
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.377 0.974

Namibia (NAM)

Best Models Test AUPRC
vit_h_14 0.955
convnext_large 0.954
swin_v2_s 0.949
vsc_ensemble 0.980
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.315 0.914

Rwanda (RWA)

Best Models Test AUPRC
vit_h_14 0.983
swin_v2_t 0.982
convnext_base 0.978
vsc_ensemble 0.998
Best CAM Method Prob Threshold F2 Score
GradCam 0.344 0.978

Senegal (SEN)

Best Models Test AUPRC
vit_h_14 0.980
convnext_large 0.978
swin_v2_t 0.967
vsc_ensemble 0.993
Best CAM Method Prob Threshold F2 Score
GradCam 0.355 0.985

South Sudan (SSD)

Best Models Test AUPRC
vit_h_14 0.971
convnext_base 0.964
swin_v2_t 0.964
vsc_ensemble 0.995
Best CAM Method Prob Threshold F2 Score
GradCam 0.378 0.962

Tajikistan (TJK)

Best Models Test AUPRC
convnext_large 0.962
vit_h_14 0.955
swin_v2_s 0.944
vsc_ensemble 0.986
Best CAM Method Prob Threshold F2 Score
GradCamElementWise 0.342 0.967

Zimbabwe (ZWE)

Best Models Test AUPRC
vit_h_14 0.971
convnext_base 0.967
swin_v2_b 0.961
vsc_ensemble 0.996
Best CAM Method Prob Threshold F2 Score
GradCam++ 0.327 0.977