Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

theoritical understanding of map results #3187

Open
buzdarbalooch opened this issue May 18, 2019 · 13 comments
Open

theoritical understanding of map results #3187

buzdarbalooch opened this issue May 18, 2019 · 13 comments

Comments

@buzdarbalooch
Copy link

hi i recently used yolo to train the model (video) of approximately 3 mins and 30 seconds , video was a sequence of clips where each clip was 10 to 15 seconds.

i executed the script To see the map values at different iou thresholds. mAP@IoU=50: darknet.exe detector map data/obj.data yolo-obj.cfg backup\yolo-obj_7000.weights

but after subitting the paper to a conference somhow its hard to satisfy reviewers how these values are reached. Mostly they point out. "Results depicted in Table 1, ... high accuracy" is not clear and should be reformulated.".

i am attaching that small paragraph from paper and also the pic attached when i get these results.
results-in-paper
prec-recall

@AlexeyAB if u can kindly an idea how can i satiisfy the reviwers about these results. i shall be greatful.
@AlexeyAB
Copy link
Owner

@buzdarbalooch Hi,

In your image "AP(0.75) means the AP with IoU=0.75"
I think it should be
"AP(0.75) means the AP for True Positives with IoU>=0.75"
or
"AP(0.75) means the AP with IoU_threshold=0.75"

Do they want to know how AP was calculated, or want to know how did you train the model?

And read how mAP and APs are calculated in different datasets:

If you want, you can see this code that calculates mAP and APs:

darknet/src/detector.c

Lines 649 to 1088 in 0ae5191

float validate_detector_map(char *datacfg, char *cfgfile, char *weightfile, float thresh_calc_avg_iou, const float iou_thresh, const int map_points, network *existing_net)
{
int j;
list *options = read_data_cfg(datacfg);
char *valid_images = option_find_str(options, "valid", "data/train.txt");
char *difficult_valid_images = option_find_str(options, "difficult", NULL);
char *name_list = option_find_str(options, "names", "data/names.list");
int names_size = 0;
char **names = get_labels_custom(name_list, &names_size); //get_labels(name_list);
//char *mapf = option_find_str(options, "map", 0);
//int *map = 0;
//if (mapf) map = read_map(mapf);
FILE* reinforcement_fd = NULL;
network net;
//int initial_batch;
if (existing_net) {
char *train_images = option_find_str(options, "train", "data/train.txt");
valid_images = option_find_str(options, "valid", train_images);
net = *existing_net;
remember_network_recurrent_state(*existing_net);
free_network_recurrent_state(*existing_net);
}
else {
net = parse_network_cfg_custom(cfgfile, 1, 1); // set batch=1
if (weightfile) {
load_weights(&net, weightfile);
}
//set_batch_network(&net, 1);
fuse_conv_batchnorm(net);
calculate_binary_weights(net);
}
if (net.layers[net.n - 1].classes != names_size) {
printf(" Error: in the file %s number of names %d that isn't equal to classes=%d in the file %s \n",
name_list, names_size, net.layers[net.n - 1].classes, cfgfile);
getchar();
}
srand(time(0));
printf("\n calculation mAP (mean average precision)...\n");
list *plist = get_paths(valid_images);
char **paths = (char **)list_to_array(plist);
char **paths_dif = NULL;
if (difficult_valid_images) {
list *plist_dif = get_paths(difficult_valid_images);
paths_dif = (char **)list_to_array(plist_dif);
}
layer l = net.layers[net.n - 1];
int classes = l.classes;
int m = plist->size;
int i = 0;
int t;
const float thresh = .005;
const float nms = .45;
//const float iou_thresh = 0.5;
int nthreads = 4;
if (m < 4) nthreads = m;
image* val = (image*)calloc(nthreads, sizeof(image));
image* val_resized = (image*)calloc(nthreads, sizeof(image));
image* buf = (image*)calloc(nthreads, sizeof(image));
image* buf_resized = (image*)calloc(nthreads, sizeof(image));
pthread_t* thr = (pthread_t*)calloc(nthreads, sizeof(pthread_t));
load_args args = { 0 };
args.w = net.w;
args.h = net.h;
args.c = net.c;
args.type = IMAGE_DATA;
//args.type = LETTERBOX_DATA;
//const float thresh_calc_avg_iou = 0.24;
float avg_iou = 0;
int tp_for_thresh = 0;
int fp_for_thresh = 0;
box_prob* detections = (box_prob*)calloc(1, sizeof(box_prob));
int detections_count = 0;
int unique_truth_count = 0;
int* truth_classes_count = (int*)calloc(classes, sizeof(int));
// For multi-class precision and recall computation
float *avg_iou_per_class = (float*)calloc(classes, sizeof(float));
int *tp_for_thresh_per_class = (int*)calloc(classes, sizeof(int));
int *fp_for_thresh_per_class = (int*)calloc(classes, sizeof(int));
for (t = 0; t < nthreads; ++t) {
args.path = paths[i + t];
args.im = &buf[t];
args.resized = &buf_resized[t];
thr[t] = load_data_in_thread(args);
}
time_t start = time(0);
for (i = nthreads; i < m + nthreads; i += nthreads) {
fprintf(stderr, "\r%d", i);
for (t = 0; t < nthreads && i + t - nthreads < m; ++t) {
pthread_join(thr[t], 0);
val[t] = buf[t];
val_resized[t] = buf_resized[t];
}
for (t = 0; t < nthreads && i + t < m; ++t) {
args.path = paths[i + t];
args.im = &buf[t];
args.resized = &buf_resized[t];
thr[t] = load_data_in_thread(args);
}
for (t = 0; t < nthreads && i + t - nthreads < m; ++t) {
const int image_index = i + t - nthreads;
char *path = paths[image_index];
char *id = basecfg(path);
float *X = val_resized[t].data;
network_predict(net, X);
int nboxes = 0;
float hier_thresh = 0;
detection *dets;
if (args.type == LETTERBOX_DATA) {
int letterbox = 1;
dets = get_network_boxes(&net, val[t].w, val[t].h, thresh, hier_thresh, 0, 1, &nboxes, letterbox);
}
else {
int letterbox = 0;
dets = get_network_boxes(&net, 1, 1, thresh, hier_thresh, 0, 0, &nboxes, letterbox);
}
//detection *dets = get_network_boxes(&net, val[t].w, val[t].h, thresh, hier_thresh, 0, 1, &nboxes, letterbox); // for letterbox=1
if (nms) do_nms_sort(dets, nboxes, l.classes, nms);
char labelpath[4096];
replace_image_to_label(path, labelpath);
int num_labels = 0;
box_label *truth = read_boxes(labelpath, &num_labels);
int i, j;
for (j = 0; j < num_labels; ++j) {
truth_classes_count[truth[j].id]++;
}
// difficult
box_label *truth_dif = NULL;
int num_labels_dif = 0;
if (paths_dif)
{
char *path_dif = paths_dif[image_index];
char labelpath_dif[4096];
replace_image_to_label(path_dif, labelpath_dif);
truth_dif = read_boxes(labelpath_dif, &num_labels_dif);
}
const int checkpoint_detections_count = detections_count;
for (i = 0; i < nboxes; ++i) {
int class_id;
for (class_id = 0; class_id < classes; ++class_id) {
float prob = dets[i].prob[class_id];
if (prob > 0) {
detections_count++;
detections = (box_prob*)realloc(detections, detections_count * sizeof(box_prob));
detections[detections_count - 1].b = dets[i].bbox;
detections[detections_count - 1].p = prob;
detections[detections_count - 1].image_index = image_index;
detections[detections_count - 1].class_id = class_id;
detections[detections_count - 1].truth_flag = 0;
detections[detections_count - 1].unique_truth_index = -1;
int truth_index = -1;
float max_iou = 0;
for (j = 0; j < num_labels; ++j)
{
box t = { truth[j].x, truth[j].y, truth[j].w, truth[j].h };
//printf(" IoU = %f, prob = %f, class_id = %d, truth[j].id = %d \n",
// box_iou(dets[i].bbox, t), prob, class_id, truth[j].id);
float current_iou = box_iou(dets[i].bbox, t);
if (current_iou > iou_thresh && class_id == truth[j].id) {
if (current_iou > max_iou) {
max_iou = current_iou;
truth_index = unique_truth_count + j;
}
}
}
// best IoU
if (truth_index > -1) {
detections[detections_count - 1].truth_flag = 1;
detections[detections_count - 1].unique_truth_index = truth_index;
}
else {
// if object is difficult then remove detection
for (j = 0; j < num_labels_dif; ++j) {
box t = { truth_dif[j].x, truth_dif[j].y, truth_dif[j].w, truth_dif[j].h };
float current_iou = box_iou(dets[i].bbox, t);
if (current_iou > iou_thresh && class_id == truth_dif[j].id) {
--detections_count;
break;
}
}
}
// calc avg IoU, true-positives, false-positives for required Threshold
if (prob > thresh_calc_avg_iou) {
int z, found = 0;
for (z = checkpoint_detections_count; z < detections_count - 1; ++z) {
if (detections[z].unique_truth_index == truth_index) {
found = 1; break;
}
}
if (truth_index > -1 && found == 0) {
avg_iou += max_iou;
++tp_for_thresh;
avg_iou_per_class[class_id] += max_iou;
tp_for_thresh_per_class[class_id]++;
}
else{
fp_for_thresh++;
fp_for_thresh_per_class[class_id]++;
}
}
}
}
}
unique_truth_count += num_labels;
//static int previous_errors = 0;
//int total_errors = fp_for_thresh + (unique_truth_count - tp_for_thresh);
//int errors_in_this_image = total_errors - previous_errors;
//previous_errors = total_errors;
//if(reinforcement_fd == NULL) reinforcement_fd = fopen("reinforcement.txt", "wb");
//char buff[1000];
//sprintf(buff, "%s\n", path);
//if(errors_in_this_image > 0) fwrite(buff, sizeof(char), strlen(buff), reinforcement_fd);
free_detections(dets, nboxes);
free(id);
free_image(val[t]);
free_image(val_resized[t]);
}
}
if ((tp_for_thresh + fp_for_thresh) > 0)
avg_iou = avg_iou / (tp_for_thresh + fp_for_thresh);
int class_id;
for(class_id = 0; class_id < classes; class_id++){
if ((tp_for_thresh_per_class[class_id] + fp_for_thresh_per_class[class_id]) > 0)
avg_iou_per_class[class_id] = avg_iou_per_class[class_id] / (tp_for_thresh_per_class[class_id] + fp_for_thresh_per_class[class_id]);
}
// SORT(detections)
qsort(detections, detections_count, sizeof(box_prob), detections_comparator);
typedef struct {
double precision;
double recall;
int tp, fp, fn;
} pr_t;
// for PR-curve
pr_t** pr = (pr_t**)calloc(classes, sizeof(pr_t*));
for (i = 0; i < classes; ++i) {
pr[i] = (pr_t*)calloc(detections_count, sizeof(pr_t));
}
printf("\n detections_count = %d, unique_truth_count = %d \n", detections_count, unique_truth_count);
int* detection_per_class_count = (int*)calloc(classes, sizeof(int));
for (j = 0; j < detections_count; ++j) {
detection_per_class_count[detections[j].class_id]++;
}
int* truth_flags = (int*)calloc(unique_truth_count, sizeof(int));
int rank;
for (rank = 0; rank < detections_count; ++rank) {
if (rank % 100 == 0)
printf(" rank = %d of ranks = %d \r", rank, detections_count);
if (rank > 0) {
int class_id;
for (class_id = 0; class_id < classes; ++class_id) {
pr[class_id][rank].tp = pr[class_id][rank - 1].tp;
pr[class_id][rank].fp = pr[class_id][rank - 1].fp;
}
}
box_prob d = detections[rank];
// if (detected && isn't detected before)
if (d.truth_flag == 1) {
if (truth_flags[d.unique_truth_index] == 0)
{
truth_flags[d.unique_truth_index] = 1;
pr[d.class_id][rank].tp++; // true-positive
} else
pr[d.class_id][rank].fp++;
}
else {
pr[d.class_id][rank].fp++; // false-positive
}
for (i = 0; i < classes; ++i)
{
const int tp = pr[i][rank].tp;
const int fp = pr[i][rank].fp;
const int fn = truth_classes_count[i] - tp; // false-negative = objects - true-positive
pr[i][rank].fn = fn;
if ((tp + fp) > 0) pr[i][rank].precision = (double)tp / (double)(tp + fp);
else pr[i][rank].precision = 0;
if ((tp + fn) > 0) pr[i][rank].recall = (double)tp / (double)(tp + fn);
else pr[i][rank].recall = 0;
if (rank == (detections_count - 1) && detection_per_class_count[i] != (tp + fp)) { // check for last rank
printf(" class_id: %d - detections = %d, tp+fp = %d, tp = %d, fp = %d \n", i, detection_per_class_count[i], tp+fp, tp, fp);
}
}
}
free(truth_flags);
double mean_average_precision = 0;
for (i = 0; i < classes; ++i) {
double avg_precision = 0;
// MS COCO - uses 101-Recall-points on PR-chart.
// PascalVOC2007 - uses 11-Recall-points on PR-chart.
// PascalVOC2010�2012 - uses Area-Under-Curve on PR-chart.
// ImageNet - uses Area-Under-Curve on PR-chart.
// correct mAP calculation: ImageNet, PascalVOC 2010-2012
if (map_points == 0)
{
double last_recall = pr[i][detections_count - 1].recall;
double last_precision = pr[i][detections_count - 1].precision;
for (rank = detections_count - 2; rank >= 0; --rank)
{
double delta_recall = last_recall - pr[i][rank].recall;
last_recall = pr[i][rank].recall;
if (pr[i][rank].precision > last_precision) {
last_precision = pr[i][rank].precision;
}
avg_precision += delta_recall * last_precision;
}
}
// MSCOCO - 101 Recall-points, PascalVOC - 11 Recall-points
else
{
int point;
for (point = 0; point < map_points; ++point) {
double cur_recall = point * 1.0 / (map_points-1);
double cur_precision = 0;
for (rank = 0; rank < detections_count; ++rank)
{
if (pr[i][rank].recall >= cur_recall) { // > or >=
if (pr[i][rank].precision > cur_precision) {
cur_precision = pr[i][rank].precision;
}
}
}
//printf("class_id = %d, point = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, point, cur_recall, cur_precision);
avg_precision += cur_precision;
}
avg_precision = avg_precision / map_points;
}
printf("class_id = %d, name = %s, ap = %2.2f%% \t (TP = %d, FP = %d) \n",
i, names[i], avg_precision * 100, tp_for_thresh_per_class[i], fp_for_thresh_per_class[i]);
float class_precision = (float)tp_for_thresh_per_class[i] / ((float)tp_for_thresh_per_class[i] + (float)fp_for_thresh_per_class[i]);
float class_recall = (float)tp_for_thresh_per_class[i] / ((float)tp_for_thresh_per_class[i] + (float)(truth_classes_count[i] - tp_for_thresh_per_class[i]));
//printf("Precision = %1.2f, Recall = %1.2f, avg IOU = %2.2f%% \n\n", class_precision, class_recall, avg_iou_per_class[i]);
mean_average_precision += avg_precision;
}
const float cur_precision = (float)tp_for_thresh / ((float)tp_for_thresh + (float)fp_for_thresh);
const float cur_recall = (float)tp_for_thresh / ((float)tp_for_thresh + (float)(unique_truth_count - tp_for_thresh));
const float f1_score = 2.F * cur_precision * cur_recall / (cur_precision + cur_recall);
printf("\n for thresh = %1.2f, precision = %1.2f, recall = %1.2f, F1-score = %1.2f \n",
thresh_calc_avg_iou, cur_precision, cur_recall, f1_score);
printf(" for thresh = %0.2f, TP = %d, FP = %d, FN = %d, average IoU = %2.2f %% \n",
thresh_calc_avg_iou, tp_for_thresh, fp_for_thresh, unique_truth_count - tp_for_thresh, avg_iou * 100);
mean_average_precision = mean_average_precision / classes;
printf("\n IoU threshold = %2.0f %%, ", iou_thresh * 100);
if (map_points) printf("used %d Recall-points \n", map_points);
else printf("used Area-Under-Curve for each unique Recall \n");
printf(" mean average precision (mAP@%0.2f) = %f, or %2.2f %% \n", iou_thresh, mean_average_precision, mean_average_precision * 100);
for (i = 0; i < classes; ++i) {
free(pr[i]);
}
free(pr);
free(detections);
free(truth_classes_count);
free(detection_per_class_count);
free(avg_iou_per_class);
free(tp_for_thresh_per_class);
free(fp_for_thresh_per_class);
fprintf(stderr, "Total Detection Time: %f Seconds\n", (double)(time(0) - start));
printf("\nSet -points flag:\n");
printf(" `-points 101` for MS COCO \n");
printf(" `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) \n");
printf(" `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset\n");
if (reinforcement_fd != NULL) fclose(reinforcement_fd);
// free memory
free_ptrs((void**)names, net.layers[net.n - 1].classes);
free_list_contents_kvp(options);
free_list(options);
if (existing_net) {
//set_batch_network(&net, initial_batch);
//free_network_recurrent_state(*existing_net);
restore_network_recurrent_state(*existing_net);
//randomize_network_recurrent_state(*existing_net);
}
else {
free_network(net);
}
return mean_average_precision;
}

@buzdarbalooch
Copy link
Author

buzdarbalooch commented May 19, 2019 via email

@buzdarbalooch
Copy link
Author

@AlexeyAB i have read those two articles u shared. but if u can just summarize in two to three lines, by looking at fig i shared how we reached to AP or how AP was calculated? that would be soo nice of u. thanks

@AlexeyAB
Copy link
Owner

Value of average precision (AP) is equal to the area under Precision-Recall curve, where Precision and Recall are calculated for every possible confidence threshold (for confidence of each detection), and for specified IoU_threshold = 0.75 (75%).

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
TP - number of true positives
FP - number of false positives
FN - number of false negatives

@buzdarbalooch
Copy link
Author

OK thanks alot dear.

@buzdarbalooch
Copy link
Author

@AlexeyAB can i ask whats the difference between confidence threshold (confidence of each detection) and IoU threshold?

i know
IoU measures how much overlap exists between the ground truth and actual prediction: this measures how good is our prediction in the object detector with the ground truth (the real object boundary).

@buzdarbalooch
Copy link
Author

hi @AlexeyAB

can i ask whats the difference between confidence threshold (confidence of each detection) and IoU threshold?

@AlexeyAB
Copy link
Owner

  • Confidence_threshold - is a threshold for confidence score, when we think that object is detected
  • IoU_threshold - is a threshold for IoU( detection, truth), when we think that object is detected correctly

@buzdarbalooch
Copy link
Author

cofidence threshold is always fixed ?

@AlexeyAB
Copy link
Owner

  • mAP is calculated for every possible thresholds from 0.001 to 1.0

  • during detection you can set it by using flag -thresh 0.20

@buzdarbalooch
Copy link
Author

but map is calculated for Iou threshold not for confidence threshold, corrext me if i am wrong.

@buzdarbalooch
Copy link
Author

and during the detection whats the standard fixed value one uses for the confidence threshold?

@AlexeyAB
Copy link
Owner

but map is calculated for Iou threshold not for confidence threshold, corrext me if i am wrong.

Read these articles:

https://mc.ai/which-one-to-measure-the-performance-of-object-detectors-ap-or-olrp/

https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173

and during the detection whats the standard fixed value one uses for the confidence threshold?

0.25

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AlexeyAB @buzdarbalooch