YOLOv3 性能指标

如何评价训练好的网络：

首先网络有一个参数是loss值，这反应了你训练好的网络得到的结果和真实值之间的差距，具体的公式后续会补充，不过查看loss曲线随着迭代次数的增多，如何变化，有助于查看训练是否过拟合，是否学习率太小。

训练指令：

./darknet detector train cfg/voc-hand.data cfg/yolov3-voc-hand.cfg darknet53.conv.74 -gpu 2>1 | tee visualization/train_yolov3_hand.log

训练指令解释：

1）在DarkNet主目录下编译完成，则DarkNet框架训练环境搭建完成。可以使用命令：“./darknet”
2）detector为训练文件（底层c程序）
3）train表示该命令为训练命令（当测试时输入测试命令将train改为test）
4）cfg/voc-hand.data:表示配置的数据集的参数文件（注意路径）
5）cfg/yolov3-voc-hand.cfg ：表示配置的网络参数文件（注意路径）
6）darknet53.conv.74：表示训练所需要的预训练模型（注意路径，此处在darknet文件夹下）
7）| tee visualization/train_yolov3_hand.log：表示训练时保存日志文件（保存在DarkNet/visualization目录下）

训练产生的log图示：

Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000002, .5R: -nan, .75R: -nan,  count: 0
Region 94 Avg IOU: 0.898378, Class: 0.999900, Obj: 0.999876, No Obj: 0.000432, .5R: 1.000000, .75R: 1.000000,  count: 2
Region 106 Avg IOU: 0.868176, Class: 0.999792, Obj: 0.999554, No Obj: 0.000157, .5R: 1.000000, .75R: 1.000000,  count: 2
Region 82 Avg IOU: 0.577505, Class: 0.995153, Obj: 0.583519, No Obj: 0.000268, .5R: 1.000000, .75R: 0.000000,  count: 1
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000246, .5R: -nan, .75R: -nan,  count: 0
Region 106 Avg IOU: 0.747150, Class: 0.998767, Obj: 0.996712, No Obj: 0.000056, .5R: 1.000000, .75R: 0.000000,  count: 1
3690: 0.026425, 0.025064 avg, 0.000010 rate, 3.401929 seconds, 236160 images

训练产生的log的解释：

1）Region 82（94/106）：表示cfg文件中yolo-layer的索引值，三个值，82/94/106
2）Avg IOU：表示在训练过程中预测的bounding box与标注的bounding box的交并比（两个框的交集/两个框的并集），该值期望越大越好，目标值为1
3）Class：表示标注物体的分类准确率，该值期望越大越好，目标期望值为1
4）obj：表示预测有目标的概率,该值期望越大越好，目标值为1
5）No obj：表示预测没有目标的概率，该值期望越小越好，目标期望值为0
6）.5R：表示以IOU=0.5为阈值时被召回（recall）。recall=检出的正样本/实际的正样本
7）.75R：表示以IOU=0.75为阈值时被召回（recall）
8)count:表示正样本的数量

3690: 0.026425, 0.025064 avg, 0.000010 rate, 3.401929 seconds, 236160 images

1）3690：表示当前训练的迭代次数
2）0.026425：表示训练总的Loss损失
3）0.025064 avg： 表示平均Loss，该值期望越低越好，一般该值低于0.060730 avg就可以终止训练了。
4）0.000100 rate： 代表当前的学习率，是在.cfg文件中定义的。
5）3.401929 seconds： 表示当前批次训练花费的总时间（batch/subdivisions）
6）236160 images：表示到目前所参与训练的总的图片数量

查看训练网络的召回率

更改darknet/example/detector.c

//  list *plist = get_paths("data/coco_val_5k.list");
    list *plist = get_paths("2007_test.txt");  (2007_test.txt 在 darknet目录下)

  289   710   746       RPs/Img: 21.33  IOU: 75.44%     Recall:95.17%
  290   711   748       RPs/Img: 21.34  IOU: 75.38%     Recall:95.05%
  291   713   750       RPs/Img: 21.32  IOU: 75.41%     Recall:95.07%
  292   715   752       RPs/Img: 21.37  IOU: 75.37%     Recall:95.08%
  293   717   754       RPs/Img: 21.33  IOU: 75.40%     Recall:95.09%
  294   719   756       RPs/Img: 21.32  IOU: 75.42%     Recall:95.11%
  295   721   758       RPs/Img: 21.32  IOU: 75.41%     Recall:95.12%
  296   722   759       RPs/Img: 21.28  IOU: 75.42%     Recall:95.13%
  297   722   759       RPs/Img: 21.36  IOU: 75.42%     Recall:95.13%
  298   722   759       RPs/Img: 21.42  IOU: 75.42%     Recall:95.13%
  299   724   761       RPs/Img: 21.41  IOU: 75.44%     Recall:95.14%
  300   725   762       RPs/Img: 21.43  IOU: 75.44%     Recall:95.14%
  301   727   764       RPs/Img: 21.42  IOU: 75.48%     Recall:95.16%
  302   728   765       RPs/Img: 21.43  IOU: 75.46%     Recall:95.16%
  303   728   765       RPs/Img: 21.40  IOU: 75.46%     Recall:95.16%
  304   730   767       RPs/Img: 21.40  IOU: 75.48%     Recall:95.18%
  305   734   771       RPs/Img: 21.42  IOU: 75.50%     Recall:95.20%
  306   746   783       RPs/Img: 21.45  IOU: 75.59%     Recall:95.27%
  307   748   785       RPs/Img: 21.43  IOU: 75.62%     Recall:95.29%
  308   750   787       RPs/Img: 21.42  IOU: 75.59%     Recall:95.30%
  309   752   789       RPs/Img: 21.43  IOU: 75.62%     Recall:95.31%
  310   754   791       RPs/Img: 21.44  IOU: 75.63%     Recall:95.32%

具体的解释如下：

Number Correct Total Rps/Img IOU Recall 

Number：  表示处理到第几张图片。
Correct： 表示正确的识别除了多少bbox。这个值算出来的步骤是这样的，丢进网络一张图片，网络会预测出很多bbox，每个bbox都有其置信概率，概率大于threshold的bbox与实际的bbox，也就是labels中txt的内容计算IOU，找出IOU最大的bbox，如果这个最大值大于预设的IOU的threshold，那么correct加一。
Total：   表示实际有多少个bbox。
Rps/img： 表示平均每个图片会预测出来多少个bbox。
IOU：     这个是预测出的bbox和实际标注的bbox的交集 除以 他们的并集。显然，这个数值越大，说明预测的结果越好。
Recall：  召回率 意思是检测出物体的个数 除以 标注的所有物体个数。通过代码我们也能看出来就是Correct除以Total的值。

参考链接：https://www.jianshu.com/p/7ae10c8f7d77/

综合计算性能指标：

修改yolov3的darknet文件夹example文件夹中的detector程序（计算ap值、recall值、avg、iou值、precision值、fp、tp以及绘制P-R曲线指标所需的precision和recall值等值）

在detector文件中添加计算map的函数并进行设置。
具体操作步骤：
1. 在detector.c文件中找到以下代码：

if(0==strcmp(argv[2], "test")) test_detector(datacfg, cfg, weights, filename, thresh, hier_thresh, outfile, fullscreen);
    else if(0==strcmp(argv[2], "train")) train_detector(datacfg, cfg, weights, gpus, ngpus, clear);
    else if(0==strcmp(argv[2], "valid")) validate_detector(datacfg, cfg, weights, outfile);
    else if(0==strcmp(argv[2], "valid2")) validate_detector_flip(datacfg, cfg, weights, outfile);
    else if(0==strcmp(argv[2], "recall")) validate_detector_recall(datacfg, cfg, weights);

2. 在其后面加上一条代码语句：

else if(0==strcmp(argv[2], "map")) validate_detector_map(datacfg, cfg, weights, thresh);

3. 在detector.c程序中的任意位置添加以下代码：

typedef struct {
    box b;
    float p;
    int class_id;
    int image_index;
    int truth_flag;
    int unique_truth_index;
} box_prob;
 
int detections_comparator(const void *pa, const void *pb)
{
    box_prob a = *(box_prob *)pa;
    box_prob b = *(box_prob *)pb;
    float diff = a.p - b.p;
    if (diff < 0) return 1;
    else if (diff > 0) return -1;
    return 0;
}
 
void validate_detector_map(char *datacfg, char *cfgfile, char *weightfile, float thresh_calc_avg_iou)
{
    list *options = read_data_cfg(datacfg);                                          //get .data file
    char *valid_images = option_find_str(options, "valid", "data/train.txt");        //point to the path of valid images
    char *difficult_valid_images = option_find_str(options, "difficult", NULL);      //get the path to the 'difficult', if it doesn't exist,replace it with NULL
 
 
    char *name_list = option_find_str(options, "names", "data/names.list");          // find name of each category 
    char **names = get_labels(name_list);
    //char *mapf = option_find_str(options, "map", 0);               // get the 'map', what is the map
    //int *map = 0;
    //if (mapf) map = read_map(mapf);
    FILE* reinforcement_fd = NULL;
 
    network *net = load_network(cfgfile, weightfile, 0);
    set_batch_network(net, 1);
    //fuse_conv_batchnorm(net);
    //calculate_binary_weights(net);
    srand(time(0));
 
    list *plist = get_paths(valid_images);
    char **paths = (char **)list_to_array(plist);
 
    char **paths_dif = NULL;
    if (difficult_valid_images) {
        list *plist_dif = get_paths(difficult_valid_images);
        paths_dif = (char **)list_to_array(plist_dif);
    }
 
 
    layer l = net->layers[net->n - 1];
    int classes = l.classes;
 
    int m = plist->size;
    int i = 0;
    int t;
 
    const float thresh = .005;
    const float nms = .45;
    const float iou_thresh = 0.5;
 
    int nthreads = 4;
    image *val = calloc(nthreads, sizeof(image));
    image *val_resized = calloc(nthreads, sizeof(image));
    image *buf = calloc(nthreads, sizeof(image));
    image *buf_resized = calloc(nthreads, sizeof(image));
    pthread_t *thr = calloc(nthreads, sizeof(pthread_t));
 
    load_args args = {0};
    args.w = net->w;
    args.h = net->h;
    //args.type = IMAGE_DATA;
    args.type = LETTERBOX_DATA;
 
    //const float thresh_calc_avg_iou = 0.24;
    float avg_iou = 0;
    int tp_for_thresh = 0;
    int fp_for_thresh = 0;
 
    box_prob *detections = calloc(1, sizeof(box_prob));
    int detections_count = 0;
    int unique_truth_count = 0;
 
    int *truth_classes_count = calloc(classes, sizeof(int));
 
    for (t = 0; t < nthreads; ++t) {
        args.path = paths[i + t];
        args.im = &buf[t];
        args.resized = &buf_resized[t];
        thr[t] = load_data_in_thread(args);
    }
    time_t start = time(0);
    for (i = nthreads; i < m + nthreads; i += nthreads) {
        fprintf(stderr, "%d\n", i);
        for (t = 0; t < nthreads && i + t - nthreads < m; ++t) {
            pthread_join(thr[t], 0);
            val[t] = buf[t];
            val_resized[t] = buf_resized[t];
        }
        for (t = 0; t < nthreads && i + t < m; ++t) {
            args.path = paths[i + t];
            args.im = &buf[t];
            args.resized = &buf_resized[t];
            thr[t] = load_data_in_thread(args);
        }
        for (t = 0; t < nthreads && i + t - nthreads < m; ++t) {
            const int image_index = i + t - nthreads;
            char *path = paths[image_index];
            char *id = basecfg(path);                         
            float *X = val_resized[t].data;
            network_predict(net, X);
 
            int nboxes = 0;
            float hier_thresh = 0;
            detection *dets;
            if (args.type == LETTERBOX_DATA) {
                //int letterbox = 1;
                dets = get_network_boxes(net, val[t].w, val[t].h, thresh, hier_thresh, 0, 1, &nboxes);
            }
            else {
                //int letterbox = 0;
                dets = get_network_boxes(net, 1, 1, thresh, hier_thresh, 0, 0, &nboxes);
            }
            //detection *dets = get_network_boxes(&net, val[t].w, val[t].h, thresh, hier_thresh, 0, 1, &nboxes, letterbox); // for letterbox=1
            if (nms) do_nms_sort(dets, nboxes, l.classes, nms);
 
            char labelpath[4096];
            find_replace(path, "images", "labels", labelpath);
            find_replace(labelpath, "JPEGImages", "labels", labelpath);
            find_replace(labelpath, ".jpg", ".txt", labelpath);
            find_replace(labelpath, ".JPEG", ".txt", labelpath);
 
 
            int num_labels = 0;
            box_label *truth = read_boxes(labelpath, &num_labels);
            int i, j;
            for (j = 0; j < num_labels; ++j) {
                truth_classes_count[truth[j].id]++;                  
            }
 
            // difficult
            box_label *truth_dif = NULL;
            int num_labels_dif = 0;
            if (paths_dif)
            {
                char *path_dif = paths_dif[image_index];
 
                char labelpath_dif[4096];
                //replace_image_to_label(path_dif, labelpath_dif);
                find_replace(path_dif, "images", "labels", labelpath_dif);
                find_replace(labelpath_dif, "JPEGImages", "labels", labelpath_dif);
                find_replace(labelpath_dif, ".jpg", ".txt", labelpath_dif);
                find_replace(labelpath_dif, ".JPEG", ".txt", labelpath_dif);
 
                truth_dif = read_boxes(labelpath_dif, &num_labels_dif);
            }
 
            const int checkpoint_detections_count = detections_count;
 
            for (i = 0; i < nboxes; ++i) {
 
                int class_id;
                for (class_id = 0; class_id < classes; ++class_id) {
                    float prob = dets[i].prob[class_id];
                    if (prob > 0) {
                        detections_count++;
                        detections = realloc(detections, detections_count * sizeof(box_prob));
                        detections[detections_count - 1].b = dets[i].bbox;
                        detections[detections_count - 1].p = prob;
                        detections[detections_count - 1].image_index = image_index;
                        detections[detections_count - 1].class_id = class_id;
                        detections[detections_count - 1].truth_flag = 0;
                        detections[detections_count - 1].unique_truth_index = -1;
 
                        int truth_index = -1;
                        float max_iou = 0;
                        for (j = 0; j < num_labels; ++j)
                        {
                            box t = { truth[j].x, truth[j].y, truth[j].w, truth[j].h };
                            //printf(" IoU = %f, prob = %f, class_id = %d, truth[j].id = %d \n",
                                   //box_iou(dets[i].bbox, t), prob, class_id, truth[j].id);
                            float current_iou = box_iou(dets[i].bbox, t);
                            if (current_iou > iou_thresh && class_id == truth[j].id) {
                                if (current_iou > max_iou) {
                                    max_iou = current_iou;
                                    truth_index = unique_truth_count + j;
                                }
                            }
                        }
 
                        // best IoU
                        if (truth_index > -1) {
                            detections[detections_count - 1].truth_flag = 1;
                            detections[detections_count - 1].unique_truth_index = truth_index;
                        }
                        else {
                            // if object is difficult then remove detection
                            for (j = 0; j < num_labels_dif; ++j) {
                                box t = { truth_dif[j].x, truth_dif[j].y, truth_dif[j].w, truth_dif[j].h };
                                float current_iou = box_iou(dets[i].bbox, t);
                                if (current_iou > iou_thresh && class_id == truth_dif[j].id) {
                                    --detections_count;
                                    break;
                                }
                            }
                        }
 
                        // calc avg IoU, true-positives, false-positives for required Threshold
                        if (prob > thresh_calc_avg_iou) {
                            int z, found = 0;
                            for (z = checkpoint_detections_count; z < detections_count-1; ++z)
                                if (detections[z].unique_truth_index == truth_index) {
                                    found = 1; break;
                                }
 
                            if(truth_index > -1 && found == 0) {
                                avg_iou += max_iou;
                                ++tp_for_thresh;
                            }
                            else
                                fp_for_thresh++;
                        }
                    }
                }
            }
 
            unique_truth_count += num_labels;
 
            //static int previous_errors = 0;
            //int total_errors = fp_for_thresh + (unique_truth_count - tp_for_thresh);
            //int errors_in_this_image = total_errors - previous_errors;
            //previous_errors = total_errors;
            //if(reinforcement_fd == NULL) reinforcement_fd = fopen("reinforcement.txt", "wb");
            //char buff[1000];
            //sprintf(buff, "%s\n", path);
            //if(errors_in_this_image > 0) fwrite(buff, sizeof(char), strlen(buff), reinforcement_fd);
 
            free_detections(dets, nboxes);
            free(id);
            free_image(val[t]);
            free_image(val_resized[t]);
        }
    }
 
    if((tp_for_thresh + fp_for_thresh) > 0)
        avg_iou = avg_iou / (tp_for_thresh + fp_for_thresh);
 
 
    // SORT(detections)
    qsort(detections, detections_count, sizeof(box_prob), detections_comparator);
 
    typedef struct {
        double precision;
        double recall;
        int tp, fp, fn;
    } pr_t;
 
    // for PR-curve
    pr_t **pr = calloc(classes, sizeof(pr_t*));
    for (i = 0; i < classes; ++i) {
        pr[i] = calloc(detections_count, sizeof(pr_t));
    }
    printf("detections_count = %d, unique_truth_count = %d  \n", detections_count, unique_truth_count);
 
 
    int *truth_flags = calloc(unique_truth_count, sizeof(int));
 
    int rank;
    for (rank = 0; rank < detections_count; ++rank) {
        if(rank % 100 == 0)
            printf(" rank = %d of ranks = %d \r", rank, detections_count);
 
        if (rank > 0) {
            int class_id;
            for (class_id = 0; class_id < classes; ++class_id) {
                pr[class_id][rank].tp = pr[class_id][rank - 1].tp;
                pr[class_id][rank].fp = pr[class_id][rank - 1].fp;
            }
        }
 
        box_prob d = detections[rank];
        // if (detected && isn't detected before)
        if (d.truth_flag == 1) {
            if (truth_flags[d.unique_truth_index] == 0)
            {
                truth_flags[d.unique_truth_index] = 1;
                pr[d.class_id][rank].tp++;    // true-positive
            }
        }
        else {
            pr[d.class_id][rank].fp++;    // false-positive
        }
 
        for (i = 0; i < classes; ++i)
        {
            const int tp = pr[i][rank].tp;
            const int fp = pr[i][rank].fp;
            const int fn = truth_classes_count[i] - tp;    // false-negative = objects - true-positive
            pr[i][rank].fn = fn;
 
            if ((tp + fp) > 0) pr[i][rank].precision = (double)tp / (double)(tp + fp);
            else pr[i][rank].precision = 0;
 
            if ((tp + fn) > 0) pr[i][rank].recall = (double)tp / (double)(tp + fn);
            else pr[i][rank].recall = 0;
        }
    }
 
    free(truth_flags);
 
 
    double mean_average_precision = 0;
 
    for (i = 0; i < classes; ++i) {
        double avg_precision = 0;
        int point;
        for (point = 0; point < 11; ++point) {
            double cur_recall = point * 0.1;
            double cur_precision = 0;
            for (rank = 0; rank < detections_count; ++rank)
            {
                if (pr[i][rank].recall >= cur_recall) {    // > or >=
                    if (pr[i][rank].precision > cur_precision) {
                        cur_precision = pr[i][rank].precision;
                    }
                }
            }
            printf("class_id = %d, point = %d, cur_recall = %.4f, cur_precision = %.4f \n", i, point, cur_recall, cur_precision);
 
            avg_precision += cur_precision;
        }
        avg_precision = avg_precision / 11;     //  ??
        printf("class_id = %d, name = %s, \t ap = %2.2f %% \n", i, names[i], avg_precision*100);
        mean_average_precision += avg_precision;
    }
 
 
    printf("---------------------caculate end!!------------------------\n");
 
 
    const float cur_precision = (float)tp_for_thresh / ((float)tp_for_thresh + (float)fp_for_thresh);
    const float cur_recall = (float)tp_for_thresh / ((float)tp_for_thresh + (float)(unique_truth_count - tp_for_thresh));
    const float f1_score = 2.F * cur_precision * cur_recall / (cur_precision + cur_recall);
    printf(" for thresh = %1.2f, precision = %1.2f, recall = %1.2f, F1-score = %1.2f \n",
        thresh_calc_avg_iou, cur_precision, cur_recall, f1_score);
 
    printf(" for thresh = %0.2f, TP = %d, FP = %d, FN = %d, average IoU = %2.2f %% \n",
        thresh_calc_avg_iou, tp_for_thresh, fp_for_thresh, unique_truth_count - tp_for_thresh, avg_iou * 100);
 
    mean_average_precision = mean_average_precision / classes;
    printf("\n mean average precision (mAP) = %f, or %2.2f %% \n", mean_average_precision, mean_average_precision*100);
 
 
    for (i = 0; i < classes; ++i) {
        free(pr[i]);
    }
    free(pr);
    free(detections);
    free(truth_classes_count);
 
    fprintf(stderr, "Total Detection Time: %f Seconds\n", (double)(time(0) - start));
    if (reinforcement_fd != NULL) fclose(reinforcement_fd);
}

4. 重新编译：

make clean
make

5. 终端输入命令：

./darknet detector map cfg/voc-hand.data cfg/yolov3-voc-hand-test.cfg backup/yolov3-voc-hand_final.weights