第4门课程-卷积神经网络-第三周作业(机器视觉中物体检测)

0- 背景：

本文主要介绍在自动驾驶中常用的车辆探测模型YOLO模型。
本文依赖的包如下：

import argparse
import os
import matplotlib.pyplot as plt
from matplotlib.pyplot import imshow
import scipy.io
import scipy.misc
import numpy as np
import pandas as pd
import PIL
import tensorflow as tf
from keras import backend as K
from keras.layers import Input, Lambda, Conv2D
from keras.models import load_model, Model
from yolo_utils import read_classes, read_anchors, generate_colors, preprocess_image, draw_boxes, scale_boxes
from yad2k.models.keras_yolo import yolo_head, yolo_boxes_to_corners, preprocess_true_boxes, yolo_loss, yolo_body

%matplotlib inline

1-数据介绍：

本文的数据来源于https://www.drive.ai/。训练数据集中的车辆均以方框进行标识，如下：

由于YOLO model训练时候计算量很大，我们在这里直接导入训练好的模型。

2- YOLO

YOLO (“you only look once”)模型不仅准确率高，且可以实时运行。该算法仅仅需要一次的前向传播，便可以做预测。在做非极大值抑制后，可以以方框形式输出被探测到的物体坐标。

2-1 模型细节

输入的batch图像尺寸= (m, 608, 608, 3)，输出的是一序列的探测类别及其坐标值。每个方框的坐标值由 (pc,bx,by,bh,bw,c) ( p c , b x , b y , b h , b w , c ) 这6个参数组成。如果参数 c c 展开成80维向量（假如分类结果是80类，相应位置置为1表示该类），则每个方框的维度就是85。我们采用 5 个anchor boxes.。所以YOLO architecture便如下：
IMAGE (m, 608, 608, 3) -> DEEP CNN -> ENCODING (m, 19, 19, 5, 85)
具体的框架如下：

被探测物体的中心点所位于的方格区域就表明该方格探测到该物体。
由于本文采用 5个anchor boxes, 每个都是19 x19方格组成。为简单起见，我们将(19, 19, 5, 85)的最后两个维度合并，所以Deep CNN的输出为(19, 19, 425)。
如此对于每个box的值与概率做elementwise product，得出探测类别：

可视化YOLO的输出结果：
1）对于 19x19 grid cells,计算最大的概率得分(5个 anchor boxes都要，所以是5种)。
2）对不同结果着色，以示区分。

3）绘制方框，将探测结果圈住

从上图结果，我们可以看出，探测的结果有些多，需要对这些结果再做一次过滤。这就需要用到non-max suppression算法，步骤如下：

(1)Get rid of boxes with a low score (meaning, the box is not very confident about detecting a class)
(2)Select only one box when several boxes overlap with each other and detect the same object.

2-2 Filtering with a threshold on class scores

先设定一个阈值，将阈值之下的那些box消除；
一张图输入到模型之后，其输出是19x19x5x85，其中每个box都是以85个元素来表示的。可以很方便地将其出入到

• box_confidence: tensor of shape (19×19,5,1) ( 19 × 19 , 5 , 1 ) containing pc p c (confidence probability that there’s some object) for each of the 5 boxes predicted in each of the 19x19 cells.
  • boxes: tensor of shape (19×19,5,4) ( 19 × 19 , 5 , 4 ) containing (bx,by,bh,bw) ( b x , b y , b h , b w ) for each of the 5 boxes per cell.
  • box_class_probs: tensor of shape (19×19,5,80) ( 19 × 19 , 5 , 80 ) containing the detection probabilities (c1,c2,...c80) ( c 1 , c 2 , . . . c 80 ) for each of the 80 classes for each of the 5 boxes per cell.

  • 具体代码如下：

    # GRADED FUNCTION: yolo_filter_boxes
    
    def yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = .6):
        """Filters YOLO boxes by thresholding on object and class confidence.
    
        Arguments:
        box_confidence -- tensor of shape (19, 19, 5, 1)
        boxes -- tensor of shape (19, 19, 5, 4)
        box_class_probs -- tensor of shape (19, 19, 5, 80)
        threshold -- real value, if [ highest class probability score < threshold], then get rid of the corresponding box
    
        Returns:
        scores -- tensor of shape (None,), containing the class probability score for selected boxes
        boxes -- tensor of shape (None, 4), containing (b_x, b_y, b_h, b_w) coordinates of selected boxes
        classes -- tensor of shape (None,), containing the index of the class detected by the selected boxes
    
        Note: "None" is here because you don't know the exact number of selected boxes, as it depends on the threshold. 
        For example, the actual output size of scores would be (10,) if there are 10 boxes.
        """
    
        # Step 1: Compute box scores
        ### START CODE HERE ### (≈ 1 line)
        box_scores = box_confidence * box_class_probs#(19*19, 5, 80)
        ### END CODE HERE ###
    
        # Step 2: Find the box_classes thanks to the max box_scores, keep track of the corresponding score
        ### START CODE HERE ### (≈ 2 lines)
        box_classes = K.argmax(box_scores, axis=-1)
        box_class_scores = K.max(box_scores, axis=-1, keepdims=False)
        ### END CODE HERE ###
    
        # Step 3: Create a filtering mask based on "box_class_scores" by using "threshold". The mask should have the
        # same dimension as box_class_scores, and be True for the boxes you want to keep (with probability >= threshold)
        ### START CODE HERE ### (≈ 1 line)
        filtering_mask = (box_class_scores >= threshold)
        ### END CODE HERE ###
    
        # Step 4: Apply the mask to scores, boxes and classes
        ### START CODE HERE ### (≈ 3 lines)
        #box_class_scores, boxes and box_classes 
        scores = tf.boolean_mask(box_class_scores, filtering_mask)
        boxes = tf.boolean_mask(boxes, filtering_mask)
        classes = tf.boolean_mask(box_classes, filtering_mask)
        ### END CODE HERE ###
    
        return scores, boxes, classes

    代码测试：

    with tf.Session() as test_a:
        box_confidence = tf.random_normal([19, 19, 5, 1], mean=1, stddev=4, seed = 1)
        boxes = tf.random_normal([19, 19, 5, 4], mean=1, stddev=4, seed = 1)
        box_class_probs = tf.random_normal([19, 19, 5, 80], mean=1, stddev=4, seed = 1)
        scores, boxes, classes = yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = 0.5)
        print("scores[2] = " + str(scores[2].eval()))
        print("boxes[2] = " + str(boxes[2].eval()))
        print("classes[2] = " + str(classes[2].eval()))
        print("scores.shape = " + str(scores.shape))
        print("boxes.shape = " + str(boxes.shape))
        print("classes.shape = " + str(classes.shape))

    运行结果如下：

    scores[2] = 10.7506
    boxes[2] = [ 8.42653275  3.27136683 -0.53134358 -4.94137335]
    classes[2] = 7
    scores.shape = (?,)
    boxes.shape = (?, 4)
    classes.shape = (?,)

    2-3 非极大值抑制

    在阈值滤波器过滤之后，我们可能会获得一些具有重叠的box，这里我们采用非极大值抑制的方式选择最终的box。
    
    我们定义”Intersection over Union” (IoU)如下：
    
    两个方框的IOU为交集除以并集。
    IOU计算代码如下：

    # GRADED FUNCTION: iou
    
    def iou(box1, box2):
        """Implement the intersection over union (IoU) between box1 and box2
    
        Arguments:
        box1 -- first box, list object with coordinates (x1, y1, x2, y2)
        box2 -- second box, list object with coordinates (x1, y1, x2, y2)
        """
    
        # Calculate the (y1, x1, y2, x2) coordinates of the intersection of box1 and box2. Calculate its Area.
        ### START CODE HERE ### (≈ 5 lines)
        xi1 = np.maximum(box1[0],box2[0])
        yi1 = np.maximum(box1[1],box2[1])
        xi2 = np.minimum(box1[2],box2[2])
        yi2 = np.minimum(box1[3],box2[3])
        inter_area = (xi1-xi2)*(yi1-yi2)
        ### END CODE HERE ###    
    
        # Calculate the Union area by using Formula: Union(A,B) = A + B - Inter(A,B)
        ### START CODE HERE ### (≈ 3 lines)
        box1_area = (box1[3]-box1[1]) * (box1[2]-box1[0])
        box2_area = (box2[3]-box2[1]) * (box2[2]-box2[0])
        union_area = box1_area + box2_area - inter_area
        ### END CODE HERE ###
    
        # compute the IoU
        ### START CODE HERE ### (≈ 1 line)
        iou = inter_area/union_area
        ### END CODE HERE ###
    
        return iou

    测试IOU：

    box1 = (2, 1, 4, 3)
    box2 = (1, 2, 3, 4) 
    print("iou = " + str(iou(box1, box2)))

    测试结果如下：

    iou = 0.142857142857

    在TensorFlow里面可以完成非最大值抑制如下：

    # GRADED FUNCTION: yolo_non_max_suppression
    
    def yolo_non_max_suppression(scores, boxes, classes, max_boxes = 10, iou_threshold = 0.5):
        """
        Applies Non-max suppression (NMS) to set of boxes
    
        Arguments:
        scores -- tensor of shape (None,), output of yolo_filter_boxes()
        boxes -- tensor of shape (None, 4), output of yolo_filter_boxes() that have been scaled to the image size (see later)
        classes -- tensor of shape (None,), output of yolo_filter_boxes()
        max_boxes -- integer, maximum number of predicted boxes you'd like
        iou_threshold -- real value, "intersection over union" threshold used for NMS filtering
    
        Returns:
        scores -- tensor of shape (, None), predicted score for each box
        boxes -- tensor of shape (4, None), predicted box coordinates
        classes -- tensor of shape (, None), predicted class for each box
    
        Note: The "None" dimension of the output tensors has obviously to be less than max_boxes. Note also that this
        function will transpose the shapes of scores, boxes, classes. This is made for convenience.
        """
    
        max_boxes_tensor = K.variable(max_boxes, dtype='int32')     # tensor to be used in tf.image.non_max_suppression()
        K.get_session().run(tf.variables_initializer([max_boxes_tensor])) # initialize variable max_boxes_tensor
    
        # Use tf.image.non_max_suppression() to get the list of indices corresponding to boxes you keep
        ### START CODE HERE ### (≈ 1 line)
        nms_indices = tf.image.non_max_suppression(boxes, scores, max_boxes, iou_threshold)
        ### END CODE HERE ###
    
        # Use K.gather() to select only nms_indices from scores, boxes and classes
        ### START CODE HERE ### (≈ 3 lines)
        scores = K.gather(scores, nms_indices)
        boxes = K.gather(boxes, nms_indices)
        classes = K.gather(classes, nms_indices)
        ### END CODE HERE ###
    
        return scores, boxes, classes

    测试代码：

    with tf.Session() as test_b:
        scores = tf.random_normal([54,], mean=1, stddev=4, seed = 1)
        boxes = tf.random_normal([54, 4], mean=1, stddev=4, seed = 1)
        classes = tf.random_normal([54,], mean=1, stddev=4, seed = 1)
        scores, boxes, classes = yolo_non_max_suppression(scores, boxes, classes)
        print("scores[2] = " + str(scores[2].eval()))
        print("boxes[2] = " + str(boxes[2].eval()))
        print("classes[2] = " + str(classes[2].eval()))
        print("scores.shape = " + str(scores.eval().shape))
        print("boxes.shape = " + str(boxes.eval().shape))
        print("classes.shape = " + str(classes.eval().shape))

    测试代码运行结果：

    scores[2] = 6.9384
    boxes[2] = [-5.299932    3.13798141  4.45036697  0.95942086]
    classes[2] = -2.24527
    scores.shape = (10,)
    boxes.shape = (10, 4)
    classes.shape = (10,)

    2-4 Wrapping up the filtering

    对上述过程做如下归总：

    # GRADED FUNCTION: yolo_eval
    
    def yolo_eval(yolo_outputs, image_shape = (720., 1280.), max_boxes=10, score_threshold=.6, iou_threshold=.5):
        """
        Converts the output of YOLO encoding (a lot of boxes) to your predicted boxes along with their scores, box coordinates and classes.
    
        Arguments:
        yolo_outputs -- output of the encoding model (for image_shape of (608, 608, 3)), contains 4 tensors:
                        box_confidence: tensor of shape (None, 19, 19, 5, 1)
                        box_xy: tensor of shape (None, 19, 19, 5, 2)
                        box_wh: tensor of shape (None, 19, 19, 5, 2)
                        box_class_probs: tensor of shape (None, 19, 19, 5, 80)
        image_shape -- tensor of shape (2,) containing the input shape, in this notebook we use (608., 608.) (has to be float32 dtype)
        max_boxes -- integer, maximum number of predicted boxes you'd like
        score_threshold -- real value, if [ highest class probability score < threshold], then get rid of the corresponding box
        iou_threshold -- real value, "intersection over union" threshold used for NMS filtering
    
        Returns:
        scores -- tensor of shape (None, ), predicted score for each box
        boxes -- tensor of shape (None, 4), predicted box coordinates
        classes -- tensor of shape (None,), predicted class for each box
        """
    
        ### START CODE HERE ### 
    
        # Retrieve outputs of the YOLO model (≈1 line)
        box_confidence, box_xy, box_wh, box_class_probs = yolo_outputs
    
        # Convert boxes to be ready for filtering functions 
        #Convert YOLO box predictions to bounding box corners
        boxes = yolo_boxes_to_corners(box_xy, box_wh)
    
        # Use one of the functions you've implemented to perform Score-filtering with a threshold of score_threshold (≈1 line)
        #yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = 0.5)
        scores, boxes, classes = yolo_filter_boxes(box_confidence, boxes, box_class_probs, score_threshold)
    
        # Scale boxes back to original image shape.
    
        #YOLO's network was trained to run on 608x608 images. If you are testing this data on a different size image--for example, 
        #the car detection dataset had 720x1280 images--this step rescales the boxes
        #so that they can be plotted on top of the original 720x1280 image. 
        boxes = scale_boxes(boxes, image_shape)
    
        # Use one of the functions you've implemented to perform Non-max suppression with a threshold of iou_threshold (≈1 line)
        scores, boxes, classes = yolo_non_max_suppression(scores, boxes, classes, max_boxes, iou_threshold)
    
        ### END CODE HERE ###
    
        return scores, boxes, classes

    测试代码如下：

    with tf.Session() as test_b:
        yolo_outputs = (tf.random_normal([19, 19, 5, 1], mean=1, stddev=4, seed = 1),
                        tf.random_normal([19, 19, 5, 2], mean=1, stddev=4, seed = 1),
                        tf.random_normal([19, 19, 5, 2], mean=1, stddev=4, seed = 1),
                        tf.random_normal([19, 19, 5, 80], mean=1, stddev=4, seed = 1))
        scores, boxes, classes = yolo_eval(yolo_outputs)
        print("scores[2] = " + str(scores[2].eval()))
        print("boxes[2] = " + str(boxes[2].eval()))
        print("classes[2] = " + str(classes[2].eval()))
        print("scores.shape = " + str(scores.eval().shape))
        print("boxes.shape = " + str(boxes.eval().shape))
        print("classes.shape = " + str(classes.eval().shape))

    测试结果：

    scores[2] = 138.791
    boxes[2] = [ 1292.32971191  -278.52166748  3876.98925781  -835.56494141]
    classes[2] = 54
    scores.shape = (10,)
    boxes.shape = (10, 4)
    classes.shape = (10,)

    YOLO总结:

    • Input image (608, 608, 3)
    • 图像输入到CNN, 输出结果维度= (19,19,5,85)
    • flattening最后两个维度，新的shape=(19, 19, 425):
      
      • Each cell in a 19x19 grid over the input image gives 425 numbers.
      • 425 = 5 x 85 because each cell contains predictions for 5 boxes, corresponding to 5 anchor boxes, as seen in lecture.
      • 85 = 5 + 80 where 5 is because (pc,bx,by,bh,bw) ( p c , b x , b y , b h , b w ) has 5 numbers, and and 80 is the number of classes we’d like to detect
    • 对boxes按照以下方式筛选:
      
      • Score-thresholding: throw away boxes that have detected a class with a score less than the threshold
      • Non-max suppression: Compute the Intersection over Union and avoid selecting overlapping boxes
    • This gives you YOLO’s final output.

    3-Test YOLO pretrained model on images

    3-1 定义classes, anchors和image的shape

    假设我们要探测80个classes, 采用5个anchor boxes。两者分别放于”coco_classes.txt” and “yolo_anchors.txt”文件中。待探测的图像尺寸是720x1280，在yolo_eval内部我们会调用scale_boxes使得图像转为608x608。

    sess = K.get_session()#创建session
    class_names = read_classes("model_data/coco_classes.txt")
    anchors = read_anchors("model_data/yolo_anchors.txt")
    image_shape = (720., 1280.)  

    3-2 加载预训练的模型：

    yolo_model = load_model("model_data/yolo.h5")

    yolo.h5这个模型也可以通过官网的权重参数和配置文件生成：

    1）从Darknet官方下载model：official YOLO website.
    wget http://pjreddie.com/media/files/yolo.weights
    下载权重文件。
    2）再下载配置文件yolo.cfg：
    配置文件

    3）将 Darknet YOLO_v2 model转换为Keras model.
    ./yad2k.py cfg/yolo.cfg yolo.weights model_data/yolo.h5
    测试图片位于 images/文件夹.
    ./test_yolo.py model_data/yolo.h5
    上述代码均位于:
    https://github.com/allanzelener/YAD2K

    注意，如果是直接从课程网站下载的h5文件，在本地直接加载，可能出现一些意外，如python内核崩溃，至于详细原因，目前只能是猜测h5文件的生成环境和自己电脑的环境差异造成的。为此，本人是采用在本地生成h5文件的方式，再加载该模型。
    加载完显示信息如下：

    c:\users\jason\appdata\local\programs\python\python35\lib\site-packages\keras\models.py:255: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
      warnings.warn('No training configuration found in save file: '

    查看模型的信息：

    yolo_model.summary()

    结果显示如下：

    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to                     
    ==================================================================================================
    input_1 (InputLayer)            (None, 608, 608, 3)  0                                            
    __________________________________________________________________________________________________
    conv2d_1 (Conv2D)               (None, 608, 608, 32) 864         input_1[0][0]                    
    __________________________________________________________________________________________________
    batch_normalization_1 (BatchNor (None, 608, 608, 32) 128         conv2d_1[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_1 (LeakyReLU)       (None, 608, 608, 32) 0           batch_normalization_1[0][0]      
    __________________________________________________________________________________________________
    max_pooling2d_1 (MaxPooling2D)  (None, 304, 304, 32) 0           leaky_re_lu_1[0][0]              
    __________________________________________________________________________________________________
    conv2d_2 (Conv2D)               (None, 304, 304, 64) 18432       max_pooling2d_1[0][0]            
    __________________________________________________________________________________________________
    batch_normalization_2 (BatchNor (None, 304, 304, 64) 256         conv2d_2[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_2 (LeakyReLU)       (None, 304, 304, 64) 0           batch_normalization_2[0][0]      
    __________________________________________________________________________________________________
    max_pooling2d_2 (MaxPooling2D)  (None, 152, 152, 64) 0           leaky_re_lu_2[0][0]              
    __________________________________________________________________________________________________
    conv2d_3 (Conv2D)               (None, 152, 152, 128 73728       max_pooling2d_2[0][0]            
    __________________________________________________________________________________________________
    batch_normalization_3 (BatchNor (None, 152, 152, 128 512         conv2d_3[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_3 (LeakyReLU)       (None, 152, 152, 128 0           batch_normalization_3[0][0]      
    __________________________________________________________________________________________________
    conv2d_4 (Conv2D)               (None, 152, 152, 64) 8192        leaky_re_lu_3[0][0]              
    __________________________________________________________________________________________________
    batch_normalization_4 (BatchNor (None, 152, 152, 64) 256         conv2d_4[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_4 (LeakyReLU)       (None, 152, 152, 64) 0           batch_normalization_4[0][0]      
    __________________________________________________________________________________________________
    conv2d_5 (Conv2D)               (None, 152, 152, 128 73728       leaky_re_lu_4[0][0]              
    __________________________________________________________________________________________________
    batch_normalization_5 (BatchNor (None, 152, 152, 128 512         conv2d_5[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_5 (LeakyReLU)       (None, 152, 152, 128 0           batch_normalization_5[0][0]      
    __________________________________________________________________________________________________
    max_pooling2d_3 (MaxPooling2D)  (None, 76, 76, 128)  0           leaky_re_lu_5[0][0]              
    __________________________________________________________________________________________________
    conv2d_6 (Conv2D)               (None, 76, 76, 256)  294912      max_pooling2d_3[0][0]            
    __________________________________________________________________________________________________
    batch_normalization_6 (BatchNor (None, 76, 76, 256)  1024        conv2d_6[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_6 (LeakyReLU)       (None, 76, 76, 256)  0           batch_normalization_6[0][0]      
    __________________________________________________________________________________________________
    conv2d_7 (Conv2D)               (None, 76, 76, 128)  32768       leaky_re_lu_6[0][0]              
    __________________________________________________________________________________________________
    batch_normalization_7 (BatchNor (None, 76, 76, 128)  512         conv2d_7[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_7 (LeakyReLU)       (None, 76, 76, 128)  0           batch_normalization_7[0][0]      
    __________________________________________________________________________________________________
    conv2d_8 (Conv2D)               (None, 76, 76, 256)  294912      leaky_re_lu_7[0][0]              
    __________________________________________________________________________________________________
    batch_normalization_8 (BatchNor (None, 76, 76, 256)  1024        conv2d_8[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_8 (LeakyReLU)       (None, 76, 76, 256)  0           batch_normalization_8[0][0]      
    __________________________________________________________________________________________________
    max_pooling2d_4 (MaxPooling2D)  (None, 38, 38, 256)  0           leaky_re_lu_8[0][0]              
    __________________________________________________________________________________________________
    conv2d_9 (Conv2D)               (None, 38, 38, 512)  1179648     max_pooling2d_4[0][0]            
    __________________________________________________________________________________________________
    batch_normalization_9 (BatchNor (None, 38, 38, 512)  2048        conv2d_9[0][0]                   
    __________________________________________________________________________________________________
    leaky_re_lu_9 (LeakyReLU)       (None, 38, 38, 512)  0           batch_normalization_9[0][0]      
    __________________________________________________________________________________________________
    conv2d_10 (Conv2D)              (None, 38, 38, 256)  131072      leaky_re_lu_9[0][0]              
    __________________________________________________________________________________________________
    batch_normalization_10 (BatchNo (None, 38, 38, 256)  1024        conv2d_10[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_10 (LeakyReLU)      (None, 38, 38, 256)  0           batch_normalization_10[0][0]     
    __________________________________________________________________________________________________
    conv2d_11 (Conv2D)              (None, 38, 38, 512)  1179648     leaky_re_lu_10[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_11 (BatchNo (None, 38, 38, 512)  2048        conv2d_11[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_11 (LeakyReLU)      (None, 38, 38, 512)  0           batch_normalization_11[0][0]     
    __________________________________________________________________________________________________
    conv2d_12 (Conv2D)              (None, 38, 38, 256)  131072      leaky_re_lu_11[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_12 (BatchNo (None, 38, 38, 256)  1024        conv2d_12[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_12 (LeakyReLU)      (None, 38, 38, 256)  0           batch_normalization_12[0][0]     
    __________________________________________________________________________________________________
    conv2d_13 (Conv2D)              (None, 38, 38, 512)  1179648     leaky_re_lu_12[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_13 (BatchNo (None, 38, 38, 512)  2048        conv2d_13[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_13 (LeakyReLU)      (None, 38, 38, 512)  0           batch_normalization_13[0][0]     
    __________________________________________________________________________________________________
    max_pooling2d_5 (MaxPooling2D)  (None, 19, 19, 512)  0           leaky_re_lu_13[0][0]             
    __________________________________________________________________________________________________
    conv2d_14 (Conv2D)              (None, 19, 19, 1024) 4718592     max_pooling2d_5[0][0]            
    __________________________________________________________________________________________________
    batch_normalization_14 (BatchNo (None, 19, 19, 1024) 4096        conv2d_14[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_14 (LeakyReLU)      (None, 19, 19, 1024) 0           batch_normalization_14[0][0]     
    __________________________________________________________________________________________________
    conv2d_15 (Conv2D)              (None, 19, 19, 512)  524288      leaky_re_lu_14[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_15 (BatchNo (None, 19, 19, 512)  2048        conv2d_15[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_15 (LeakyReLU)      (None, 19, 19, 512)  0           batch_normalization_15[0][0]     
    __________________________________________________________________________________________________
    conv2d_16 (Conv2D)              (None, 19, 19, 1024) 4718592     leaky_re_lu_15[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_16 (BatchNo (None, 19, 19, 1024) 4096        conv2d_16[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_16 (LeakyReLU)      (None, 19, 19, 1024) 0           batch_normalization_16[0][0]     
    __________________________________________________________________________________________________
    conv2d_17 (Conv2D)              (None, 19, 19, 512)  524288      leaky_re_lu_16[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_17 (BatchNo (None, 19, 19, 512)  2048        conv2d_17[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_17 (LeakyReLU)      (None, 19, 19, 512)  0           batch_normalization_17[0][0]     
    __________________________________________________________________________________________________
    conv2d_18 (Conv2D)              (None, 19, 19, 1024) 4718592     leaky_re_lu_17[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_18 (BatchNo (None, 19, 19, 1024) 4096        conv2d_18[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_18 (LeakyReLU)      (None, 19, 19, 1024) 0           batch_normalization_18[0][0]     
    __________________________________________________________________________________________________
    conv2d_19 (Conv2D)              (None, 19, 19, 1024) 9437184     leaky_re_lu_18[0][0]             
    __________________________________________________________________________________________________
    batch_normalization_19 (BatchNo (None, 19, 19, 1024) 4096        conv2d_19[0][0]                  
    __________________________________________________________________________________________________
    conv2d_21 (Conv2D)              (None, 38, 38, 64)   32768       leaky_re_lu_13[0][0]             
    __________________________________________________________________________________________________
    leaky_re_lu_19 (LeakyReLU)      (None, 19, 19, 1024) 0           batch_normalization_19[0][0]     
    __________________________________________________________________________________________________
    batch_normalization_21 (BatchNo (None, 38, 38, 64)   256         conv2d_21[0][0]                  
    __________________________________________________________________________________________________
    conv2d_20 (Conv2D)              (None, 19, 19, 1024) 9437184     leaky_re_lu_19[0][0]             
    __________________________________________________________________________________________________
    leaky_re_lu_21 (LeakyReLU)      (None, 38, 38, 64)   0           batch_normalization_21[0][0]     
    __________________________________________________________________________________________________
    batch_normalization_20 (BatchNo (None, 19, 19, 1024) 4096        conv2d_20[0][0]                  
    __________________________________________________________________________________________________
    space_to_depth_x2 (Lambda)      (None, 19, 19, 256)  0           leaky_re_lu_21[0][0]             
    __________________________________________________________________________________________________
    leaky_re_lu_20 (LeakyReLU)      (None, 19, 19, 1024) 0           batch_normalization_20[0][0]     
    __________________________________________________________________________________________________
    concatenate_1 (Concatenate)     (None, 19, 19, 1280) 0           space_to_depth_x2[0][0]          
                                                                     leaky_re_lu_20[0][0]             
    __________________________________________________________________________________________________
    conv2d_22 (Conv2D)              (None, 19, 19, 1024) 11796480    concatenate_1[0][0]              
    __________________________________________________________________________________________________
    batch_normalization_22 (BatchNo (None, 19, 19, 1024) 4096        conv2d_22[0][0]                  
    __________________________________________________________________________________________________
    leaky_re_lu_22 (LeakyReLU)      (None, 19, 19, 1024) 0           batch_normalization_22[0][0]     
    __________________________________________________________________________________________________
    conv2d_23 (Conv2D)              (None, 19, 19, 425)  435625      leaky_re_lu_22[0][0]             
    ==================================================================================================
    Total params: 50,983,561
    Trainable params: 50,962,889
    Non-trainable params: 20,672

    3-3 将model输入到 box tensors

    yolo_model的输出尺寸= (m, 19, 19, 5, 85) ，需要将其转为带有box参数信息的形式：

    #Convert final layer features to bounding box parameters
    yolo_outputs = yolo_head(yolo_model.output, anchors, len(class_names))

    此时的输出就是yolo模型的输出。

    3-4 boxs过滤：

    yolo_outputs输出了yolo_model的预测box。我们还需要对这些boxs进行过滤(调用之前定义的yolo_eval函数）：

    scores, boxes, classes = yolo_eval(yolo_outputs, image_shape)

    3-5 Run the graph on an image：

    图模型如下：

    1. yolo_model.input 用以接收输入到 yolo_model的数据。模型的计算结果为 yolo_model.output
    2. yolo_model.output 再通过 yolo_head处理后，作为 yolo_outputs
    3. yolo_outputs 通过 yolo_eval中的filter处理，输出其预测结果: scores, boxes, classes

    定义一个predict()函数，以运行graph,使YOLO可以对图像做探测。
    在运行TensorFlow session之前，我们需要先计算scores, boxes, classes(这三者是yolo_eval的输出)。
    对于输入的数据，我们需要先进行预处理：

    image, image_data = preprocess_image("images/" + image_file, model_image_size = (608, 608))

    其中：

    • image: 是原始图像，仅仅用于boxs的绘制，其他时候用不到。
    • image_data: 是该图像的矩阵表征，用于输入到CNN。

    具体代码如下：

    def predict(sess, image_file):
        """
        Runs the graph stored in "sess" to predict boxes for "image_file". Prints and plots the preditions.
    
        Arguments:
        sess -- your tensorflow/Keras session containing the YOLO graph
        image_file -- name of an image stored in the "images" folder.
    
        Returns:
        out_scores -- tensor of shape (None, ), scores of the predicted boxes
        out_boxes -- tensor of shape (None, 4), coordinates of the predicted boxes
        out_classes -- tensor of shape (None, ), class index of the predicted boxes
    
        Note: "None" actually represents the number of predicted boxes, it varies between 0 and max_boxes. 
        """
    
        # Preprocess your image
        image, image_data = preprocess_image("images/" + image_file, model_image_size = (608, 608))
    
        # Run the session with the correct tensors and choose the correct placeholders in the feed_dict.
        # You'll need to use feed_dict={yolo_model.input: ... , K.learning_phase(): 0})
        ### START CODE HERE ### (≈ 1 line)
        out_scores, out_boxes, out_classes = sess.run([scores,boxes,classes],feed_dict={yolo_model.input:image_data,K.learning_phase():0})
        ### END CODE HERE ###
    
        # Print predictions info
        print('Found {} boxes for {}'.format(len(out_boxes), image_file))
        # Generate colors for drawing bounding boxes.
        colors = generate_colors(class_names)
        # Draw bounding boxes on the image file
        draw_boxes(image, out_scores, out_boxes, out_classes, class_names, colors)
        # Save the predicted bounding box on the image
        image.save(os.path.join("out", image_file), quality=90)
        # Display the results in the notebook
        output_image = scipy.misc.imread(os.path.join("out", image_file))
        imshow(output_image)
    
        return out_scores, out_boxes, out_classes

    上述代码测试：

    out_scores, out_boxes, out_classes = predict(sess, "test.jpg")

    运行结果如下：

    Found 7 boxes for test.jpg
    car 0.60 (925, 285) (1045, 374)
    car 0.66 (706, 279) (786, 350)
    bus 0.67 (5, 266) (220, 407)
    car 0.70 (947, 324) (1280, 705)
    car 0.74 (159, 303) (346, 440)
    car 0.80 (761, 282) (942, 412)
    car 0.89 (367, 300) (745, 648)

    c:\users\jason\appdata\local\programs\python\python35\lib\site-packages\ipykernel_launcher.py:35: DeprecationWarning: `imread` is deprecated!
    `imread` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.
    Use ``imageio.imread`` instead.