目标检测

目标检测

一、目标检测概述

1.目标检测概述

2.目标检测算法介绍

3.目标检测的任务

4.目标定位的简单实现思路

在分类的时候我们直接输出各个类别的概率，如果再加上定位的话，我们可以考虑在网络的最后输出加上位置信息。下面我们考虑图中只有一个物体的检测时候，我们可以有以下方法去进行训练我们的模型

回归位置

两种Bounding box名称

二、R-CNN

1.目标检测-Overfeat模型

滑动窗口

Overfeat模型总结

2.目标检测-R-CNN模型

在CVPR 2014年中Ross Girshick提出R-CNN。论文名称：用于精确的对象检测和语义分割的丰富功能层次结构（Rich feature hierarchies for accurate object detection and semantic segmentation）

论文背景

完整R-CNN结构

候选区域（Region of Interest）得出（了解）

CNN特征提取

3.模型训练

CNN训练训练

分类器SVM训练

Boundingbox Regression

正负样本分配问题？

4.测试过程

非最大抑制（NMS）

import numpy as np

def nms(dets, thresh):
    # dets: 检测的 boxes 及对应的 scores
    # thresh: 设定的阈值

    # boxes 位置
    x1 = dets[:, 0]  
    y1 = dets[:, 1]
    x2 = dets[:, 2]
    y2 = dets[:, 3]

    # boxes scores
    scores = dets[:, 4]

    areas = (x2 - x1 + 1) * (y2 - y1 + 1) # 各 box 的面积
    order = scores.argsort()[::-1] # boxes 的按照 score 排序

    keep = [] # 记录保留下的 boxes
    while order.size > 0:
        i = order[0] # score 最大的 box 对应的 index
        keep.append(i) # 将本轮 score 最大的 box 的 index 保留

        # 计算剩余 boxes 与当前 box 的重叠程度 IoU
        xx1 = np.maximum(x1[i], x1[order[1:]])
        yy1 = np.maximum(y1[i], y1[order[1:]])
        xx2 = np.minimum(x2[i], x2[order[1:]])
        yy2 = np.minimum(y2[i], y2[order[1:]])

        w = np.maximum(0.0, xx2 - xx1 + 1) # IoU
        h = np.maximum(0.0, yy2 - yy1 + 1)
        inter = w * h
        ovr = inter / (areas[i] + areas[order[1:]] - inter)

        # 保留 IoU 小于设定阈值的 boxes
        inds = np.where(ovr <= thresh)[0]
        order = order[inds + 1]

    return keep

5.检测的评价指标

IoU交并比

IOU 实现（重点）

def union(au, bu, area_intersection):
    """
    计算并集
    :param au:
    :param bu:
    :param area_intersection:
    :return:
    """
    # 计算a的面积
    area_a = (au[2] - au[0]) * (au[3] - au[1])
    # 计算b的面积
    area_b = (bu[2] - bu[0]) * (bu[3] - bu[1])
    # a和b的面积-交集面积=总共面积
    area_union = area_a + area_b - area_intersection
    return area_union


def intersection(ai, bi):
    """
    计算交集
    :param ai:a框坐标
    :param bi:b框坐标
    :return:
    """
    # 1、取出交集的左上角点
    x = max(ai[0], bi[0])
    y = max(ai[1], bi[1])
    # 2、取出交集的右下角点，并减去左上角点值，计算出交集长宽
    w = min(ai[2], bi[2]) - x
    h = min(ai[3], bi[3]) - y
    # 3、如果一个为0，返回交集面积为0
    if w < 0 or h < 0:
        return 0
    return w*h


def iou(a, b):
    """
    计算交并比
    :param a: a框坐标
    :param b: b框坐标
    :return:
    """
    # 1、如果a,b 传入有问题
    if a[0] >= a[2] or a[1] >= a[3] or b[0] >= b[2] or b[1] >= b[3]:
        return 0.0

    # 2、计算IOU
    # 交集区域
    area_i = intersection(a, b)
    # 并集区域
    area_u = union(a, b, area_i)

    return float(area_i) / float(area_u + 1e-6)  # 防止分母为0，加一个稳定系数

平均精确率（mean average precision）mAP

6.论文R-CNN总结

特征提取对比

网络架构

流程总结

缺点

三、SPPNet

1.SPPNet介绍

SPPNet与RCNN对比

spatial pyramid pooling layer

映射

SPPNet 结果

代码实现

import tensorflow as tf
import math


def Spp_layer(feature_map, bins):
    ''' 使用 [3, 2, 1] denote 3*3, 2*2, 1*1 bins做金字塔赤化输出'''  
    # get feature map shape
    batch_size, a, _, _ = feature_map.get_shape().as_list()
    pooling_out_all = []

    for layer in range(len(bins)):
        # 计算每个不同池化尺寸的输出
        k_size = math.ceil(a / bins[layer])
        stride = math.floor(a / bins[layer])
        pooling_out = tf.nn.max_pool(feature_map,
                                     ksize=[1, k_size, k_size, 1],
                                     strides=[1, stride, stride, 1],
                                     padding='VALID')
        pooling_out_resized = tf.reshape(pooling_out, [batch_size, -1])
        pooling_out_all[layer] = pooling_out_resized
        # 特征图合并输出结果
    feature_map_out = tf.concat(axis=1, values=pooling_out_all)
    return feature_map_out

2.SPPNet总结

四、Fast R-CNN

1.Fast R-CNN

Fast R-CNN原理

RoI pooling

从预训练网络中进行训练

Fast RCNN检测

网络将图像（或图像金字塔，编码为图像列表）和待计算概率的R个候选框的列表作为输入。在测试的时候，R通常在2000左右。使用R-CNN算法的设置和对每个类别独立执行非最大抑制

2.Fast RCNN 效果

3.Fast R-CNN算法

五、Faster R-CNN

在Fast R-CNN还存在着瓶颈问题：Selective Search（选择性搜索）。要找出所有的候选框，那我们有没有一个更加高效的方法来求出这些候选框呢？

1.Faster R-CNN介绍

RPN网络

RPN原理

平移不变的anchor

2.Faster RCNN训练

RPN训练：

3.Faster R-CNN效果

4.Faster R-CNN总结

六、Faster RCNN接口介绍

1.COCO 数据集介绍

COCO数据集

下载

2.COCO API

格式

{
     "info":      #第一个info信息
          {       #数据集信息
                  "description": "COCO 2014 Dataset",
                  "url": "http://cocodataset.org",
                  "version": "1.0",
                  "year": 2014,
                  "contributor": "COCO Consortium",
                  "date_created": "2017/09/01"
         },
      "images":  #第二个图片信息，数组包含了多张图像
      [   {      #每张图像的具体信息
                  "license": 5,
                  "file_name": "COCO_train2014_000000057870.jpg",
                  "coco_url": "http://images.cocodataset.org/train2014/COCO_train2014_000000057870.jpg",
                  "height": 480,
                  "width": 640,
                  "date_captured": "2013-11-14 16:28:13",
                  "flickr_url": "http://farm4.staticflickr.com/3153/2970773875_164f0c0b83_z.jpg",
                  "id": 57870
           },
          ......
          ......   #此处省略很多图片
         {
                  "license": 4,
                  "file_name": "COCO_train2014_000000475546.jpg",
                  "http://images.cocodataset.org/train2014/COCO_train2014_000000475546.jpg",
                  "height": 375,
                  "width":500,；、
                  "date_captured": "2013-11-25 21:20:23",
                  "flickr_url": "http://farm1.staticflickr.com/167/423175046_6cd9d0205a_z.jpg",
                  "id": 475546
           }]，         #图像描述结束，下面开始介绍licenses


    "licenses":
         [ {
                  "url": "http://creativecommons.org/licenses/by-nc-sa/2.0/",
                  "id": 1,
                  "name": "Attribution-NonCommercial-ShareAlike License"
           },
            .....#此处省略七个license
            .....
         {
                  "url": "http://creativecommons.org/licenses/by-nc-nd/2.0/",
                  "id": 8,
                  "name": "Attribution-NonCommercial-NoDerivs License"
         }],

      "annotations":
      [   { 
                 "segmentation":[[312.29,562.89,402.25,511.49,400.96,425.38,398.39,372.69,
                                  388.11,332.85,318.71,325.14,295.58,305.86,269.88,314.86,
                                  258.31,337.99,217.19,321.29,182.49,343.13,141.37,348.27,
                                  132.37,358.55,159.36,377.83,116.95,421.53,167.07,499.92,
                                  232.61,560.32,300.72,571.89]],
                "area": 54652.9556,
                "iscrowd": 0,
                "image_id": 480023,
                "bbox": [116.95,305.86,285.3,266.03],
                "category_id": 58,"id": 86
          },
            .....#此处省略很多图像的分割标签
            .....
                "segmentation":[[312.29,562.89,402.25,511.49,400.96,425.38,398.39,372.69,
                                388.11,332.85,318.71,325.14,295.58,305.86,269.88,314.86,
                                258.31,337.99,217.19,321.29,182.49,343.13,141.37,348.27,
                                132.37,358.55,159.36,377.83,116.95,421.53,167.07,499.92,
                                232.61,560.32,300.72,571.89]],
              "area": 54652.9556,
              "iscrowd": 0,
              "image_id": 480023,
              "bbox": [116.95,305.86,285.3,266.03],
              "category_id": 58,
              "id": 86
          },
      "categories":#类别信息
     [   {
              "supercategory": "person",
              "id": 1,
              "name": "person"
          },
              .......#此处省略很多图像的类标签
              .......
          {
              "supercategory": "vehicle",
              "id": 2,
              "name": "bicycle"
          },
        {
              "supercategory": "kitchen",#大类
              "id": 50,
              "name": "spoon"
        }

利用 json 文件实例化 COCO API 对象

COCO接口获取数据代码

获取API

class CocoDataSet(object):
    def __init__(self, 
                 dataset_dir,  # dataset_dir: The root directory of the COCO dataset.
                 subset,  # subset: What to load (train, val).
                 flip_ratio=0,  # flip_ratio: Float. The ratio of flipping an image and its bounding boxes.
                 pad_mode='fixed',  # pad_mode: Which padded method to use (fixed, non-fixed)
                 mean=(0, 0, 0),  # mean: Tuple. Image mean.
                 std=(1, 1, 1),  # Tuple. Image standard deviation.
                 scale=(1024, 800),  # Tuple of two integers.
                 debug=False):

获取2017数据集代码案例

import os
import tensorflow as tf
from tensorflow import keras
import numpy as np

from detection.datasets import coco, data_generator
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

train_dataset = coco.CocoDataSet('./data/coco2017', 'train',
                                     flip_ratio=0.5,
                                     pad_mode='fixed',
                                     mean=(123.675, 116.28, 103.53),
                                     std=(1., 1., 1.),
                                     scale=(800, 1216))
train_generator = data_generator.DataGenerator(train_dataset)
train_tf_dataset = tf.data.Dataset.from_generator(
  train_generator, (tf.float32, tf.float32, tf.float32, tf.int32))
train_tf_dataset = train_tf_dataset.batch(1).prefetch(100).shuffle(100)

for (batch, inputs) in enumerate(train_tf_dataset):
  batch_imgs, batch_metas, batch_bboxes, batch_labels = inputs
  print(batch_imgs, batch_metas, batch_bboxes, batch_labels)

打印结果当中包含图片数据，批次元信息，GT位置信息，目标标签

3.Faster RCNN 训练以及预测

Faster RCNN流程图详解

开源keras Faster RCNN 模型代码结构

FaterRCNN 模型训练使用

from detection.models.detectors import faster_rcnn

# 2、建立模型
num_classes = len(train_dataset.get_categories())
model = faster_rcnn.FasterRCNN(num_classes=num_classes)
optimizer = keras.optimizers.SGD(1e-3, momentum=0.9, nesterov=True)

# 3、迭代训练
for epoch in range(1):

    loss_history = []
    for (batch, inputs) in enumerate(train_tf_dataset):

        batch_imgs, batch_metas, batch_bboxes, batch_labels = inputs
        with tf.GradientTape() as tape:
            rpn_class_loss, rpn_bbox_loss, rcnn_class_loss, rcnn_bbox_loss = model(
                (batch_imgs, batch_metas, batch_bboxes, batch_labels), training=True)

            loss_value = rpn_class_loss + rpn_bbox_loss + rcnn_class_loss + rcnn_bbox_loss

        grads = tape.gradient(loss_value, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))

        loss_history.append(loss_value.numpy())

        if batch % 10 == 0:
            print('迭代次数: %d, 批次数: %d, 损失: %f' % (epoch+1, batch+1, np.mean(loss_history)))

4.代码核心解析

模型主体过程

 def call(self, inputs, training=True):
        """

        :param inputs: [1, 1216, 1216, 3], [1, 11], [1, 14, 4], [1, 14]
        :param training:
        :return:
        """
        if training: # training
            imgs, img_metas, gt_boxes, gt_class_ids = inputs
        else: # inference
            imgs, img_metas = inputs
        # 1、主干网络计算，ResNet五层特征输出保留
        # [1, 304, 304, 256] => [1, 152, 152, 512]=>[1,76,76,1024]=>[1,38,38,2048]
        C2, C3, C4, C5 = self.backbone(imgs, 
                                       training=training)
        # [1, 304, 304, 256] <= [1, 152, 152, 256]<=[1,76,76,256]<=[1,38,38,256]=>[1,19,19,256]
        # FPN进行处理金字塔特征
        P2, P3, P4, P5, P6 = self.neck([C2, C3, C4, C5], 
                                       training=training)
        # 用于RPN计算
        rpn_feature_maps = [P2, P3, P4, P5, P6]
        # 用于后面的RCNN计算
        rcnn_feature_maps = [P2, P3, P4, P5]
        # [1, 369303, 2] [1, 369303, 2], [1, 369303, 4], includes all anchors on pyramid level of features
        # 2、RPN计算输出2000候选框
        # 得到输出结果
        rpn_class_logits, rpn_probs, rpn_deltas = self.rpn_head(
            rpn_feature_maps, training=training)
        # [369303, 4] => [215169, 4], valid => [6000, 4], performance =>[2000, 4],  NMS
        # 过滤
        proposals_list = self.rpn_head.get_proposals(
            rpn_probs, rpn_deltas, img_metas)
        # 进行区域分配GT，标记正负样本
        if training: # get target value for these proposal target label and target delta
            rois_list, rcnn_target_matchs_list, rcnn_target_deltas_list = \
                self.bbox_target.build_targets(
                    proposals_list, gt_boxes, gt_class_ids, img_metas)
        else:
            rois_list = proposals_list
        # rois_list only contains coordinates, rcnn_feature_maps save the 5 features data=>[192,7,7,256]
        # 对多层特征进行ROIpooling操作
        # Implements ROI Pooling on multiple levels of the feature pyramid.
        pooled_regions_list = self.roi_align(#
            (rois_list, rcnn_feature_maps, img_metas), training=training)
        # [192, 81], [192, 81], [192, 81, 4]
        # RCNN部分计算输出
        rcnn_class_logits_list, rcnn_probs_list, rcnn_deltas_list = \
            self.bbox_head(pooled_regions_list, training=training)
                # 3、RPN损失以及输出RCNN的损失计算
        if training:
            rpn_class_loss, rpn_bbox_loss = self.rpn_head.loss(
                rpn_class_logits, rpn_deltas, gt_boxes, gt_class_ids, img_metas)

            rcnn_class_loss, rcnn_bbox_loss = self.bbox_head.loss(
                rcnn_class_logits_list, rcnn_deltas_list, 
                rcnn_target_matchs_list, rcnn_target_deltas_list)

            return [rpn_class_loss, rpn_bbox_loss, 
                    rcnn_class_loss, rcnn_bbox_loss]
        else:
            detections_list = self.bbox_head.get_bboxes(
                rcnn_probs_list, rcnn_deltas_list, rois_list, img_metas)

            return detections_list

FPN（了解）

RPN

def call(self, inputs, training=True):
    '''
    Args
    ---
        inputs: [batch_size, feat_map_height, feat_map_width, channels] 
            one level of pyramid feat-maps.

    Returns
    ---
        rpn_class_logits: [batch_size, num_anchors, 2]
        rpn_probs: [batch_size, num_anchors, 2]
        rpn_deltas: [batch_size, num_anchors, 4]
    '''

    layer_outputs = []

    for feat in inputs: # for every anchors feature maps
        """
        (1, 304, 304, 256)
        (1, 152, 152, 256)
        (1, 76, 76, 256)
        (1, 38, 38, 256)
        (1, 19, 19, 256)
        rpn_class_raw: (1, 304, 304, 6)
        rpn_class_logits: (1, 277248, 2)
        rpn_delta_pred: (1, 304, 304, 12)
        rpn_deltas: (1, 277248, 4)
        rpn_class_raw: (1, 152, 152, 6)
        rpn_class_logits: (1, 69312, 2)
        rpn_delta_pred: (1, 152, 152, 12)
        rpn_deltas: (1, 69312, 4)
        rpn_class_raw: (1, 76, 76, 6)
        rpn_class_logits: (1, 17328, 2)
        rpn_delta_pred: (1, 76, 76, 12)
        rpn_deltas: (1, 17328, 4)
        rpn_class_raw: (1, 38, 38, 6)
        rpn_class_logits: (1, 4332, 2)
        rpn_delta_pred: (1, 38, 38, 12)
        rpn_deltas: (1, 4332, 4)
        rpn_class_raw: (1, 19, 19, 6)
        rpn_class_logits: (1, 1083, 2)
        rpn_delta_pred: (1, 19, 19, 12)
        rpn_deltas: (1, 1083, 4)

        """
        # print(feat.shape)
        shared = self.rpn_conv_shared(feat)
        shared = tf.nn.relu(shared)

        x = self.rpn_class_raw(shared)
        # print('rpn_class_raw:', x.shape)
        rpn_class_logits = tf.reshape(x, [tf.shape(x)[0], -1, 2])
        rpn_probs = tf.nn.softmax(rpn_class_logits)
        # print('rpn_class_logits:', rpn_class_logits.shape)

        x = self.rpn_delta_pred(shared)
        # print('rpn_delta_pred:', x.shape)
        rpn_deltas = tf.reshape(x, [tf.shape(x)[0], -1, 4])
        # print('rpn_deltas:', rpn_deltas.shape)

        layer_outputs.append([rpn_class_logits, rpn_probs, rpn_deltas])
        # print(rpn_class_logits.shape, rpn_probs.shape, rpn_deltas.shape)
        """
        (1, 277248, 2) (1, 277248, 2) (1, 277248, 4)
        (1, 69312, 2) (1, 69312, 2) (1, 69312, 4)
        (1, 17328, 2) (1, 17328, 2) (1, 17328, 4)
        (1, 4332, 2) (1, 4332, 2) (1, 4332, 4)
        (1, 1083, 2) (1, 1083, 2) (1, 1083, 4)

        """

    outputs = list(zip(*layer_outputs))
    outputs = [tf.concat(list(o), axis=1) for o in outputs]
    rpn_class_logits, rpn_probs, rpn_deltas = outputs
    # (1, 369303, 2) (1, 369303, 2) (1, 369303, 4)
    # print(rpn_class_logits.shape, rpn_probs.shape, rpn_deltas.shape)

    return rpn_class_logits, rpn_probs, rpn_deltas

根据输出进行筛选proposals：过滤分数、进行NMS等操作

    def get_proposals(self, 
                      rpn_probs, 
                      rpn_deltas, 
                      img_metas, 
                      with_probs=False):
        '''
        Calculate proposals.

        Args
        ---
            rpn_probs: [batch_size, num_anchors, (bg prob, fg prob)]
            rpn_deltas: [batch_size, num_anchors, (dy, dx, log(dh), log(dw))]
            img_metas: [batch_size, 11]
            with_probs: bool.

        Returns
        ---
            proposals_list: list of [num_proposals, (y1, x1, y2, x2)] in 
                normalized coordinates if with_probs is False. 
                Otherwise, the shape of proposals in proposals_list is 
                [num_proposals, (y1, x1, y2, x2, score)]

        Note that num_proposals is no more than proposal_count. And different 
           images in one batch may have different num_proposals.
        '''
        anchors, valid_flags = self.generator.generate_pyramid_anchors(img_metas)
        # [369303, 4], [b, 11]
        # [b, N, (background prob, foreground prob)], get anchor's foreground prob, [1, 369303]
        rpn_probs = rpn_probs[:, :, 1]
        # [[1216, 1216]]
        pad_shapes = calc_pad_shapes(img_metas)

        proposals_list = [
            self._get_proposals_single(
                rpn_probs[i], rpn_deltas[i], anchors, valid_flags[i], pad_shapes[i], with_probs)
            for i in range(img_metas.shape[0])
        ]

        return proposals_list

计算RPN的损失，分类和回归损失：

def loss(self, rpn_class_logits, rpn_deltas, gt_boxes, gt_class_ids, img_metas):
  """

        :param rpn_class_logits: [N, 2]
        :param rpn_deltas: [N, 4]
        :param gt_boxes:  [GT_N]
        :param gt_class_ids:  [GT_N]
        :param img_metas: [11]
        :return:
        """
  # valid_flags indicates anchors located in padded area or not.
  anchors, valid_flags = self.generator.generate_pyramid_anchors(img_metas)

  # 进行anhor匹配
  rpn_target_matchs, rpn_target_deltas = self.anchor_target.build_targets(
    anchors, valid_flags, gt_boxes, gt_class_ids)

  rpn_class_loss = self.rpn_class_loss(
    rpn_target_matchs, rpn_class_logits)
  rpn_bbox_loss = self.rpn_bbox_loss(
    rpn_target_deltas, rpn_target_matchs, rpn_deltas)

  return rpn_class_loss, rpn_bbox_loss

测试输出

读取验证集合一张图片，输入模型进行预测输出

def test():

    train_dataset = coco.CocoDataSet('./data/coco2017', 'val')

    # 获取数据和模型
    train_generator = data_generator.DataGenerator(train_dataset)
    tf_dataset = tf.data.Dataset.from_generator(train_generator,
                                                (tf.float32, tf.float32, tf.float32, tf.float32))
    tf_dataset = tf_dataset.batch(1).prefetch(100).shuffle(100)
    num_classes = len(train_dataset.get_categories())
    model = faster_rcnn.FasterRCNN(num_classes=num_classes)

    for (batch, inputs) in enumerate(tf_dataset):
        img, img_meta, _, _ = inputs
        print(img, img_meta)

        detections_list = model((img, img_meta), training=False)

        print(detections_list)


if __name__ == '__main__':
    # train()
    test()

输出结果

[<tf.Tensor: id=10027, shape=(20, 6), dtype=float32, numpy=
array([[0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 3.300e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [0.000e+00, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 7.000e+00, 1.000e+00],
       [1.024e+03, 1.024e+03, 1.024e+03, 1.024e+03, 2.700e+01, 1.000e+00]],
      dtype=float32)>]

七、YOLO算法

1.YOLO介绍

YOLO结构

统一检测过程理解

单元格(grid cell)

网格输出筛选

训练

2.与Faster R-CNN比较

3.YOLO总结

八、YOLOV2&V3

1.YOLOV2介绍

改进方法

结构与训练

联合分类与检测训练-YOLO9000（了解）

总结

2.YOLOV3介绍

改进

总结

九、SSD算法原理

1.SSD

简介

结构

流程

Detector & classifier

PriorBox层-default boxes

1、SSD网络prior_box:打印出来的形状为：
Tensor("concat_2:0", shape=(?, 7308, 8), dtype=float32)

# 其中某一层的结构输出
 layer {
  name: "conv6_2_mbox_priorbox"
  type: "PriorBox"
  bottom: "conv6_2"
  bottom: "data"
  top: "conv6_2_mbox_priorbox"
  prior_box_param {
    min_size: 111.0
    max_size: 162.0
    aspect_ratio: 2.0
    aspect_ratio: 3.0
    flip: true
    clip: false
    variance: 0.10000000149
    variance: 0.10000000149
    variance: 0.20000000298
    variance: 0.20000000298
    step: 32.0
    offset: 0.5
}

prior box使用过程

def decode(default_boxes, locs, variance=[0.1, 0.2]):
    """ 对default_boxes进行解码到坐标coordinates
    Args:
        default_boxes: tensor (num_default, 4)
                       of format (cx, cy, w, h)
        locs: tensor (batch_size, num_default, 4)
              of format (cx, cy, w, h)
        variance: variance for center point and size
    Returns:
        boxes: tensor (num_default, 4)
               of format (xmin, ymin, xmax, ymax)
    """
    # 解码过程
    locs = tf.concat([
        locs[..., :2] * variance[0] *
        default_boxes[:, 2:] + default_boxes[:, :2],
        tf.math.exp(locs[..., 2:] * variance[1]) * default_boxes[:, 2:]], axis=-1)

    # (cx, cy, w, h)->(xmin, ymin, xmax, ymax)
    boxes = transform_center_to_corner(locs)

    return boxes

def encode(default_boxes, boxes, variance=[0.1, 0.2]):
    """ Compute regression values
    Args:
        default_boxes: tensor (num_default, 4)
                       of format (cx, cy, w, h)
        boxes: tensor (num_default, 4)
               of format (xmin, ymin, xmax, ymax)
        variance: variance for center point and size
    Returns:
        locs: regression values, tensor (num_default, 4)
    """
    # (xmin, ymin, xmax, ymax)->(cx, cy, w, h)
    transformed_boxes = transform_corner_to_center(boxes)

    # 编码过程
    locs = tf.concat([
        (transformed_boxes[..., :2] - default_boxes[:, :2]
         ) / (default_boxes[:, 2:] * variance[0]),
        tf.math.log(transformed_boxes[..., 2:] / default_boxes[:, 2:]) / variance[1]],
        axis=-1)

    return locs

2.训练与测试流程

train流程

test流程

3.SSD实验与总结

实验

总结

十、案例：KITTI 人、车物体检测

1.KITTI 数据集介绍

2.YOLOV3源码实现分析

源码模型下载

YOLOV3-Tensorflow2.0源码分析

def YoloV3(size=None, channels=3, anchors=yolo_anchors,
           masks=yolo_anchor_masks, classes=80, training=False):
    x = inputs = Input([size, size, channels])

    x_36, x_61, x = Darknet(name='yolo_darknet')(x)

    # 下面通过YOLO的后续
    x = YoloConv(512, name='yolo_conv_0')(x)
    output_0 = YoloOutput(512, len(masks[0]), classes, name='yolo_output_0')(x)

    x = YoloConv(256, name='yolo_conv_1')((x, x_61))
    output_1 = YoloOutput(256, len(masks[1]), classes, name='yolo_output_1')(x)

    x = YoloConv(128, name='yolo_conv_2')((x, x_36))
    output_2 = YoloOutput(128, len(masks[2]), classes, name='yolo_output_2')(x)

    if training:
        return Model(inputs, (output_0, output_1, output_2), name='yolov3')

    boxes_0 = Lambda(lambda x: yolo_boxes(x, anchors[masks[0]], classes),
                     name='yolo_boxes_0')(output_0)
    boxes_1 = Lambda(lambda x: yolo_boxes(x, anchors[masks[1]], classes),
                     name='yolo_boxes_1')(output_1)
    boxes_2 = Lambda(lambda x: yolo_boxes(x, anchors[masks[2]], classes),
                     name='yolo_boxes_2')(output_2)

    outputs = Lambda(lambda x: yolo_nms(x, anchors, masks, classes),
                     name='yolo_nms')((boxes_0[:3], boxes_1[:3], boxes_2[:3]))

    return Model(inputs, outputs, name='yolov3')

def YoloV3Tiny(size=None, channels=3, anchors=yolo_tiny_anchors,
               masks=yolo_tiny_anchor_masks, classes=80, training=False):
    x = inputs = Input([size, size, channels])

    x_8, x = DarknetTiny(name='yolo_darknet')(x)

    x = YoloConvTiny(256, name='yolo_conv_0')(x)
    output_0 = YoloOutput(256, len(masks[0]), classes, name='yolo_output_0')(x)

    x = YoloConvTiny(128, name='yolo_conv_1')((x, x_8))
    output_1 = YoloOutput(128, len(masks[1]), classes, name='yolo_output_1')(x)

    if training:
        return Model(inputs, (output_0, output_1), name='yolov3')

    boxes_0 = Lambda(lambda x: yolo_boxes(x, anchors[masks[0]], classes),
                     name='yolo_boxes_0')(output_0)
    boxes_1 = Lambda(lambda x: yolo_boxes(x, anchors[masks[1]], classes),
                     name='yolo_boxes_1')(output_1)
    outputs = Lambda(lambda x: yolo_nms(x, anchors, masks, classes),
                     name='yolo_nms')((boxes_0[:3], boxes_1[:3]))
    return Model(inputs, outputs, name='yolov3_tiny')

# 1、重复若干层DarknetBlock，里面包含残差模块，输出包含三层特征
def Darknet(name=None):
    x = inputs = Input([None, None, 3])
    x = DarknetConv(x, 32, 3)
    x = DarknetBlock(x, 64, 1)
    x = DarknetBlock(x, 128, 2)  # skip connection
    x = x_36 = DarknetBlock(x, 256, 8)  # skip connection
    x = x_61 = DarknetBlock(x, 512, 8)
    x = DarknetBlock(x, 1024, 4)
    return tf.keras.Model(inputs, (x_36, x_61, x), name=name)

def DarknetBlock(x, filters, blocks):
    x = DarknetConv(x, filters, 3, strides=2)
    for _ in range(blocks):
        x = DarknetResidual(x, filters)
    return x

# 2、实现删除残差模块，进行若干层采样，并且输出只有两层特征
def DarknetTiny(name=None):
    x = inputs = Input([None, None, 3])
    x = DarknetConv(x, 16, 3)
    x = MaxPool2D(2, 2, 'same')(x)
    x = DarknetConv(x, 32, 3)
    x = MaxPool2D(2, 2, 'same')(x)
    x = DarknetConv(x, 64, 3)
    x = MaxPool2D(2, 2, 'same')(x)
    x = DarknetConv(x, 128, 3)
    x = MaxPool2D(2, 2, 'same')(x)
    x = x_8 = DarknetConv(x, 256, 3)  # skip connection
    x = MaxPool2D(2, 2, 'same')(x)
    x = DarknetConv(x, 512, 3)
    x = MaxPool2D(2, 1, 'same')(x)
    x = DarknetConv(x, 1024, 3)
    return tf.keras.Model(inputs, (x_8, x), name=name)

3.KITTI人车检测项目

项目目录与模块

项目步骤分析

4.数据集类型转换-KITTI数据集转换成TFRecords文件

TFRecord-TensorFlow 数据集存储格式

[
    {   # example 1 (tf.train.Example)
        'feature_1': tf.train.Feature,
        ...
        'feature_k': tf.train.Feature
    },
    ...
    {   # example N (tf.train.Example)
        'feature_1': tf.train.Feature,
        ...
        'feature_k': tf.train.Feature
    }
]
# 字典结构如
feature = {
                'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image])),  # 图片是一个 Bytes 对象
                'label': tf.train.Feature(int64_list=tf.train.Int64List(value=[label]))  # 标签是一个 Int 对象
            }

import os
import tensorflow as tf
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

train_cats_dir = './cats_vs_dogs/train/cats/'
train_dogs_dir = './cats_vs_dogs/train/dogs/'
tfrecord_file = './cats_vs_dogs/train.tfrecords'

train_cat_filenames = [train_cats_dir + filename for filename in os.listdir(train_cats_dir)]
train_dog_filenames = [train_dogs_dir + filename for filename in os.listdir(train_dogs_dir)]
train_filenames = train_cat_filenames + train_dog_filenames
train_labels = [0] * len(train_cat_filenames) + [1] * len(train_dog_filenames)  # 将 cat 类的标签设为0，dog 类的标签设为1

with tf.io.TFRecordWriter(tfrecord_file) as writer:
        for filename, label in zip(train_filenames, train_labels):
            # 1、读取数据集图片到内存，image 为一个 Byte 类型的字符串
            image = open(filename, 'rb').read()
            # 2、建立 tf.train.Feature 字典
            feature = {
                'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image])),  # 图片是一个 Bytes 对象
                'label': tf.train.Feature(int64_list=tf.train.Int64List(value=[label]))  # 标签是一个 Int 对象
            }
            # 3、通过字典建立 Example
            example = tf.train.Example(features=tf.train.Features(feature=feature))
            # 4\将Example序列化并写入 TFRecord 文件
            writer.write(example.SerializeToString())

# 1、读取 TFRecord 文件
    raw_dataset = tf.data.TFRecordDataset(tfrecord_file)

    # 2、定义Feature结构，告诉解码器每个Feature的类型是什么
    feature_description = {
        'image': tf.io.FixedLenFeature([], tf.string),
        'label': tf.io.FixedLenFeature([], tf.int64),
    }

    # 3、将 TFRecord 文件中的每一个序列化的 tf.train.Example 解码
    def _parse_example(example_string):
        feature_dict = tf.io.parse_single_example(example_string, feature_description)
        feature_dict['image'] = tf.io.decode_jpeg(feature_dict['image'])  # 解码JPEG图片
        return feature_dict['image'], feature_dict['label']

    dataset = raw_dataset.map(_parse_example)

    for image, label in dataset:
        print(image, label)

KITTI数据集转换成TFRecords文件

import hashlib
import io
import os

import numpy as np
import PIL.Image as pil
from PIL import Image
import tensorflow as tf

import feature_parse
from IoU import iou
import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument('--data_dir', type=str, default='../data/kitti/',
                    help='kitti数据集的位置')
parser.add_argument('--output_path', type=str, default='../data/kitti_tfrecords/',
                    help='TFRecord文件的输出位置')
parser.add_argument('--classes_to_use', default='car ,van ,truck ,pedestrian ,cyclist ,tram', help='KITTI中需要检测的类别')
parser.add_argument('--validation_set_size', type=int, default=500,
                    help='验证集数据集使用大小')

编写的主函数逻辑如下：

def convert_kitti_to_tfrecords(data_dir, output_path, classes_to_use,
                               validation_set_size):
    """
    将KITTI detection 转换成TFRecords.
    :param data_dir: 源数据目录
    :param output_path: 输出文件目录
    :param classes_to_use: 选择需要使用的类别
    :param validation_set_size: 验证集大小
    :return:
    """
    train_count = 0
    val_count = 0

    # 1、创建KITTI训练和验证集的tfrecord位置
    # 标注信息位置
    annotation_dir = os.path.join(data_dir,
                                  'training',
                                  'label_2')

    # 图片位置
    image_dir = os.path.join(data_dir,
                             'data_object_image_2',
                             'training',
                             'image_2')

    train_writer = tf.io.TFRecordWriter(output_path + 'train.tfrecord')
    val_writer = tf.io.TFRecordWriter(output_path + 'val.tfrecord')

    # 2、列出所有的图片，进行每张图片的内容和标注信息的获取，写入到tfrecords文件
    images = sorted(os.listdir(image_dir))
    for img_name in images:

        # （1）获取当前图片的编号数据，并拼接读取相应标注文件
        img_num = int(img_name.split('.')[0])

        # （2）读取标签文件函数
        # 整数需要进行填充成与标签文件相同的6位字符串
        img_anno = read_annotation_file(os.path.join(annotation_dir,
                                                     str(img_num).zfill(6) + '.txt'))

        # （3）过滤标签函数
        # 当前图片的标注中 过滤掉一些没有用的类别和dontcare区域的annotations
        annotation_for_image = filter_annotations(img_anno, classes_to_use)

        # （4）写入训练和验证集合TFRecord文件
        # 读取拼接的图片路径，然后与过滤之后的标注结果进行合并到一个example中
        image_path = os.path.join(image_dir, img_name)
        example = prepare_example(image_path, annotation_for_image)
        # 如果小于验证集数量大小就直接写入验证集，否则写入训练集
        is_validation_img = img_num < validation_set_size
        if is_validation_img:
            val_writer.write(example.SerializeToString())
            val_count += 1
        else:
            train_writer.write(example.SerializeToString())
            train_count += 1

    train_writer.close()
    val_writer.close()


def main(args):

    convert_kitti_to_tfrecords(
        data_dir=args.data_dir,
        output_path=args.output_path,
        classes_to_use=args.classes_to_use.split(','),
        validation_set_size=args.validation_set_size)


if __name__ == '__main__':

    args = parser.parse_args(sys.argv[1:])
    main(args)

读取读取标签文件函数

def read_annotation_file(filename):

    with open(filename) as f:
        content = f.readlines()
    # 分割解析内容
    content = [x.strip().split(' ') for x in content]
    # 保存内容到字典结构
    anno = dict()
    anno['type'] = np.array([x[0].lower() for x in content])
    anno['truncated'] = np.array([float(x[1]) for x in content])
    anno['occluded'] = np.array([int(x[2]) for x in content])
    anno['alpha'] = np.array([float(x[3]) for x in content])

    anno['2d_bbox_left'] = np.array([float(x[4]) for x in content])
    anno['2d_bbox_top'] = np.array([float(x[5]) for x in content])
    anno['2d_bbox_right'] = np.array([float(x[6]) for x in content])
    anno['2d_bbox_bottom'] = np.array([float(x[7]) for x in content])
    return anno

过滤标签函数

def filter_annotations(img_all_annotations, used_classes):
    """
    过滤掉一些没有用的类别和dontcare区域的annotations
    :param img_all_annotations: 图片的所有标注
    :param used_classes: 需要留下记录的列别
    :return:
    """
    img_filtered_annotations = {}

    # 1、过滤这个图片中标注的我们训练指定不需要的类别，把索引记录下来
    # 方便后面在处理对应的一些坐标时候使用
    relevant_annotation_indices = [
        i for i, x in enumerate(img_all_annotations['type']) if x in used_classes
    ]

    # 2、获取过滤后的下标对应某个标记物体的其它信息
    for key in img_all_annotations.keys():
        img_filtered_annotations[key] = (
            img_all_annotations[key][relevant_annotation_indices])

    # 3、如果dontcare在我们要获取的类别里面，也进行组合获取，然后过滤相关的bboxes不符合要求的
    if 'dontcare' in used_classes:
        dont_care_indices = [i for i,
                             x in enumerate(img_filtered_annotations['type'])
                             if x == 'dontcare']

        # bounding box的格式[y_min, x_min, y_max, x_max]
        all_boxes = np.stack([img_filtered_annotations['2d_bbox_top'],
                              img_filtered_annotations['2d_bbox_left'],
                              img_filtered_annotations['2d_bbox_bottom'],
                              img_filtered_annotations['2d_bbox_right']],
                             axis=1)

        # 计算bboxesIOU，比如这样的
        # Truck 0.00 0 -1.57 599.41 156.40 629.75 189.25 2.85 2.63 12.34 0.47 1.49 69.44 -1.56
        # DontCare -1 -1 -10 503.89 169.71 590.61 190.13 -1 -1 -1 -1000 -1000 -1000 -10
        # DontCare -1 -1 -10 511.35 174.96 527.81 187.45 -1 -1 -1 -1000 -1000 -1000 -10
        # DontCare -1 -1 -10 532.37 176.35 542.68 185.27 -1 -1 -1 -1000 -1000 -1000 -10
        # DontCare -1 -1 -10 559.62 175.83 575.40 183.15 -1 -1 -1 -1000 -1000 -1000 -10
        ious = iou(boxes1=all_boxes,
                   boxes2=all_boxes[dont_care_indices])

        # 删除所有 bounding boxes 与 dontcare region 重叠的区域
        if ious.size > 0:
            # 找出下标
            boxes_to_remove = np.amax(ious, axis=1) > 0.0
            for key in img_all_annotations.keys():
                img_filtered_annotations[key] = (
                    img_filtered_annotations[key][np.logical_not(boxes_to_remove)])

    return img_filtered_annotations

读取文件标准信息、过滤标注信息、进行构造example的feature字典

def prepare_example(image_path, annotations):
    """
    对一个图片的Annotations转换成tf.Example proto.
    :param image_path:
    :param annotations:
    :return:
    """
    # 1、读取图片内容，转换成数组格式
    with open(image_path, 'rb') as fid:
        encoded_png = fid.read()
    encoded_png_io = io.BytesIO(encoded_png)
    image = pil.open(encoded_png_io)
    image = np.asarray(image)

    # 2、构造协议中需要的字典键的值
    # sha256加密结果
    key = hashlib.sha256(encoded_png).hexdigest()

    # 进行坐标处理
    width = int(image.shape[1])
    height = int(image.shape[0])
    # 存储极坐标归一化格式
    xmin_norm = annotations['2d_bbox_left'] / float(width)
    ymin_norm = annotations['2d_bbox_top'] / float(height)
    xmax_norm = annotations['2d_bbox_right'] / float(width)
    ymax_norm = annotations['2d_bbox_bottom'] / float(height)

    # 其他信息，难度以及字符串类别
    difficult_obj = [0] * len(xmin_norm)
    classes_text = [x.encode('utf8') for x in annotations['type']]

    # 3、构造协议example
    example = tf.train.Example(features=tf.train.Features(feature={
        'image/height': feature_parse.int64_feature(height),
        'image/width': feature_parse.int64_feature(width),
        'image/filename': feature_parse.bytes_feature(image_path.encode('utf8')),
        'image/source_id': feature_parse.bytes_feature(image_path.encode('utf8')),
        'image/key/sha256': feature_parse.bytes_feature(key.encode('utf8')),
        'image/encoded': feature_parse.bytes_feature(encoded_png),
        'image/format': feature_parse.bytes_feature('png'.encode('utf8')),
        'image/object/bbox/xmin': feature_parse.float_list_feature(xmin_norm),
        'image/object/bbox/xmax': feature_parse.float_list_feature(xmax_norm),
        'image/object/bbox/ymin': feature_parse.float_list_feature(ymin_norm),
        'image/object/bbox/ymax': feature_parse.float_list_feature(ymax_norm),
        'image/object/class/text': feature_parse.bytes_list_feature(classes_text),
        'image/object/difficult': feature_parse.int64_list_feature(difficult_obj),
        'image/object/truncated': feature_parse.float_list_feature(
            annotations['truncated'])
    }))

    return example

十一、案例：KITTI人、车检测案例-训练、检测

1.训练过程实现

import logging
import tensorflow as tf
import numpy as np
import cv2
import sys
import argparse
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

from tensorflow.keras.callbacks import (
    ReduceLROnPlateau,
    EarlyStopping,
    ModelCheckpoint,
    TensorBoard
)
from yolov3_tf2.models import (
    YoloV3, YoloV3Tiny, YoloLoss,
    yolo_anchors, yolo_anchor_masks,
    yolo_tiny_anchors, yolo_tiny_anchor_masks
)
from yolov3_tf2.utils import freeze_all
import yolov3_tf2.dataset as dataset

parser = argparse.ArgumentParser()
parser.add_argument('--dataset', type=str, default='./data/kitti_tfrecords/train.tfrecord',
                    help='训练数据集路径')
parser.add_argument('--val_dataset', type=str, default='./data/kitti_tfrecords/val.tfrecord',
                    help='验证集目录')
parser.add_argument('--tiny', type=bool, default=False, help='加载的模型类型yolov3 or yolov3-tiny')
parser.add_argument('--weights', type=str, default='./checkpoints/yolov3_train_1.tf',
                    help='模型预训练权重路径')
parser.add_argument('--classes', type=str, default='./data/kitti.names',
                    help='类别文件路径')
parser.add_argument('--mode', type=str, default='fit', choices=['fit', 'eager_tf'],
                    help='fit: model.fit模式, eager_tf: 自定义GradientTape模式')
parser.add_argument('--transfer', type=str, default='none', choices=['none', 'darknet',
                                                                     'no_output', 'frozen',
                                                                     'fine_tune'],
                    help='none: 全部进行训练'
                         '迁移学习并冻结所有, fine_tune: 迁移并只冻结darknet')
parser.add_argument('--size', type=int, default=416,
                    help='图片大小')
parser.add_argument('--epochs', type=int, default=2,
                    help='迭代次数')
parser.add_argument('--batch_size', type=int, default=8,
                    help='每批次大小')
parser.add_argument('--learning_rate', type=float, default=1e-3,
                    help='学习率')
parser.add_argument('--num_classes', type=int, default=6,
                    help='类别数量')

# 2、获取传入参数的训练数据
if args.dataset:
    train_dataset = dataset.load_tfrecord_dataset(
        args.dataset, args.classes)
train_dataset = train_dataset.shuffle(buffer_size=1024)
train_dataset = train_dataset.batch(args.batch_size)
train_dataset = train_dataset.map(lambda x, y: (
    dataset.transform_images(x, args.size),
    dataset.transform_targets(y, anchors, anchor_masks, 6)))
train_dataset = train_dataset.prefetch(
    buffer_size=tf.data.experimental.AUTOTUNE)

# 3、获取传入参数的验证集数据
if args.val_dataset:
    val_dataset = dataset.load_tfrecord_dataset(
        args.val_dataset, args.classes)
val_dataset = val_dataset.batch(args.batch_size)
val_dataset = val_dataset.map(lambda x, y: (
    dataset.transform_images(x, args.size),
    dataset.transform_targets(y, anchors, anchor_masks, 6)))

# 4、判断是否进行迁移学习
if args.transfer != 'none':
    # 加载与训练模型'./data/yolov3.weights'
    model.load_weights(args.weights)
    if args.transfer == 'fine_tune':
        # 冻结darknet
        darknet = model.get_layer('yolo_darknet')
        freeze_all(darknet)
    elif args.mode == 'frozen':
        # 冻结所有层
        freeze_all(model)
    else:
        # 重置网络后端结构
        if args.tiny:
            init_model = YoloV3Tiny(
                args.size, training=True, classes=args.num_classes)
        else:
            init_model = YoloV3(
                args.size, training=True, classes=args.num_classes)

        # 如果迁移指的是darknet
        if args.transfer == 'darknet':
            # 获取网络的权重
            for l in model.layers:
                if l.name != 'yolo_darknet' and l.name.startswith('yolo_'):
                    l.set_weights(init_model.get_layer(
                        l.name).get_weights())
                else:
                    freeze_all(l)
        elif args.transfer == 'no_output':
            for l in model.layers:
                if l.name.startswith('yolo_output'):
                    l.set_weights(init_model.get_layer(
                        l.name).get_weights())
                else:
                    freeze_all(l)

# 需要从模型文件中导入utils中的freeze_all函数
def freeze_all(model, frozen=True):
    model.trainable = not frozen
    if isinstance(model, tf.keras.Model):
        for l in model.layers:
            freeze_all(l, frozen)

if args.mode == 'eager_tf':
    # 1、定义评估方式
    avg_loss = tf.keras.metrics.Mean('loss', dtype=tf.float32)
    avg_val_loss = tf.keras.metrics.Mean('val_loss', dtype=tf.float32)

    # 2、迭代优化
    for epoch in range(1, args.epochs + 1):
        for batch, (images, labels) in enumerate(train_dataset):
            with tf.GradientTape() as tape:
                # 1、计算模型输出和损失
                outputs = model(images, training=True)
                regularization_loss = tf.reduce_sum(model.losses)
                pred_loss = []
                for output, label, loss_fn in zip(outputs, labels, loss):
                    # 根据输出和标签计算出损失
                    pred_loss.append(loss_fn(label, output))

                # 计算总损失 = 平均损失 + regularization_loss
                total_loss = tf.reduce_sum(pred_loss) + regularization_loss

            # 计算梯度以及更新梯度
            grads = tape.gradient(total_loss, model.trainable_variables)
            optimizer.apply_gradients(
                zip(grads, model.trainable_variables))

            # 打印日志
            logging.info("{}_train_{}, {}, {}".format(
                epoch, batch, total_loss.numpy(),
                list(map(lambda x: np.sum(x.numpy()), pred_loss))))
            avg_loss.update_state(total_loss)

        # 验证数据集验证输出计算
        for batch, (images, labels) in enumerate(val_dataset):
            outputs = model(images)
            # 求损失
            regularization_loss = tf.reduce_sum(model.losses)
            pred_loss = []
            # 输出结果和标签计算损失
            for output, label, loss_fn in zip(outputs, labels, loss):
                pred_loss.append(loss_fn(label, output))
            total_loss = tf.reduce_sum(pred_loss) + regularization_loss

            # 打印总损失
            logging.info("{}_val_{}, {}, {}".format(
                epoch, batch, total_loss.numpy(),
                list(map(lambda x: np.sum(x.numpy()), pred_loss))))
            avg_val_loss.update_state(total_loss)

        logging.info("{}, train: {}, val: {}".format(
            epoch,
            avg_loss.result().numpy(),
            avg_val_loss.result().numpy()))
        # 保存模型位置
        avg_loss.reset_states()
        avg_val_loss.reset_states()
        model.save_weights(
            'checkpoints/yolov3_train_{}.tf'.format(epoch))
else:
    # 指定相关回调函数，自定义需求，对于检测来讲不需要太多的优化算法方式
    model.compile(optimizer=optimizer, loss=loss)

    callbacks = [
      EarlyStopping(patience=3, verbose=1),
      ModelCheckpoint('checkpoints/yolov3_train_{epoch}.tf',
                      verbose=1, save_weights_only=True),
      TensorBoard(log_dir='logs')
    ]

      history = model.fit(train_dataset,
                          epochs=args.epochs,
                          callbacks=callbacks,
                          validation_data=val_dataset)

2.测试过程实现

import time
import logging
import cv2
import numpy as np
import tensorflow as tf
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

from yolov3_tf2.models import (
    YoloV3, YoloV3Tiny
)
from yolov3_tf2.dataset import transform_images
from yolov3_tf2.utils import draw_outputs
import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument('--classes', type=str, default='./data/kitti.names',
                    help='类别配置路径')
parser.add_argument('--weights', type=str, default='./checkpoints/yolov3_train_1.tf',
                    help='训练好的模型位置')
parser.add_argument('--tiny', type=bool, default=False, help='加载的模型类型yolov3 or yolov3-tiny')
parser.add_argument('--size', type=int, default=416,
                    help='图片大小')
parser.add_argument('--image', type=str, default='./data/kitti/data_object_image_2/testing/image_2/000008.png',
                    help='输入预测图片的位置')
parser.add_argument('--output', type=str, default='./output.jpg',
                    help='输出图片结果的位置')
parser.add_argument('--num_classes', type=int, default=6,
                    help='总共类别数量')

整体过程实现逻辑：

def main(args):
    # 1、初始化模型并加载权重
    if args.tiny:
        yolo = YoloV3Tiny(classes=args.num_classes)
    else:
        yolo = YoloV3(classes=args.num_classes)

    yolo.load_weights(args.weights)
    logging.info('加载模型权重weights')

    # 加载目标类型
    class_names = [c.strip() for c in open(args.classes).readlines()]

    # 2、加载图片处理图片并使用模型进行预测
    img = tf.image.decode_image(open(args.image, 'rb').read(), channels=3)
    img = tf.expand_dims(img, 0)
    img = transform_images(img, args.size)

    # 记录时间
    t1 = time.time()
    boxes, scores, classes, nums = yolo(img)
    t2 = time.time()
    logging.info('耗时: {}'.format(t2 - t1))

    logging.info('检测结果:')
    print(boxes, scores, classes, nums)
    for i in range(nums[0]):
        logging.info('\t{}, {}, {}'.format(class_names[int(classes[0][i])],
                                           np.array(scores[0][i]),
                                           np.array(boxes[0][i])))
    # 3、显示图片并将图片框画出
    img = cv2.imread(args.image)
    img = draw_outputs(img, (boxes, scores, classes, nums), class_names)
    cv2.imwrite(args.output, img)
    logging.info('output saved to: {}'.format(args.output))


if __name__ == '__main__':
    args = parser.parse_args(sys.argv[1:])
    main(args)

def yolo_nms(outputs, anchors, masks, classes):
    # boxes, conf, type
    b, c, t = [], [], []

    for o in outputs:
        b.append(tf.reshape(o[0], (tf.shape(o[0])[0], -1, tf.shape(o[0])[-1])))
        c.append(tf.reshape(o[1], (tf.shape(o[1])[0], -1, tf.shape(o[1])[-1])))
        t.append(tf.reshape(o[2], (tf.shape(o[2])[0], -1, tf.shape(o[2])[-1])))

    bbox = tf.concat(b, axis=1)
    confidence = tf.concat(c, axis=1)
    class_probs = tf.concat(t, axis=1)

    scores = confidence * class_probs
    boxes, scores, classes, valid_detections = tf.image.combined_non_max_suppression(
        boxes=tf.reshape(bbox, (tf.shape(bbox)[0], -1, 1, 4)),
        scores=tf.reshape(
            scores, (tf.shape(scores)[0], -1, tf.shape(scores)[-1])),
        max_output_size_per_class=100,
        max_total_size=100,
        iou_threshold=0.5,
        score_threshold=0.5
    )

    return boxes, scores, classes, valid_detections