深度学习（三）——tiny YOLO算法实现实时目标检测（tensorflow实现）
一、背景介绍
YOLO算法全称You Only Look Once，是Joseph Redmon等人于15年3月发表的一篇文章。本实验目标为实现YOLO算法，借鉴了一部分材料，最终实现了轻量级的简化版YOLO——tiny YOLO，其优势在于实现简单，目标检测迅速。
[1]文章链接：https://arxiv.org/abs/1506.02640
[2]YOLO官网链接：https://pjreddie.com/darknet/yolo/
二、算法原理简述
相较于RCNN系列算法，YOLO算法最大的创新在于将物体检测作为回归问题来求解，而RCNN系列算法是将目标检测用一个region proposal + CNN来作为分类问题求解。 如下图所示，YOLO通过对输入图像进行推测，得到图中所有物体的位置及其所属类别的相应概率
YOLO的网络模型结构包含有24个卷积层和2个全链接层，其具体结构如下：
作者将YOLO算法应用于了不同数据集，进行过算法准确度的验证，平均来看，YOLO的目标检测准确度约为60%左右，这个精度已经算不错了。同时，YOLO的识别速度可以达到45帧，改进版的fast YOLO可以达到155帧，下面是从官网获取的关于COCO Dataset的模型应用结果统计：
从中可以看到， Tiny YOLO虽然准确度平均只有23.7%，但是其识别速度可以达到244帧。
下面再给出论文里的模型识别结果图，效果还是不错的：
最后，附上几个网上关于YOLO模型几个比较好的解读：
[3]YOLO_原理详述
[4][目标检测]YOLO原理
本文重点是实现简化版的tiny YOLO模型，主要参考了代码：
[5]https://github.com/gliese581gg/YOLO_tensorflow
三、算法实现
1.所用文件
首先要介绍一下所有用到的文件及其位置的安放。我的文件具体包含：
-- test (测试图像文件夹)
|------ 000001.jpg (测试文件)
-- weights (权重文件夹)
|------ YOLO_tiny.ckpt (权值文件)
-- main.py (运行文件)
首先是test文件夹，里面放置需要测试的jpg文件就可以了。
其次是weights文件夹，里面放置的是作者训练好的ckpt文件，该文件的下载可以从google drive中下载：
不过从google drive中下载需要自己手动，而且下载速度会非常慢，我将该文件传到了自己的百度云上，有需要的话可以自行下载：
链接：https://pan.baidu.com/s/1bug9ZX5P4OghfRxvE389fQ
提取码：8tqz
最后是main.py文件，具体如何写下面我会详细介绍。
2.算法实现
我的main.py文件是参考了程序YOLO_tiny_tf.py，并加上了自己的一些改进实现的。先来看一下tiny YOLO的模型结构：
可以看到，tiny YOLO基本为VGG19模型的改进，然后将模型应用于图像中，对目标进行检测，可以按照这个思路，编写main.py文件，具体代码为：
import numpy as np
import tensorflow as tf
import cv2
import time
class YOLO_TF:
fromfile = 'test/person.jpg'
tofile_img = 'test/output.jpg'
tofile_txt = 'test/output.txt'
imshow = True
filewrite_img = False
filewrite_txt = False
disp_console = True
weights_file = 'weights/YOLO_tiny.ckpt'
alpha = 0.1
threshold = 0.2
iou_threshold = 0.5
num_class = 20
num_box = 2
grid_size = 7
classes = ["aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"]
w_img = 640
h_img = 480
def __init__(self, fromfile=None, tofile_img=None, tofile_txt=None):
self.fromfile = fromfile
self.tofile_img = tofile_img
self.filewrite_img = True
self.tofile_txt = tofile_txt
self.filewrite_txt = True
self.imshow = True
self.disp_console = True
self.build_networks()
if self.fromfile is not None: self.detect_from_file(self.fromfile)
def build_networks(self):
if self.disp_console: print("Building YOLO_tiny graph...")
self.x = tf.placeholder('float32', [None, 448, 448, 3])
self.conv_1 = self.conv_layer(1, self.x, 16, 3, 1)
self.pool_2 = self.pooling_layer(2, self.conv_1, 2, 2)
self.conv_3 = self.conv_layer(3, self.pool_2, 32, 3, 1)
self.pool_4 = self.pooling_layer(4, self.conv_3, 2, 2)
self.conv_5 = self.conv_layer(5, self.pool_4, 64, 3, 1)
self.pool_6 = self.pooling_layer(6, self.conv_5, 2, 2)
self.conv_7 = self.conv_layer(7, self.pool_6, 128, 3, 1)
self.pool_8 = self.pooling_layer(8, self.conv_7, 2, 2)
self.conv_9 = self.conv_layer(9, self.pool_8, 256, 3, 1)
self.pool_10 = self.pooling_layer(10, self.conv_9, 2, 2)
self.conv_11 = self.conv_layer(11, self.pool_10, 512, 3, 1)
self.pool_12 = self.pooling_layer(12, self.conv_11, 2, 2)
self.conv_13 = self.conv_layer(13, self.pool_12, 1024, 3, 1)
self.conv_14 = self.conv_layer(14, self.conv_13, 1024, 3, 1)
self.conv_15 = self.conv_layer(15, self.conv_14, 1024, 3, 1)
self.fc_16 = self.fc_layer(16, self.conv_15, 256, flat=True, linear=False)
self.fc_17 = self.fc_layer(17, self.fc_16, 4096, flat=False, linear=False)
# skip dropout_18
self.fc_19 = self.fc_layer(19, self.fc_17, 1470, flat=False, linear=True)
self.sess = tf.Session()
self.sess.run(tf.global_variables_initializer())
self.saver = tf.train.Saver()
self.saver.restore(self.sess, self.weights_file)
if self.disp_console: print("Loading complete!" + '\n')
def conv_layer(self, idx, inputs, filters, size, stride):
channels = inputs.get_shape()[3]
weight = tf.Variable(tf.truncated_normal([size, size, int(channels), filters], stddev=0.1))
biases = tf.Variable(tf.constant(0.1, shape=[filters]))
pad_size = size // 2
pad_mat = np.array([[0, 0], [pad_size, pad_size], [pad_size, pad_size], [0, 0]])
inputs_pad = tf.pad(inputs, pad_mat)
conv = tf.nn.conv2d(inputs_pad, weight, strides=[1, stride, stride, 1], padding='VALID',
name=str(idx) + '_conv')
conv_biased = tf.add(conv, biases, name=str(idx) + '_conv_biased')
if self.disp_console: print(
' Layer %d : Type = Conv, Size = %d * %d, Stride = %d, Filters = %d, Input channels = %d' % (
idx, size, size, stride, filters, int(channels)))
return tf.maximum(self.alpha * conv_biased, conv_biased, name=str(idx) + '_leaky_relu')
def pooling_layer(self, idx, inputs, size, stride):
if self.disp_console: print(
' Layer %d : Type = Pool, Size = %d * %d, Stride = %d' % (idx, size, size, stride))
return tf.nn.max_pool(inputs, ksize=[1, size, size, 1], strides=[1, stride, stride, 1], padding='SAME',
name=str(idx) + '_pool')
def fc_layer(self, idx, inputs, hiddens, flat=False, linear=False):
input_shape = inputs.get_shape().as_list()
if flat:
dim = input_shape[1] * input_shape[2] * input_shape[3]
inputs_transposed = tf.transpose(inputs, (0, 3, 1, 2))
inputs_processed = tf.reshape(inputs_transposed, [-1, dim])
else:
dim = input_shape[1]
inputs_processed = inputs
weight = tf.Variable(tf.truncated_normal([dim, hiddens], stddev=0.1))
biases = tf.Variable(tf.constant(0.1, shape=[hiddens]))
if self.disp_console: print(
' Layer %d : Type = Full, Hidden = %d, Input dimension = %d, Flat = %d, Activation = %d' % (
idx, hiddens, int(dim), int(flat), 1 - int(linear)))
if linear: return tf.add(tf.matmul(inputs_processed, weight), biases, name=str(idx) + '_fc')
ip = tf.add(tf.matmul(inputs_processed, weight), biases)
return tf.maximum(self.alpha * ip, ip, name=str(idx) + '_fc')
def detect_from_cvmat(self, img):
s = time.time()
self.h_img, self.w_img, _ = img.shape
img_resized = cv2.resize(img, (448, 448))
img_RGB = cv2.cvtColor(img_resized, cv2.COLOR_BGR2RGB)
img_resized_np = np.asarray(img_RGB)
inputs = np.zeros((1, 448, 448, 3), dtype='float32')
inputs[0] = (img_resized_np / 255.0) * 2.0 - 1.0
in_dict = {self.x: inputs}
net_output = self.sess.run(self.fc_19, feed_dict=in_dict)
self.result = self.interpret_output(net_output[0])
self.show_results(img, self.result)
strtime = str(time.time() - s)
if self.disp_console: print('Elapsed time : ' + strtime + ' secs' + '\n')
def detect_from_file(self, filename):
if self.disp_console: print('Detect from ' + filename)
img = cv2.imread(filename)
self.detect_from_cvmat(img)
def interpret_output(self, output):
probs = np.zeros((7, 7, 2, 20))
class_probs = np.reshape(output[0:980], (7, 7, 20))
scales = np.reshape(output[980:1078], (7, 7, 2))
boxes = np.reshape(output[1078:], (7, 7, 2, 4))
offset = np.transpose(np.reshape(np.array([np.arange(7)] * 14), (2, 7, 7)), (1, 2, 0))
boxes[:, :, :, 0] += offset
boxes[:, :, :, 1] += np.transpose(offset, (1, 0, 2))
boxes[:, :, :, 0:2] = boxes[:, :, :, 0:2] / 7.0
boxes[:, :, :, 2] = np.multiply(boxes[:, :, :, 2], boxes[:, :, :, 2])
boxes[:, :, :, 3] = np.multiply(boxes[:, :, :, 3], boxes[:, :, :, 3])
boxes[:, :, :, 0] *= self.w_img
boxes[:, :, :, 1] *= self.h_img
boxes[:, :, :, 2] *= self.w_img
boxes[:, :, :, 3] *= self.h_img
for i in range(2):
for j in range(20):
probs[:, :, i, j] = np.multiply(class_probs[:, :, j], scales[:, :, i])
filter_mat_probs = np.array(probs >= self.threshold, dtype='bool')
filter_mat_boxes = np.nonzero(filter_mat_probs)
boxes_filtered = boxes[filter_mat_boxes[0], filter_mat_boxes[1], filter_mat_boxes[2]]
probs_filtered = probs[filter_mat_probs]
classes_num_filtered = np.argmax(filter_mat_probs, axis=3)[
filter_mat_boxes[0], filter_mat_boxes[1], filter_mat_boxes[2]]
argsort = np.array(np.argsort(probs_filtered))[::-1]
boxes_filtered = boxes_filtered[argsort]
probs_filtered = probs_filtered[argsort]
classes_num_filtered = classes_num_filtered[argsort]
for i in range(len(boxes_filtered)):
if probs_filtered[i] == 0: continue
for j in range(i + 1, len(boxes_filtered)):
if self.iou(boxes_filtered[i], boxes_filtered[j]) > self.iou_threshold:
probs_filtered[j] = 0.0
filter_iou = np.array(probs_filtered > 0.0, dtype='bool')
boxes_filtered = boxes_filtered[filter_iou]
probs_filtered = probs_filtered[filter_iou]
classes_num_filtered = classes_num_filtered[filter_iou]
result = []
for i in range(len(boxes_filtered)):
result.append([self.classes[classes_num_filtered[i]], boxes_filtered[i][0], boxes_filtered[i][1],
boxes_filtered[i][2], boxes_filtered[i][3], probs_filtered[i]])
return result
def show_results(self, img, results):
img_cp = img.copy()
if self.filewrite_txt:
ftxt = open(self.tofile_txt, 'w')
for i in range(len(results)):
x = int(results[i][1])
y = int(results[i][2])
w = int(results[i][3]) // 2
h = int(results[i][4]) // 2
if self.disp_console: print(
' class : ' + results[i][0] + ' , [x,y,w,h]=[' + str(x) + ',' + str(y) + ',' + str(
int(results[i][3])) + ',' + str(int(results[i][4])) + '], Confidence = ' + str(results[i][5]))
if self.filewrite_img or self.imshow:
cv2.rectangle(img_cp, (x - w, y - h), (x + w, y + h), (0, 255, 0), 2)
cv2.rectangle(img_cp, (x - w, y - h - 20), (x + w, y - h), (125, 125, 125), -1)
cv2.putText(img_cp, results[i][0] + ' : %.2f' % results[i][5], (x - w + 5, y - h - 7),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0), 1)
if self.filewrite_txt:
ftxt.write(results[i][0] + ',' + str(x) + ',' + str(y) + ',' + str(w) + ',' + str(h) + ',' + str(
results[i][5]) + '\n')
if self.filewrite_img:
if self.disp_console: print(' image file writed : ' + self.tofile_img)
cv2.imwrite(self.tofile_img, img_cp)
if self.imshow:
cv2.imshow('YOLO_tiny detection', img_cp)
cv2.waitKey(1)
if self.filewrite_txt:
if self.disp_console: print(' txt file writed : ' + self.tofile_txt)
ftxt.close()
def iou(self, box1, box2):
tb = min(box1[0] + 0.5 * box1[2], box2[0] + 0.5 * box2[2]) - max(box1[0] - 0.5 * box1[2],
box2[0] - 0.5 * box2[2])
lr = min(box1[1] + 0.5 * box1[3], box2[1] + 0.5 * box2[3]) - max(box1[1] - 0.5 * box1[3],
box2[1] - 0.5 * box2[3])
if tb < 0 or lr < 0:
intersection = 0
else:
intersection = tb * lr
return intersection / (box1[2] * box1[3] + box2[2] * box2[3] - intersection)
if __name__ == '__main__':
fromfile = 'test/000001.jpg'
tofile_img = 'test/output.jpg'
tofile_txt = 'test/output.txt'
yolo = YOLO_TF(fromfile=fromfile, tofile_img=tofile_img, tofile_txt=tofile_txt)
cv2.waitKey(1000)四、效果测试
直接运行上述代码，便可执行程序。根据代码：
classes = ["aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"]
tiny YOLO只可识别上述常见的20类对象。关于上述代码的使用，每次测试图像时，只用修改倒数第5行的fromfile参数，然后直接运行便可执行目标检测。
下面给出目标检测的效果，虽然人检测了出来，但是自行车没有被检测到，还有将猫错误识别成狗的：
目前来看，虽然识别精度不高，但是主要对象还是能够识别出来的。
五、分析
1.tiny YOLO目前是需要下载别人训练好的文件进行实验，如何训练还有待于进一步学习。
2.tiny YOLO目前的识别精度不是很高，不过识别速度很快。另外对于一些具有重叠部分的对象，其识别效果可能会比较差。