Faster-RCNN-tensorflow-Python2.7-smallcorgi版本训练自己的数据集实战笔记（二）
Faster-RCNN-tensorflow-Python2.7-smallcorgi版本训练自己的数据集实战笔记（二）
参考资料
https://github.com/DexterInd/GoPiGo/issues/210
https://stackoverflow.com/questions/14219092/bash-script-and-bin-bashm-bad-interpreter-no-such-file-or-directory
xuforeverloveCSDN博客
https://github.com/smallcorgi/Faster-RCNN_TF
训练PASCAL VOC 2007数据集
-
1、下载数据集
这边可以使用以下命令从官方下载数据，不过如果没法速度可能会很慢，所以这边有我的百度云分享链接：https://pan.baidu.com/s/1L5sA1LmhXofIuYvUmlb7Gw
提取码：indswget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCdevkit_08-Jun-2007.tar如果你使用百度云下载的话里面应该包括4个文件，除了使用指令下载的3个tar文件外，还要多一个
VGG_imagenet.npy文件，不用怀疑，另外一个文件需要在之后的步骤中使用到，如果你是使用指令下载的话，可以去官方提供的下载地址Google Drive或Dropbox下载该文件 -
2、解压并放置数据集和相关文件
首先进入你所下载的文件目录打开终端，输入以下指令解压3个tar文件
tar xvf VOCtrainval_06-Nov-2007.tar tar xvf VOCtest_06-Nov-2007.tar tar xvf VOCdevkit_08-Jun-2007.tar解压后的文件会在该目录生成一个
VOCdevkit文件夹，打开该文件夹应该含有以下文件：[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-9N6ywRj5-1593142595437)(E:\project_1\faster-RCNN\mdimage\vocdevkit.png)]
接下来的步骤和官方给的步骤有所区别，但是原理上是一样的。这里我们不做文件链接，需要将上述
VOCdevkit文件夹改名为VOCdevkit2007并将其放到FRCNN_ROOT/data路径下，然后在该目录下自己新建一个文件夹命名为pretrain_model并将VGG_image.npy文件放到该文件夹下。所以此时你FRCNN_ROOT/data下应该是这样的：[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-eMKwZYAT-1593142595442)(E:\project_1\faster-RCNN\mdimage\data.png)]
-
3、运行脚本进行训练和测试
回到主目录输入以下指令进行训练和测试：
cd $FRCN_ROOT ./experiments/scripts/faster_rcnn_end2end.sh $DEVICE $DEVICE_ID VGG16 pascal_voc #上个语句的示例 ./experiments/scripts/faster_rcnn_end2end.sh gpu 0 VGG16 pascal_voc这一步可能出现的问题及解决办法：
问题一：权限问题
bash: ./experiments/scripts/faster_rcnn_end2end.sh: Permission denied解决办法：在终端输入以下指令
chmod 777 ./experiments/scripts/faster_rcnn_end2end.sh问题二：sh文件的格式问题
/bin/bash^M: bad interpreter: No such file or directory解决办法：将sh文件格式有doc格式更改为unix格式，具体方式参考 https://blog.csdn.net/xuforeverlove/article/details/85334726?utm_medium=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromMachineLearnPai2-1.nonecase
问题三：文件索引错误
如果你使用官方方法可能程序可能寻找不到
train.txt文件而出现如下报错：AssertionError: Path does not exist: /home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/data/VOCdevkit2007/VOC2007/ImageSets/Main/trainval.txt解决办法：这边提供两种办法
1、根据官方方法重新建立文件链接，注意文件夹路径和链接文件名
2、根据我提供的方法直接将上述
VOCdevkit文件夹改名为VOCdevkit2007并将其放到FRCNN_ROOT/data路径下。问题四：模型文件保存错误
由于
./experiments/scripts/faster_rcnn_end2end.sh文件中对 ``faster_rcnn_end2end_faster_rcnn_end2end_VGG16_.2020-06-19_14-46-49.txt`的路径设置有误，程序找不到该路径，无法创建该文件。报错如下：grep: experiments/logs/faster_rcnn_end2end_VGG16_.2020-06-19_14-46-49.txt: No such file or directory解决办法：在
./experiments文件夹下创建logs文件夹或者修改./experiments/scripts/faster_rcnn_end2end.sh文件中对于LOG变量的定义，将路径更改到已有路径。问题五：由于新旧版本变化导致的模型保存文件变为三个（旧版为一个）从而test_net.py接受的参数数量错误
报错如下：
usage: test_net.py [-h] [--device DEVICE] [--device_id DEVICE_ID] [--def PROTOTXT] [--weights MODEL] [--cfg CFG_FILE] [--wait WAIT] [--imdb IMDB_NAME] [--comp] [--network NETWORK_NAME] test_net.py: error: argument --weights: expected one argument解决办法：更改文件
ROOT/lib/fast_rcnn/train.py将函数train_net中的self.saver = tf.train.Saver(max_to_keep=100)更改为self.saver = tf.train.Saver(max_to_keep=100,write_version=saver_pb2.SaverDef.V1)并在文件最前面添加from tensorflow.core.protobuf import saver_pb2正常运行结果如下：
VOC07 metric? Yes AP for aeroplane = -1.0000 AP for bicycle = -1.0000 AP for bird = -1.0000 AP for boat = -1.0000 AP for bottle = -1.0000 AP for bus = 0.0075 AP for car = 0.0288 AP for cat = 0.0130 AP for chair = 0.0036 AP for cow = -1.0000 AP for diningtable = -1.0000 AP for dog = 0.0000 AP for horse = 0.0048 AP for motorbike = 0.0035 AP for person = 0.0489 AP for pottedplant = 0.0068 AP for sheep = -1.0000 AP for sofa = -1.0000 AP for train = 0.0091 AP for tvmonitor = -1.0000 Mean AP = -0.4937 ~~~~~~~~ Results: -1.000 -1.000 -1.000 -1.000 -1.000 0.008 0.029 0.013 0.004 -1.000 -1.000 0.000 0.005 0.003 0.049 0.007 -1.000 -1.000 0.009 -1.000 -0.494 ~~~~~~~~ -------------------------------------------------------------- Results computed with the **unofficial** Python eval code. Results should be very close to the official MATLAB eval code. Recompute with `./tools/reval.py --matlab ...` for your paper. -- Thanks, The Management -------------------------------------------------------------- real 8m0.321s user 97m19.020s sys 1m43.796s
训练自己的数据集
详细步骤见 https://blog.csdn.net/zcy0xy/article/details/79614862
遇见的问题及解决办法
问题一
Traceback (most recent call last):
File "./tools/train_net.py", line 96, in
max_iters=args.max_iters)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 260, in train_net
roidb = filter_roidb(roidb)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 251, in filter_roidb
filtered_roidb = [entry for entry in roidb if is_valid(entry)]
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 240, in is_valid
overlaps = entry['max_overlaps']
KeyError: 'max_overlaps'
解决办法：
删除文件夹 ROOT/data/cache和 ROOT/data/VOCdevkit2007/annotations_cache
问题二
Computing bounding-box regression targets...
Traceback (most recent call last):
File "./tools/train_net.py", line 96, in
max_iters=args.max_iters)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 263, in train_net
sw = SolverWrapper(sess, saver, network, imdb, roidb, output_dir, pretrained_model=pretrained_model)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 39, in __init__
self.bbox_means, self.bbox_stds = rdl_roidb.add_bbox_regression_targets(roidb)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/roi_data_layer/roidb.py", line 48, in add_bbox_regression_targets
assert len(roidb) > 0
AssertionError
解决办法：
索引文件的记录格式有问题，根据PASCAL VOC 2007的格式查看索引文件记录的方式差异，根据差异修改 generate_txt.py文件相关代码。
问题三
Traceback (most recent call last):
File "./tools/train_net.py", line 83, in
roidb = get_training_roidb(imdb)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 205, in get_training_roidb
imdb.append_flipped_images()
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/datasets/imdb.py", line 106, in append_flipped_images
widths = self._get_widths()
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/datasets/imdb.py", line 102, in _get_widths
for i in xrange(self.num_images)]
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/datasets/pascal_voc.py", line 64, in image_path_at
return self.image_path_from_index(self._image_index[i])
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/datasets/pascal_voc.py", line 73, in image_path_from_index
'Path does not exist: {}'.format(image_path)
AssertionError: Path does not exist: /home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/data/VOCdevkit2007/VOC2007/JPEGImages/b38.jpg
解决办法：
这是图片文件的后缀名大小写不通用产生的问题，可以更改图片文件名，将后缀名全部改为小写的jpg，或者更改 ROOT/lib/datasets/pascal_voc.py文件，具体更改如下：
将
self._image_ext = '.jpg'
更改为：
self._image_ext = '.JPG'
问题四
Traceback (most recent call last):
File "./tools/train_net.py", line 83, in
roidb = get_training_roidb(imdb)
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/fast_rcnn/train.py", line 205, in get_training_roidb
imdb.append_flipped_images()
File "/home/qianslab/linbin/chaosheng/Faster_RCNN/Faster-RCNN_TF-python2.7/tools/../lib/datasets/imdb.py", line 108, in append_flipped_images
boxes = self.roidb[i]['boxes'].copy()
IndexError: list index out of range
解决办法：
ROOT/data/cache 文件夹没有更新所致，只需要把该文件夹删除后重新更新即可。