faster RCNN 训练新数据集（kitti）

基于pytorch版的faster rcnn源码：https://github.com/jwyang/faster-rcnn.pytorch

一、训练(trainval)

1. 制作voc格式的kitti数据集，并链接到data/

数据集软链接
格式

ln -s $VOCdevkit VOCdevkit2007

我们的：（这三个都要链过去，只链一个1819不够的）

cd data/
ln -s /home/zhaoxl/jiaqun/lujialin/data/VOCdevkit2018 VOCdevkit2018
ln -s /home/zhaoxl/jiaqun/lujialin/data/VOCdevkit2019 VOCdevkit2019
ln -s /home/zhaoxl/jiaqun/lujialin/data/VOCdevkit1819 VOCdevkit1819

注：
直接将kitti的label(.txt)转换成voc格式的(.xml)

这是最简单方便的办法，因为faster rcnn源码中，制作imdb数据的过程中集成了pascalvoc.py 和coco.py ，也就是已经写好了对着两种数据格式的解析。如果不转换格式，就需要自己写解析文件（将.txt转为数据训练时需要用的imdb）

2. train_val.py中仿照pascal_voc数据集改动一些内容

-（1）42行左右，–dataset中把预设值改成自制数据集的名字’citykitti’

parser.add_argument('--dataset', dest='dataset',
                      help='training dataset',
                      default='citykitti', type=str)

（2）159行左右， if args.dataset == “pascal_voc”:后面添加一个elif，内容如下，用于将自制数据集根据‘数据集/年份/trainval’，制作imdb丢进gpu训练，其中：

• voc_2018是自制kitti数据集
• voc_2019是自制cityscape数据集
• 两者相加表示两个一起训练
• 在训练（trainval.py）的时候只会用到args.imdb_name，在测试（test.py）的时候会用到args.imdbval_name

elif args.dataset == "citykitti":
      args.imdb_name = "voc_2018_train+voc_2019_train"
      args.imdbval_name = "voc_2018_val+voc_2019_val"
      args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '50']

3. lib/datasets/factory.py 解析voc数据集时，加上新加入的两个年份

第20行加上’2018’,‘2019’,改完如下：

# Set up voc__
for year in ['2007', '2012','2018','2019']: # 自制数据集名称是VOC2018和VOC2019
  for split in ['train', 'val', 'trainval', 'test']:
    name = 'voc_{}_{}'.format(year, split)
    __sets[name] = (lambda split=split, year=year: pascal_voc(split, year))

4. lib/datasets/pascalvoc.py的 class，改成我们的

第47行

self._classes = ('__background__',  # always index 0
                         'pedestrian','car')

出现问题及解决：

问题1. target_ratio类型转换

File "/home/zhaoxl/jiaqun/lujialin/faster-rcnn-kitti-light/lib/roi_data_layer/roibatchLoader.py", line 54, in __init__
    self.ratio_list_batch[left_idx:(right_idx+1)] = target_ratio
TypeError: can't assign a numpy.int64 to a torch.FloatTensor

• 解决方法

在lib/roi_data_layer/roibatchLoader.py
第53行，加上：

target_ratio = int(target_ratio)

问题2. 数据整理

File "/home/zhaoxl/jiaqun/lujialin/faster-rcnn.cittykitti/lib/datasets/imdb.py", line 123, in append_flipped_images
    assert (boxes[:, 2] >= boxes[:, 0]).all()
AssertionError

• 解决方法

修改lib/datasets/imdb.py，append_flipped_images()函数。
在 boxes[:, 2] = widths[i] - oldx1 - 1 的下面加入代码：

for b in range(len(boxes)):
  if boxes[b][2]< boxes[b][0]:
    boxes[b][0] = 0

问题3：loss=nan

第二个iter时，

fg/bg = 256/0

各种loss = nan

• 解决方法

查了很多教程，大家遇到的问题和解决方法基本就这几种，下面进行总结：

方法1.删掉data/cache

这个最终解决了我的问题！！

这里删除catch的意义是！把之前生成的imdb给删掉，也就是说，如果用后面的方法改了一些代码，也必须要重新删除一次cache！才能奏效！

方法2. 确定gt坐标不会为负值（方法1）

修改lib/datasets/imdb.py，append_flipped_images()函数
数据整理，在一行代码为 boxes[:, 2] = widths[i] - oldx1 - 1下加入代码：

for b in range(len(boxes)):
  if boxes[b][2]< boxes[b][0]:
    boxes[b][0] = 0

改完代码记得删cache再运行

方法3. 确定gt坐标不会为负值（方法2）

如果你自己制作了voc pascal或者coco数据集格式，那么你需要注意，看看是否有类似下面的报错

RuntimeWarning: invalid value encountered in log targets_dw = np.log(gt_widths / ex_widths)

这种报错说明数据集的数据有一些问题，多出现在没有控制好边界的情况，首先，打开lib/database/pascal_voc.py文件，找到208行，将208行至211行每一行后面的-1删除，如下所示：

x1 = float(bbox.find(‘xmin’).text) 
y1 = float(bbox.find(‘ymin’).text) 
x2 = float(bbox.find(‘xmax’).text) 
y2 = float(bbox.find(‘ymax’).text)

原因是因为我们制作的xml文件中有些框的坐标是从左上角开始的，也就是（0,0）如果再减一就会出现log（-1）的情况

改完代码记得删cache再运行

方法4. 取消数据翻转（先做2,3，不行再做4。最好不要取消）

打开./lib/model/config.py文件，找到flipp选项，将其置为False

__C.TRAIN.USE_FLIPPED = False

改完代码记得删cache再运行

方法5. 调小学习率（最后进行，最好不要进行）

其他一些有帮助的链接

解决fasterrcnn制作新数据集

https://blog.csdn.net/xzzppp/article/details/52036794
https://blog.csdn.net/ksws0292756/article/details/80702704
https://blog.csdn.net/qq_14839543/article/details/72900863
https://blog.csdn.net/flztiii/article/details/73881954

解决loss=nan的

https://www.cnblogs.com/hypnus-ly/p/9895885.html

检查输入数据和target中是否有 nan 值

    print('-------------new iter----------------')
    for step in range(iters_per_epoch):

      data = next(data_iter)

      im_data.data.resize_(data[0].size()).copy_(data[0])
      im_info.data.resize_(data[1].size()).copy_(data[1])
      gt_boxes.data.resize_(data[2].size()).copy_(data[2])
      num_boxes.data.resize_(data[3].size()).copy_(data[3])
      print(im_data.size(),gt_boxes.size())
      
      tempa = im_data.view(-1).cpu().numpy()
      tempb = gt_boxes.view(-1).cpu().numpy()
      tempa = pd.Series(tempa)
      tempb = pd.Series(tempb)
      anull = tempa.isnull().values.any()
      bnull = tempb.isnull().values.any()
      print('im_data: ',anull)
      print('gt_boxes: ',bnull)

      if anull or bnull:
          sys.exit()

loss=nan 程序停止

if isnan(loss_temp):
          sys.exit()

打印每一层的weight和gred

model改成fasterRCNN，把这段代码放在 iter的循环里。

for params in model.named_parameters():
    [name, param] = params

    if param.grad is not None:
        print(name, end='\t')
        print('weight:{}'.format(param.data.mean()), end='\t')
        print('grad:{}'.format(param.grad.data.mean()))

二、测试(test)

先改 test.py

类似trainval.py的修改模式, 这段复制过去，并着重把args.imdbval_name，改成测试或验证集的名称

elif args.dataset == "citykitti":
      args.imdb_name = "voc_2018_train+voc_2019_train"
      args.imdbval_name = "voc_2018_val+voc_2019_val"
      args.set_cfgs = ['ANCHOR_SCALES', '[8, 16, 32]', 'ANCHOR_RATIOS', '[0.5,1,2]', 'MAX_NUM_GT_BOXES', '50']

重点改 lib/datasets/voc_eval.py

• 主要参考这篇，但他有一点点说的不全: https://blog.csdn.net/forteenscoops/article/details/79738870
• 主旨就是把 voc_eval 里的关于’difficult’的都删了，因为我们自己做数据集的时候没有标注这个东西。但在删的时候，有一个地方不删，而是改一下

npos = npos + sum(~difficult)

改为

npos = npos + len®

因为改动的地方比较多，这里贴出全部的代码~

# --------------------------------------------------------
# Fast/er R-CNN
# Licensed under The MIT License [see LICENSE for details]
# Written by Bharath Hariharan
# --------------------------------------------------------
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import xml.etree.ElementTree as ET
import os
import pickle
import numpy as np

def parse_rec(filename):
  """ Parse a PASCAL VOC xml file """
  tree = ET.parse(filename)
  objects = []
  for obj in tree.findall('object'):
    obj_struct = {}
    obj_struct['name'] = obj.find('name').text
    # obj_struct['pose'] = obj.find('pose').text
    # obj_struct['truncated'] = int(obj.find('truncated').text)
    # obj_struct['difficult'] = int(obj.find('difficult').text)
    bbox = obj.find('bndbox')
    obj_struct['bbox'] = [int(bbox.find('xmin').text),
                          int(bbox.find('ymin').text),
                          int(bbox.find('xmax').text),
                          int(bbox.find('ymax').text)]
    objects.append(obj_struct)

  return objects


def voc_ap(rec, prec, use_07_metric=False):
  """ ap = voc_ap(rec, prec, [use_07_metric])
  Compute VOC AP given precision and recall.
  If use_07_metric is true, uses the
  VOC 07 11 point method (default:False).
  """
  if use_07_metric:
    # 11 point metric
    ap = 0.
    for t in np.arange(0., 1.1, 0.1):
      if np.sum(rec >= t) == 0:
        p = 0
      else:
        p = np.max(prec[rec >= t])
      ap = ap + p / 11.
  else:
    # correct AP calculation
    # first append sentinel values at the end
    mrec = np.concatenate(([0.], rec, [1.]))
    mpre = np.concatenate(([0.], prec, [0.]))

    # compute the precision envelope
    for i in range(mpre.size - 1, 0, -1):
      mpre[i - 1] = np.maximum(mpre[i - 1], mpre[i])

    # to calculate area under PR curve, look for points
    # where X axis (recall) changes value
    i = np.where(mrec[1:] != mrec[:-1])[0]

    # and sum (\Delta recall) * prec
    ap = np.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1])
  return ap


def voc_eval(detpath,
             annopath,
             imagesetfile,
             classname,
             cachedir,
             ovthresh=0.5,
             use_07_metric=False):
  """rec, prec, ap = voc_eval(detpath,
                              annopath,
                              imagesetfile,
                              classname,
                              [ovthresh],
                              [use_07_metric])

  Top level function that does the PASCAL VOC evaluation.

  detpath: Path to detections
      detpath.format(classname) should produce the detection results file.
  annopath: Path to annotations
      annopath.format(imagename) should be the xml annotations file.
  imagesetfile: Text file containing the list of images, one image per line.
  classname: Category name (duh)
  cachedir: Directory for caching the annotations
  [ovthresh]: Overlap threshold (default = 0.5)
  [use_07_metric]: Whether to use VOC07's 11 point AP computation
      (default False)
  """
  # assumes detections are in detpath.format(classname)
  # assumes annotations are in annopath.format(imagename)
  # assumes imagesetfile is a text file with each line an image name
  # cachedir caches the annotations in a pickle file

  # first load gt
  if not os.path.isdir(cachedir):
    os.mkdir(cachedir)
  cachefile = os.path.join(cachedir, '%s_annots.pkl' % imagesetfile)
  # read list of images
  with open(imagesetfile, 'r') as f:
    lines = f.readlines()
  imagenames = [x.strip() for x in lines]

  if not os.path.isfile(cachefile):
    # load annotations
    recs = {}
    for i, imagename in enumerate(imagenames):
      recs[imagename] = parse_rec(annopath.format(imagename))
      if i % 100 == 0:
        print('Reading annotation for {:d}/{:d}'.format(
          i + 1, len(imagenames)))
    # save
    print('Saving cached annotations to {:s}'.format(cachefile))
    with open(cachefile, 'wb') as f:
      pickle.dump(recs, f)
  else:
    # load
    with open(cachefile, 'rb') as f:
      try:
        recs = pickle.load(f)
      except:
        recs = pickle.load(f, encoding='bytes')

  # extract gt objects for this class
  class_recs = {}
  npos = 0 
  for imagename in imagenames:
    R = [obj for obj in recs[imagename] if obj['name'] == classname]
    bbox = np.array([x['bbox'] for x in R])
    # difficult = np.array([x['difficult'] for x in R]).astype(np.bool)
    det = [False] * len(R)
    # npos = npos + sum(~difficult)
    npos = npos + len(R)
    class_recs[imagename] = {'bbox': bbox,
                             #'difficult': difficult,
                             'det': det}

  # read dets
  detfile = detpath.format(classname)
  with open(detfile, 'r') as f:
    lines = f.readlines()

  splitlines = [x.strip().split(' ') for x in lines]
  image_ids = [x[0] for x in splitlines]
  confidence = np.array([float(x[1]) for x in splitlines])
  BB = np.array([[float(z) for z in x[2:]] for x in splitlines])

  nd = len(image_ids)
  tp = np.zeros(nd)
  fp = np.zeros(nd)

  if BB.shape[0] > 0:
    # sort by confidence
    sorted_ind = np.argsort(-confidence)
    sorted_scores = np.sort(-confidence)
    BB = BB[sorted_ind, :]
    image_ids = [image_ids[x] for x in sorted_ind]

    # go down dets and mark TPs and FPs
    for d in range(nd):
      R = class_recs[image_ids[d]]
      bb = BB[d, :].astype(float)
      ovmax = -np.inf
      BBGT = R['bbox'].astype(float)

      if BBGT.size > 0:
        # compute overlaps
        # intersection
        ixmin = np.maximum(BBGT[:, 0], bb[0])
        iymin = np.maximum(BBGT[:, 1], bb[1])
        ixmax = np.minimum(BBGT[:, 2], bb[2])
        iymax = np.minimum(BBGT[:, 3], bb[3])
        iw = np.maximum(ixmax - ixmin + 1., 0.)
        ih = np.maximum(iymax - iymin + 1., 0.)
        inters = iw * ih

        # union
        uni = ((bb[2] - bb[0] + 1.) * (bb[3] - bb[1] + 1.) +
               (BBGT[:, 2] - BBGT[:, 0] + 1.) *
               (BBGT[:, 3] - BBGT[:, 1] + 1.) - inters)

        overlaps = inters / uni
        ovmax = np.max(overlaps)
        jmax = np.argmax(overlaps)
        
#      if ovmax > ovthresh:
#        if not R['difficult'][jmax]:
#          if not R['det'][jmax]:
#            tp[d] = 1.
#            R['det'][jmax] = 1
#          else:
#            fp[d] = 1.
#      else:
#        fp[d] = 1.
        
      if ovmax > ovthresh:
        if not R['det'][jmax]:
          tp[d] = 1.
          R['det'][jmax] = 1
        else:
          fp[d] = 1.
      else:
        fp[d] = 1.

  # compute precision recall
  fp = np.cumsum(fp)
  tp = np.cumsum(tp)
  rec = tp / float(npos)
  # avoid divide by zero in case the first detection matches a difficult
  # ground truth
  prec = tp / np.maximum(tp + fp, np.finfo(np.float64).eps)
  ap = voc_ap(rec, prec, use_07_metric)

  return rec, prec, ap

test可视化结果

# 首先
arg.vis=True

# 然后test.py文件最后：
if vis:
          cv2.imwrite('result.png', im2show)
          # pdb.set_trace()
          cv2.imshow('test', im2show)
          cv2.waitKey(0)