Domain adaptive faster-RCNN github 复现笔记
笔记目录
- 1.项目代码链接
- *2.环境配置（很重要）*
-
- 2.1.概述
- 2.2.细节
-
- 2.2.1.更新Ubuntu18.04源
- 2.2.2.gcc/g++降级
- 2.2.3.下载安装查看的CUDA9.0，pytorch0.4.0，cudnn7.6.5
-
- 2.2.3.1 CUDA
- 2.2.3.2 cudnn
- 2.2.2.3 Pytorch
- 3. DEBUG
-
- 3.1.安装所需python模块
- 3.2.下载数据集，预训练权重文件
- 3.3.配置faster-rcnn.pytorch
- 3.4.常见报错及其解决方法
1.项目代码链接
链接: da-faster-rcnn-Pytorch.
2.环境配置（很重要）
2.1.概述
DA-fasterrcnn的复现的主要工作在环境的配置，目前大部分的和Domain adaptive detection相关的paper和github大部分都是基于pytorch0.4.0（主要）和pytorch0.4.1两个老版本进行复现的，因此，对于显卡，CUDA，cudnn的对应配置不能使用太高或者太新的版本。
原始配置：RTX3090，CUDA11.1，cudnn8.0.5，python3.7（未复现成功）
租用服务器配置：1080ti，CUDA9.0，cudnn7.6.5，python3.6（复现成功）
另外，这里还要注意不同的，要满足CUDA版本和对应显卡的驱动的版本之间的关系，可以查看下图：
参考: CUDA版本与显卡驱动版本的对应关系.
2.2.细节
2.2.1.更新Ubuntu18.04源
①备份原始文件
cp /etc/apt/sources.list /etc/apt/sources.list.bak
②添加源
sudo gedit /etc/apt/sources.list
添加阿里源
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
添加清华源
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
添加中科大源
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
添加163源
deb http://mirrors.163.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ bionic-backports main restricted universe multiverse
2.2.2.gcc/g++降级
ubuntu18.04操作系统自带7版本gcc/g++，cuda9.0编译器要求6及其以下版本，因此需要将降级gcc/g++版本。
①查看现有gcc/g++版本
cd /usr/bin
ll gcc g++
图来源：Ubuntu18.04安装nVIDIA显卡驱动+cuda9.0+cuDNN7
②安装6版本gcc/g++
sudo apt install gcc-6 g++-6
③降级gcc/g++
cd /usr/bin
#删除软连接
sudo rm gcc
#重新建立指向gcc-6的gcc软链接
sudo ln -s gcc-6 gcc
sudo rm g++
sudo ln -s g++-6 g++
④查看gcc/g++
ll gcc g++
2.2.3.下载安装查看的CUDA9.0，pytorch0.4.0，cudnn7.6.5
2.2.3.1 CUDA
①安装CUDA
#安装CUDA-9.0
sh cuda_9.0.176_384.81_linux.run --override
②安装补丁包
sh cuda_9.0.176.1_linux.run
sh cuda_9.0.176.2_linux.run
sh cuda_9.0.176.3_linux.run
sh cuda_9.0.176.4_linux.run
③配置环境变量
gedit ~/.bashrc
④在~/.bashrc末尾添加以下内容
#cuda9.0
expert PATH=/usr/local/cuda-9.0/bin:$PATH
expert LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:$LD_LIBRARY_PATH
⑤保存后关闭文件，生效配置文件
source ~/.bashrc
参考：Ubuntu18.04安装nVIDIA显卡驱动+cuda9.0+cuDNN7
⑥查看CUDA
nvcc -V
⑦多版本CUDA切换
首先将~/.bashrc下与cuda相关的路径均修改为/usr/local/cuda，而不使用/usr/local/cuda-9.0或者/usr/local/cuda-10.2
#删除之前在安装cuda时建立的软连接
sudo rm -rf /usr/local/cuda/
#建立新的软连接
sudo ln -s /usr/local/cuda-9.0/ /usr/local/cuda/
#查看当前版本CUDA
nvcc -V
参考：安装多版本 cuda ，多版本之间切换.
2.2.3.2 cudnn
①下载cudnn
cudnn的版本必须要和pytorch版本，pytorch对应的cudnn版本在安装pytorch时可以看到：
可以到 cudnn 下载。
②安装cudnn
#解压cudnn
tar -xzvf cudnn-9.0-linux-x64-v7.6.5.32.tgz
#复制cudnn到cuda
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
③查看cudnn
cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
④卸载cudnn
sudo rm -rf /usr/local/cuda/include/cudnn.h
sudo rm -rf /usr/local/cuda/lib64/libcudnn*
2.2.2.3 Pytorch
①安装pytorch
conda install pytorch=0.4.0
#自动安装对应版本的torchvison
codna install torchvision
②查看pytorch
conda list
3. DEBUG
3.1.安装所需python模块
pip install -r requirements.txt
3.2.下载数据集，预训练权重文件
数据集：cityscapes.
预训练权重： VGG16.链接: ResNet101.
da-faster-rcnn-pytorch
--data
--cityscapes
--pretrained-model
--vgg16_caffe.pth
--resnet101_caffe.pth
3.3.配置faster-rcnn.pytorch
cd lib
sh make.sh
3.4.常见报错及其解决方法
报错1：
TypeError:load() missing 1 required positional argument:'Loader'
解决方法：
#降级pyyaml到5.4.1
pip install pyyaml==5.4.1
报错2：
PermissionError:[Errno 13] Permission denied: ' /data'
解决方法：
进入da_trainval_net.py
line 69:
原始代码：
parser.add_argument('--save_dir', dest='save_dir',
help='directory to save models', default="/data/ztc/adaptation/Experiment/da_model",type=str)
修改代码：
parser.add_argument('--save_dir', dest='save_dir',
help='directory to save models', default="./data",type=str)
报错3：
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp:74
报错4：
NameError: name 'nms_gpu' is not defined
解决方法:
#为CUDA,cudnn指定GPU
os.environ['CDUA_DEVICE_ORDER'] = 'PCI_BUS_ID' #按照PCI_BUS_ID顺序从0开始排列GPU设备
os.environ['CUDA_VISIBLE_DEVICES'] = '0' #设置当前使用的GPU设备仅为0号设备 设备名称为'/gpu:0'
注意：由于python的编译顺序是从上到下，因此，为了保证程序能顺利的利用GPU进行训练，应将指定GPU的模块放置在torch等需要使用cuda模块的前面。
报错4：
FileNotFoundError: [Errno 2] No such file or directory: '/data/ztc/detectionModel/vgg16_caffe.pth'
解决方法（resnet101_caffe.pth同理）：
cd da-faster-rcnn-Pytorch/lib/model/da_faster_rcnn
进入vgg16.py
line 21:
原始代码：
self.model_path = '/data/ztc/detectionModel/vgg16_caffe.pth'
修改代码：
self.model_path = '/data/pretrained_model/detectionModel/vgg16_caffe.pth'#修改预训练权重路径
