Ubuntu16.04配置caffe-GPU环境

1 安装NVIDIA driver、CUDA Toolkit和cuDNN

安装过程请参考另外一篇文章:ubuntu16.04安装CUDA8.0+cuDNN5.1。

2 安装opencv

参考opencv官方的安装文档:Install OpenCV-Python in Ubuntu。安装opencv有两种方式:

  • Installing OpenCV-Python from Pre-built Binaries
  • Building OpenCV from source
    这两种安装方式的区别是:ubuntu的系统标准库中已经包含特定版本的预编译好的opencv安装包,虽然版本不是最新的,目前是opencv2.4.9,但是因为caffe只用到了opencv中的基本IO和图像处理函数,所以opencv2.4.9足够了;从源代码安装opencv的优点是可以安装任何版本的opencv,如果你想使用最新版本的opencv,只能从源码安装。两种安装方式对于caffe的配置都OK。

2.1 使用apt命令安装opencv

ubuntu的系统标准库中已经包含了某个版本的预编译好的opencv安装包,可直接使用apt命令安装:

sudo apt install python-opencv

安装完后在终端打开python,使用以下命令查看使用apt命令安装的opencv版本:

ys@ysubuntu:~$ python
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.9.1
>>> 

可以看到,使用apt安装的opencv版本比较老,是opencv2.4.9。

2.2 从源代码编译Opencv3.3

这部分内容参考自:OpenCV 3.3 Installation Guide on Ubuntu 16.04。

也可以参考opencv官方的安装文档:Install OpenCV-Python in Ubuntu。

还有一篇也是官方的参考文档:Installation in Linux。

先依次使用以下命令安装必要的依赖项(其中某些应该是不需要的,但都先装上吧):

sudo apt-get install -y build-essential cmake git pkg-config
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libatlas-base-dev libboost-all-dev libgflags-dev libgoogle-glog-dev liblmdb-dev
​
sudo apt-get install -y python-pip
sudo apt-get install -y libopencv-dev
​
sudo apt-get install --assume-yes pkg-config unzip ffmpeg qtbase5-dev python-dev python3-dev python-numpy python3-numpy
​
sudo apt-get install --assume-yes libopencv-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev
​
sudo apt-get install --assume-yes libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev
​
sudo apt-get install --assume-yes libv4l-dev libtbb-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev
​
sudo apt-get install --assume-yes libvorbis-dev libxvidcore-dev v4l-utils vtk6
sudo apt-get install --assume-yes liblapacke-dev libopenblas-dev libgdal-dev checkinstall

如果安装过程中某些安装包因为HASH值不匹配无法获取,就加上--fix-missing命令安装。

这里使用的是系统自带的python2.7,不另外安装anconda2了。

从这里下载python-3.3,0.zip源代码文件。

下载下来后,使用:

unzip opencv-3.3.0.zip

将其解压,为了简单,将得到的opencv-3.3.0文件夹重命名为opencv文件夹。

然后从命令行进入opencv文件,执行以下命令:

mkdir build
cd build/ 
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j8

这是一个漫长的编译过程,大概需要约半小时,如果一切顺利,最后会输出下图的结果:


Ubuntu16.04配置caffe-GPU环境_第1张图片
Screenshot from 2018-01-19 16-10-55.png

继续执行以下命令:

sudo make install
sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig
sudo apt-get update

sudo make install完成时,如果没出错应该如下图的输出:

Ubuntu16.04配置caffe-GPU环境_第2张图片
Screenshot from 2018-01-19 16-16-34.png

执行完上面步骤之后,重启电脑。
opencv3.3编译完成。
编译时可能遇到的问题
问题1:
在高版本的cuda上无法编译低版本的opencv,比如我安装的cuda9.0,此时安装opencv3.3.0就会失败,但是安装较新的opencv3.4.4就没问题。
问题2:
在安装高版本的opencv时,可能遇到opencv编译安装成功,但是在python2/python3中仍然无法导入的情况,此时只需执行:

sudo cp path/to/build/lib/cv2.so /usr/local/lib/python2.7/dist-packages/
sudo cp path/to/build/lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so /usr/local/lib/python3.5/dist-packages/

就是把build文件夹下面的两个文件拷贝到相应的python模块文件夹下面。

2.3 使用pip命令安装opencv

一种十分方便的安装opencv的方式,可指定安装特定版本

pip install opencv-python==3.3.0

如果没有该版本,错误信息中会有可用版本号提示:

Collecting opencv-python==3.3.0
  Could not find a version that satisfies the requirement opencv-python==3.3.0 (from versions: 3.1.0.0, 3.1.0.1, 3.1.0.2, 3.1.0.3, 3.1.0.4, 3.1.0.5, 3.2.0.6, 3.2.0.7, 3.2.0.8, 3.3.0.9, 3.3.0.10, 3.3.1.11, 3.4.0.12, 3.4.0.14, 3.4.1.15, 3.4.2.16, 3.4.2.17, 3.4.3.18, 3.4.4.19, 3.4.5.20, 4.0.0.21, 4.0.1.23, 4.0.1.24, 4.1.0.25)
No matching distribution found for opencv-python==3.3.0

这时就可选择其中想要安装的版本了。

3 安装必要的依赖项

官网的caffe安装教程中,编译caffe前需要安装以下依赖项:

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev libopenblas-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

4 编译caffe

首先从Github上面下载caffe源代码:

git clone https://github.com/BVLC/caffe.git

首先修改Makefile.config文件,在命令行进入caffe主目录,执行

cp Makefile.config.example Makefile.config

使用atom打开Makefile.config配置文件:

atom Makefile.config

Makefile.config文件的修改参考这篇文章:编译caffe-Makefile.config解析。
比如,我使用的GTX1080ti、CUDA8.0.61、Cudnn v5.1、系统自带的python2.7、opencv2.4.9。
修改过后的Makefile.config文件如下:

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!
​
# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1
​
# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1
​
# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
​
# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#  You should not set this flag if you will be reading LMDBs with any
#  possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1
​
# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3
​
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++
​
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr
​
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH :=
#  -gencode arch=compute_20,code=sm_20 \
#  -gencode arch=compute_20,code=sm_21 \
#  -gencode arch=compute_30,code=sm_30 \
 -gencode arch=compute_35,code=sm_35 \
 -gencode arch=compute_50,code=sm_50 \
 -gencode arch=compute_52,code=sm_52 \
 -gencode arch=compute_60,code=sm_60 \
 -gencode arch=compute_61,code=sm_61 \
 -gencode arch=compute_61,code=compute_61
​
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas
​
# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
​
# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app
​
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
 /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
 # $(ANACONDA_HOME)/include/python2.7 \
 # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include
​
# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include
​
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
​
# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
​
# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1
​
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial
​
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib
​
# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1
​
# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1
​
# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
​
# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1
​
# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
​
# enable pretty build (comment to see full commands)
Q ?= @

接着修改Makefile文件,将大约415行处的:

NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

替换为:

NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

最后修改/usr/local/cuda/include/host_config.h 文件,使用atom打开:

sudo atom /usr/local/cuda/include/host_config.h

将以下这句:

#error-- unsupported GNU version! gcc versions later than 5 are not supported!

改为:

//#error-- unsupported GNU version! gcc versions later than 5 are not supported!

也就是把它注释掉。这一步如果不注释掉的话,就要像很多网上教程那样,手动将gcc版本降级。

准备工作完毕,依次执行以下命令编译caffe:

make all -j8
make test -j8
make runtest -j8

加上-j8能大大提高编译过程,这是利用了多核处理器的并行同步执行的优点,这里的数字取决于你的电脑CPU的线程数量,一般最多和CPU的线程数量相等。

5 编译pycaffe

以上编译过程只是完成了caffe的命令行接口,下面安装caffe的python接口。
首先cd到path/to/your/caffe/python/目录,就是你clone的caffe下的python目录,然后执行:

for req in $(cat requirements.txt); do sudo -H pip install $req; done

这一句是安装几个pycaffe依赖的python模块。这些模块被安装到了/usr/local/lib/python2.7/dist-packages目录下。
然后cd回caffe主目录,执行:

make pycaffe -j8 #编译pycaffe接口

最后为了能在python中import caffe,需要在~/.bashrc中添加以下路径:

export PYTHONPATH=/home/yan/caffe/python:$PYTHONPATH

因为我把caffe下载到了home目录,具体的路径根据你下载的caffe位置而定。
这样就能顺利import caffe了:

yan@yanubuntu:~$ python
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>>

6 MNIST数据集测试

在caffe主目录下打开终端,执行脚本:

bash data/mnist/get_mnist.sh

这个脚本的功能是从网上下载MNIST数据集二进制文件,执行完后,在data/mnist/下面得到如下4个文件:

Ubuntu16.04配置caffe-GPU环境_第3张图片
Screenshot from 2018-01-20 15-15-30.png

得到数据集之后,接着需要将二进制的数据集格式转换成caffe需要的LMDB格式。
执行以下命令:

bash example/mnist/create_mnist.sh

examples/mnist/文件夹下得到两个LMDB格式的数据集文件:

Ubuntu16.04配置caffe-GPU环境_第4张图片
Screenshot from 2018-01-20 15-22-14.png

接着执行:

bash example/mnist/train_lenet.sh

如果一切顺利,系统就开始训练模型了,最终会得到大约0.99的准确率。

MNIST数据集测试完成。

你可能感兴趣的:(Ubuntu16.04配置caffe-GPU环境)