1 安装NVIDIA driver、CUDA Toolkit和cuDNN
安装过程请参考另外一篇文章:ubuntu16.04安装CUDA8.0+cuDNN5.1。
2 安装opencv
参考opencv官方的安装文档:Install OpenCV-Python in Ubuntu。安装opencv有两种方式:
- Installing OpenCV-Python from Pre-built Binaries
- Building OpenCV from source
这两种安装方式的区别是:ubuntu的系统标准库中已经包含特定版本的预编译好的opencv安装包,虽然版本不是最新的,目前是opencv2.4.9,但是因为caffe只用到了opencv中的基本IO和图像处理函数,所以opencv2.4.9足够了;从源代码安装opencv的优点是可以安装任何版本的opencv,如果你想使用最新版本的opencv,只能从源码安装。两种安装方式对于caffe的配置都OK。
2.1 使用apt命令安装opencv
ubuntu的系统标准库中已经包含了某个版本的预编译好的opencv安装包,可直接使用apt命令安装:
sudo apt install python-opencv
安装完后在终端打开python,使用以下命令查看使用apt命令安装的opencv版本:
ys@ysubuntu:~$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> print cv2.__version__
2.4.9.1
>>>
可以看到,使用apt安装的opencv版本比较老,是opencv2.4.9。
2.2 从源代码编译Opencv3.3
这部分内容参考自:OpenCV 3.3 Installation Guide on Ubuntu 16.04。
也可以参考opencv官方的安装文档:Install OpenCV-Python in Ubuntu。
还有一篇也是官方的参考文档:Installation in Linux。
先依次使用以下命令安装必要的依赖项(其中某些应该是不需要的,但都先装上吧):
sudo apt-get install -y build-essential cmake git pkg-config
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libatlas-base-dev libboost-all-dev libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install -y python-pip
sudo apt-get install -y libopencv-dev
sudo apt-get install --assume-yes pkg-config unzip ffmpeg qtbase5-dev python-dev python3-dev python-numpy python3-numpy
sudo apt-get install --assume-yes libopencv-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev
sudo apt-get install --assume-yes libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev
sudo apt-get install --assume-yes libv4l-dev libtbb-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev
sudo apt-get install --assume-yes libvorbis-dev libxvidcore-dev v4l-utils vtk6
sudo apt-get install --assume-yes liblapacke-dev libopenblas-dev libgdal-dev checkinstall
如果安装过程中某些安装包因为HASH值不匹配无法获取,就加上--fix-missing
命令安装。
这里使用的是系统自带的python2.7,不另外安装anconda2了。
从这里下载python-3.3,0.zip源代码文件。
下载下来后,使用:
unzip opencv-3.3.0.zip
将其解压,为了简单,将得到的opencv-3.3.0文件夹重命名为opencv文件夹。
然后从命令行进入opencv文件,执行以下命令:
mkdir build
cd build/
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j8
这是一个漫长的编译过程,大概需要约半小时,如果一切顺利,最后会输出下图的结果:
继续执行以下命令:
sudo make install
sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig
sudo apt-get update
sudo make install完成时,如果没出错应该如下图的输出:
执行完上面步骤之后,重启电脑。
opencv3.3编译完成。
编译时可能遇到的问题:
问题1:
在高版本的cuda上无法编译低版本的opencv,比如我安装的cuda9.0,此时安装opencv3.3.0就会失败,但是安装较新的opencv3.4.4就没问题。
问题2:
在安装高版本的opencv时,可能遇到opencv编译安装成功,但是在python2/python3中仍然无法导入的情况,此时只需执行:
sudo cp path/to/build/lib/cv2.so /usr/local/lib/python2.7/dist-packages/
sudo cp path/to/build/lib/python3/cv2.cpython-35m-x86_64-linux-gnu.so /usr/local/lib/python3.5/dist-packages/
就是把build文件夹下面的两个文件拷贝到相应的python模块文件夹下面。
2.3 使用pip命令安装opencv
一种十分方便的安装opencv的方式,可指定安装特定版本
pip install opencv-python==3.3.0
如果没有该版本,错误信息中会有可用版本号提示:
Collecting opencv-python==3.3.0
Could not find a version that satisfies the requirement opencv-python==3.3.0 (from versions: 3.1.0.0, 3.1.0.1, 3.1.0.2, 3.1.0.3, 3.1.0.4, 3.1.0.5, 3.2.0.6, 3.2.0.7, 3.2.0.8, 3.3.0.9, 3.3.0.10, 3.3.1.11, 3.4.0.12, 3.4.0.14, 3.4.1.15, 3.4.2.16, 3.4.2.17, 3.4.3.18, 3.4.4.19, 3.4.5.20, 4.0.0.21, 4.0.1.23, 4.0.1.24, 4.1.0.25)
No matching distribution found for opencv-python==3.3.0
这时就可选择其中想要安装的版本了。
3 安装必要的依赖项
官网的caffe安装教程中,编译caffe前需要安装以下依赖项:
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev libopenblas-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
4 编译caffe
首先从Github上面下载caffe源代码:
git clone https://github.com/BVLC/caffe.git
首先修改Makefile.config
文件,在命令行进入caffe主目录,执行
cp Makefile.config.example Makefile.config
使用atom打开Makefile.config
配置文件:
atom Makefile.config
Makefile.config
文件的修改参考这篇文章:编译caffe-Makefile.config解析。
比如,我使用的GTX1080ti、CUDA8.0.61、Cudnn v5.1、系统自带的python2.7、opencv2.4.9。
修改过后的Makefile.config
文件如下:
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!
# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1
# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1
# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
# You should not set this flag if you will be reading LMDBs with any
# possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1
# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH :=
# -gencode arch=compute_20,code=sm_20 \
# -gencode arch=compute_20,code=sm_21 \
# -gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_60,code=sm_60 \
-gencode arch=compute_61,code=sm_61 \
-gencode arch=compute_61,code=compute_61
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas
# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python2.7 \
# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include
# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
# /usr/lib/python3.5/dist-packages/numpy/core/include
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib
# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1
# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1
# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1
# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
# enable pretty build (comment to see full commands)
Q ?= @
接着修改Makefile文件,将大约415行处的:
NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
替换为:
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
最后修改/usr/local/cuda/include/host_config.h
文件,使用atom打开:
sudo atom /usr/local/cuda/include/host_config.h
将以下这句:
#error-- unsupported GNU version! gcc versions later than 5 are not supported!
改为:
//#error-- unsupported GNU version! gcc versions later than 5 are not supported!
也就是把它注释掉。这一步如果不注释掉的话,就要像很多网上教程那样,手动将gcc版本降级。
准备工作完毕,依次执行以下命令编译caffe:
make all -j8
make test -j8
make runtest -j8
加上-j8
能大大提高编译过程,这是利用了多核处理器的并行同步执行的优点,这里的数字取决于你的电脑CPU的线程数量,一般最多和CPU的线程数量相等。
5 编译pycaffe
以上编译过程只是完成了caffe的命令行接口,下面安装caffe的python接口。
首先cd到path/to/your/caffe/python/
目录,就是你clone的caffe下的python目录,然后执行:
for req in $(cat requirements.txt); do sudo -H pip install $req; done
这一句是安装几个pycaffe依赖的python模块。这些模块被安装到了/usr/local/lib/python2.7/dist-packages
目录下。
然后cd回caffe主目录,执行:
make pycaffe -j8 #编译pycaffe接口
最后为了能在python中import caffe,需要在~/.bashrc中添加以下路径:
export PYTHONPATH=/home/yan/caffe/python:$PYTHONPATH
因为我把caffe下载到了home目录,具体的路径根据你下载的caffe位置而定。
这样就能顺利import caffe了:
yan@yanubuntu:~$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>>
6 MNIST数据集测试
在caffe主目录下打开终端,执行脚本:
bash data/mnist/get_mnist.sh
这个脚本的功能是从网上下载MNIST数据集二进制文件,执行完后,在data/mnist/
下面得到如下4个文件:
得到数据集之后,接着需要将二进制的数据集格式转换成caffe需要的LMDB格式。
执行以下命令:
bash example/mnist/create_mnist.sh
在examples/mnist/
文件夹下得到两个LMDB格式的数据集文件:
接着执行:
bash example/mnist/train_lenet.sh
如果一切顺利,系统就开始训练模型了,最终会得到大约0.99的准确率。
MNIST数据集测试完成。