搭建机器学习软件平台

在进行以下操作的时候请使用VPN

安装必要的支持软件

sudo apt-get update
sudo apt-get upgrade

安装图像、视频和人机界面等工具包

sudo apt-get install build-essential cmake git unzip pkg-config
sudo apt-get install libjpeg-dev libtiff5-dev libjasper-dev libpng12-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install libgtk-3-dev
sudo apt-get install libhdf5-serial-dev graphviz
sudo apt-get install libopenblas-dev libatlas-base-dev gfortran
sudo apt-get install python-tk python3-tk python-imaging-tk

sudo apt-get install vim

 

sudo apt-get install build-essential 
sudo apt-get install cmake git unzip zip 
sudo apt-get install python2.7-dev python3-dev pylint

 

 

开始创建virtualenv独立环境，用来处理不同项目需要不同版本软件的问题

 

wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo python3 get-pip.py
sudo pip install virtualenv virtualenvwrapper
sudo rm -rf ~/.cache/pip get-pip.py

安装virtualenv和virtualenvwrapper之后，更新~/.bashrc文件，在文档最后加上以下几行

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

建立‘ruzw’虚拟工作环境，并在‘ruzw’中安装numpy

source ~/.bashrc
mkvirtualenv ruzw -p python3
workon ruzw
pip install numpy

编译和安装OpenCV（注意要在ruzw环境中安装）

cd ~
wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.3.0.zip
wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.3.0.zip
unzip opencv.zip
unzip opencv_contrib.zip

创建build路径进行CMake操作

cd ~/opencv-3.3.0/
mkdir build
cd build

输入以下指令：

cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D WITH_CUDA=OFF \
    -D INSTALL_PYTHON_EXAMPLES=ON \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.3.0/modules \
    -D BUILD_EXAMPLES=ON ..

现在编译OpenCV

rm -rf opencv-3.3.0 opencv.zip
rm -rf opencv_contrib-3.3.0 opencv_contrib.zip

 

make -j4

接下来就是安装OpenCV3.3

sudo make install
sudo ldconfig
cd ~

 

链接OpenCV和虚拟环境‘ruzw’

cd ~/.virtualenvs/ruzw/lib/python3.5/site-packages/
ln -s /usr/local/lib/python3.5/site-packages/cv2.cpython-35m-x86_64-linux-gnu.so cv2.so
cd ~

测试OpenCV安装和链接情况

workon ruzw
python
import cv2
cv2.__version__
'3.3.0'

安装TensorFlow（tensorflow 1.7.0 GPU、CUDA Toolkit 9.1 、cuDNN 7.1.2）

首先测试当地是否有英伟达GPU

lspci | grep -i nvidia

测试linux版本（ x86_64说明该系统是 64-bit系统，被cuda 9.1支持）

uname -m && cat /etc/*release

安装linux 核的抬头

uname -r
sudo apt-get install linux-headers-$(uname -r)

下载安装英伟达KUDA Toolkit

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo dpkg -i cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda-9.1

重启系统加载英伟达驱动

reboot

编辑~/.bashrc文件

vim ~/.bashrc

在最后一行加上

export PATH=/usr/local/cuda-9.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-9.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

 

测试驱动版本

source ~/.bashrc
sudo ldconfig
nvidia-smi

登录注册网站https://developer.nvidia.com/cudnn下载以下文件

 

cuDNN v7.1.2 Runtime Library for Ubuntu16.04 (Deb)

cuDNN v7.1.2 Developer Library for Ubuntu16.04 (Deb)

cuDNN v7.1.2 Code Samples and User Guide for Ubuntu16.04 (Deb)

在terminal中前往下载文件文件夹，进行如下操作

sudo dpkg -i libcudnn7_7.1.2.21-1+cuda9.1_amd64.deb
sudo dpkg -i libcudnn7-dev_7.1.2.21-1+cuda9.1_amd64.deb
sudo dpkg -i libcudnn7-doc_7.1.2.21-1+cuda9.1_amd64.deb

安装确认cuDNN

cp -r /usr/src/cudnn_samples_v7/ $HOME
cd  $HOME/cudnn_samples_v7/mnistCUDNN
make clean && make
./mnistCUDNN

安装libcupti (必须)

sudo apt-get install libcupti-dev
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc

安装Bazel (必须)

sudo apt-get install openjdk-8-jdk
wget https://github.com/bazelbuild/bazel/releases/download/0.11.1/bazel_0.11.1-linux-x86_64.deb
sudo dpkg -i bazel_0.11.1-linux-x86_64.deb

Python3.0版本

sudo apt-get install python3-numpy python3-dev python3-pip python3-wheel

安装TensorFlow

source ~/.bashrc
sudo ldconfig
wget https://github.com/tensorflow/tensorflow/archive/v1.7.0.zip
unzip v1.7.0.zip
cd tensorflow-1.7.0
./configure

设置Python默认地址

Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3

完成以下环境设置

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: Y
Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: Y
Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: Y
Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: Y
Do you wish to build TensorFlow with Apache Kafka Platform support? [y/N]: N
Do you wish to build TensorFlow with XLA JIT support? [y/N]: N
Do you wish to build TensorFlow with GDR support? [y/N]: N
Do you wish to build TensorFlow with VERBS support? [y/N]: N
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: N
Do you wish to build TensorFlow with CUDA support? [y/N]: Y
Please specify the CUDA SDK version you want to use, e.g. 7.0. [Leave empty to default to CUDA 9.0]: 9.1
Please specify the location where CUDA 9.1 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/local/cuda
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7.0]: 7.1.2
Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/lib/x86_64-linux-gnu
Do you wish to build TensorFlow with TensorRT support? [y/N]: N

Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 5.0] 5.0
Do you want to use clang as CUDA compiler? [y/N]: N
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: /usr/bin/gcc
Do you wish to build TensorFlow with MPI support? [y/N]: N
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: -march=native
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:N

用Bazel来安装TensorFlow（在tensorflow-1.7.0路径下，可能持续1~2小时）

sudo ln -s /usr/local/cuda/include/crt/math_functions.hpp /usr/local/cuda/include/math_functions.hpp
bazel build --config=opt --config=cuda --incompatible_load_argument_is_label=false //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package tensorflow_pkg

激活虚拟环境（python3）

cd tensorflow_pkg
pip3 install tensorflow*.whl

测试TensporFlow安装结果

python

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))

如果出现以下结果，TensorFlow就安装成功

b'Hello, TensorFlow'

安装Keras

首先确保是在虚拟环境下（ruzw）

安装必要的基础软件

pip install scipy matplotlib pillow
pip install imutils h5py requests progressbar2
pip install scikit-learn scikit-image

安装Keras

pip install keras

熟悉~/.keras/keras.json文件，确保image_data_format设置为channels_last、backend设置为tensorflow

{
    "image_data_format": "channels_last",
    "backend": "tensorflow",
    "epsilon": 1e-07,
    "floatx": "float32"
}

安装mxnet

复制mxnet中0.11.0分枝

cd ~
git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet --branch 0.11.0
cd mxnet
make -j4 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1

链接到我们的ruzw虚拟环境中

cd ~/.virtualenvs/ruzw/lib/python3.5/site-packages/
ln -s ~/mxnet/python/mxnet mxnet
cd ~

测试mxnet

python
import mxnet

注意mxnet文档不可以删除！