Ubuntu使用Anaconda部署Caffe2及Pytorch
1 预编译安装Caffe2
Anaconda安装教程：Ubuntu部署Anaconda环境及conda使用
1.0 CPU版
conda install pytorch-nightly-cpu -c pytorch
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.12
latest version: 4.6.8
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/xdq/anaconda3/envs/py37cpu
added / updated specs:
- pytorch-nightly-cpu
The following packages will be downloaded:
package | build
---------------------------|-----------------
wheel-0.33.1 | py37_0 39 KB
setuptools-40.8.0 | py37_0 643 KB
libedit-3.1.20181209 | hc058e9b_0 188 KB
numpy-base-1.16.2 | py37hde5b4d6_0 4.3 MB
mkl_fft-1.0.10 | py37ha843d7b_0 169 KB
ninja-1.8.2 | py37h6bb024c_1 1.3 MB
python-3.7.2 | h0371630_0 36.4 MB
sqlite-3.27.2 | h7b6447c_0 1.9 MB
numpy-1.16.2 | py37h7e9f1db_0 49 KB
ca-certificates-2019.1.23 | 0 126 KB
openssl-1.1.1b | h7b6447c_1 4.0 MB
cffi-1.12.2 | py37h2e261b9_1 222 KB
pytorch-nightly-cpu-1.0.0.dev20190317| py3.7_cpu_0 49.4 MB pytorch
pip-19.0.3 | py37_0 1.8 MB
certifi-2019.3.9 | py37_0 155 KB
------------------------------------------------------------
Total: 100.7 MB
The following NEW packages will be INSTALLED:
blas: 1.0-mkl
ca-certificates: 2019.1.23-0
certifi: 2019.3.9-py37_0
cffi: 1.12.2-py37h2e261b9_1
intel-openmp: 2019.1-144
libedit: 3.1.20181209-hc058e9b_0
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 8.2.0-hdf63c60_1
libgfortran-ng: 7.3.0-hdf63c60_0
libstdcxx-ng: 8.2.0-hdf63c60_1
mkl: 2019.1-144
mkl_fft: 1.0.10-py37ha843d7b_0
mkl_random: 1.0.2-py37hd81dba3_0
ncurses: 6.1-he6710b0_1
ninja: 1.8.2-py37h6bb024c_1
numpy: 1.16.2-py37h7e9f1db_0
numpy-base: 1.16.2-py37hde5b4d6_0
openssl: 1.1.1b-h7b6447c_1
pip: 19.0.3-py37_0
pycparser: 2.19-py37_0
python: 3.7.2-h0371630_0
pytorch-nightly-cpu: 1.0.0.dev20190317-py3.7_cpu_0 pytorch
readline: 7.0-h7b6447c_5
setuptools: 40.8.0-py37_0
sqlite: 3.27.2-h7b6447c_0
tk: 8.6.8-hbc83047_0
wheel: 0.33.1-py37_0
xz: 5.2.4-h14c3975_4
zlib: 1.2.11-h7b6447c_3
Proceed ([y]/n)?
输入:y
- Result
Downloading and Extracting Packages
wheel-0.33.1 | 39 KB | ##################################### | 100%
setuptools-40.8.0 | 643 KB | ##################################### | 100%
libedit-3.1.20181209 | 188 KB | ##################################### | 100%
numpy-base-1.16.2 | 4.3 MB | ##################################### | 100%
mkl_fft-1.0.10 | 169 KB | ##################################### | 100%
ninja-1.8.2 | 1.3 MB | ##################################### | 100%
python-3.7.2 | 36.4 MB | ##################################### | 100%
sqlite-3.27.2 | 1.9 MB | ##################################### | 100%
numpy-1.16.2 | 49 KB | ##################################### | 100%
ca-certificates-2019 | 126 KB | ##################################### | 100%
openssl-1.1.1b | 4.0 MB | ##################################### | 100%
cffi-1.12.2 | 222 KB | ##################################### | 100%
pytorch-nightly-cpu- | 49.4 MB | ##################################### | 100%
pip-19.0.3 | 1.8 MB | ##################################### | 100%
certifi-2019.3.9 | 155 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
1.2 GPU版
【CUDA9 cuDNN 7】
conda install pytorch-nightly -c pytorch
【CUDA8 cuDNN 7】
conda install pytorch-nightly cuda80 -c pytorch
由以上安装命令可看出,Pytorch和Caffe2已"融合".
1.3 安装相关模块
jupyter
conda install -n env_name jupyter
protobuf
conda install -n env_name protobuf
future
conda install --name env_name future
1.4 启动jupyter
# 启动conda环境
conda activate env_name
(env_name)$ jupyter notebook
1.5 测试Caffe2
from caffe2.python import core, workspace
from caffe2.proto import caffe2_pb2
2 安装Pytorch
2.1 conda安装
CPU版
conda install pytorch-cpu torchvision-cpu -c pytorch
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.5.12
latest version: 4.6.8
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/xdq/anaconda3/envs/4pytorch
added / updated specs:
- pytorch-cpu
- torchvision-cpu
The following packages will be downloaded:
package | build
---------------------------|-----------------
pillow-5.4.1 | py37h34e0f95_0 616 KB
libtiff-4.0.10 | h2733197_2 604 KB
torchvision-cpu-0.2.2 | py_3 44 KB pytorch
pytorch-cpu-1.0.1 | py3.7_cpu_2 26.8 MB pytorch
libpng-1.6.36 | hbc83047_0 346 KB
------------------------------------------------------------
Total: 28.4 MB
The following NEW packages will be INSTALLED:
blas: 1.0-mkl
ca-certificates: 2019.1.23-0
certifi: 2019.3.9-py37_0
cffi: 1.12.2-py37h2e261b9_1
freetype: 2.9.1-h8a8886c_1
intel-openmp: 2019.1-144
jpeg: 9b-h024ee3a_2
libedit: 3.1.20181209-hc058e9b_0
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 8.2.0-hdf63c60_1
libgfortran-ng: 7.3.0-hdf63c60_0
libpng: 1.6.36-hbc83047_0
libstdcxx-ng: 8.2.0-hdf63c60_1
libtiff: 4.0.10-h2733197_2
mkl: 2019.1-144
mkl_fft: 1.0.10-py37ha843d7b_0
mkl_random: 1.0.2-py37hd81dba3_0
ncurses: 6.1-he6710b0_1
ninja: 1.8.2-py37h6bb024c_1
numpy: 1.16.2-py37h7e9f1db_0
numpy-base: 1.16.2-py37hde5b4d6_0
olefile: 0.46-py37_0
openssl: 1.1.1b-h7b6447c_1
pillow: 5.4.1-py37h34e0f95_0
pip: 19.0.3-py37_0
pycparser: 2.19-py37_0
python: 3.7.2-h0371630_0
pytorch-cpu: 1.0.1-py3.7_cpu_2 pytorch
readline: 7.0-h7b6447c_5
setuptools: 40.8.0-py37_0
six: 1.12.0-py37_0
sqlite: 3.27.2-h7b6447c_0
tk: 8.6.8-hbc83047_0
torchvision-cpu: 0.2.2-py_3 pytorch
wheel: 0.33.1-py37_0
xz: 5.2.4-h14c3975_4
zlib: 1.2.11-h7b6447c_3
zstd: 1.3.7-h0b5b093_0
Proceed ([y]/n)?
输入:y
Downloading and Extracting Packages
pillow-5.4.1 | 616 KB | ##################################### | 100%
libtiff-4.0.10 | 604 KB | ##################################### | 100%
torchvision-cpu-0.2. | 44 KB | ##################################### | 100%
pytorch-cpu-1.0.1 | 26.8 MB | ##################################### | 100%
libpng-1.6.36 | 346 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
GPU版
【CUDA10.0】
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
【CUDA9.0】
conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
【CUDA8.x】
conda install pytorch torchvision cudatoolkit=8.0 -c pytorch
2.2 pip安装
CPU版
【Python 2.7】
pip install https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp27-cp27mu-linux_x86_64.whl
pip install torchvision
# if the above command does not work, then you have python 2.7 UCS2, use this command
pip install https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp27-cp27m-linux_x86_64.whl
【Python 3.5】
pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl
pip3 install torchvision
【Python 3.6】
pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl
pip3 install torchvision
【Python 3.7】
pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp37-cp37m-linux_x86_64.whl
pip3 install torchvision
GPU版
CUDA10
【CUDA10.0 Python 2.7】
pip install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp27-cp27mu-linux_x86_64.whl
pip install torchvision
# if the above command does not work, then you have python 2.7 UCS2, use this command
pip install https://download.pytorch.org/whl/cu80/torch-1.0.1.post2-cp27-cp27m-linux_x86_64.whl
【CUDA10.0 Python 3.5】
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl
pip3 install torchvision
【CUDA10.0 Python 3.6】
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl
pip3 install torchvision
【CUDA10.0 Python 3.7】
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp37-cp37m-linux_x86_64.whl
pip3 install torchvision
CUDA9
【CUDA9.0 Python2.x】
pip install pytorch torchvision
【CUDA9.0 Python3.x】
pip3 install pytorch torchvision
CUDA8
【CUDA8.x Python 2.7】
# Python 2.7
pip install https://download.pytorch.org/whl/cu80/torch-1.0.1.post2-cp27-cp27mu-linux_x86_64.whl
pip install torchvision
# if the above command does not work, then you have python 2.7 UCS2, use this command
pip install https://download.pytorch.org/whl/cu80/torch-1.0.1.post2-cp27-cp27m-linux_x86_64.whl
【CUDA8.x Python 3.5】
# Python 3.5
pip3 install https://download.pytorch.org/whl/cu80/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl
pip3 install torchvision
【CUDA8.x Python 3.6】
# Python 3.6
pip3 install https://download.pytorch.org/whl/cu80/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl
pip3 install torchvision
【CUDA8.x Python 3.7】
# Python 3.7
pip3 install https://download.pytorch.org/whl/cu80/torch-1.0.1.post2-cp37-cp37m-linux_x86_64.whl
pip3 install torchvision
2.3 安装jupyter
conda install -n env_name jupyter
2.4 启动jupyter
# 启动conda环境
conda activate env_name
(env_name)$jupyter notebook
2.4 测试pytorch
import torch
x = torch.rand(5, 3)
print("pytorch value: {}".format(x))
torch value: tensor([[0.3007, 0.3312, 0.4018],
[0.8476, 0.7637, 0.4661],
[0.3087, 0.6952, 0.9150],
[0.7295, 0.5297, 0.7579],
[0.4961, 0.4061, 0.0495]])
3 源码安装
【(1) requirements.txt】
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential \
git \
libgoogle-glog-dev \
libgtest-dev \
libiomp-dev \
libleveldb-dev \
liblmdb-dev \
libopencv-dev \
libopenmpi-dev \
libsnappy-dev \
libprotobuf-dev \
openmpi-bin \
openmpi-doc \
protobuf-compiler \
python-dev \
python-pip
pip install --user \
future \
numpy \
protobuf \
typing \
hypothesis
# for Ubuntu 14.04
sudo apt-get install -y --no-install-recommends \
libgflags2 \
cmake3
# for Ubuntu 16.04
sudo apt-get install -y --no-install-recommends \
libgflags-dev \
cmake
【(2) conda环境激活】
conda activate 4pytorch
【(2) 编译安装】
这些操作在conda的环境中操作。
# 进入文件夹如
cd xinPrj
# 下载项目源码
git clone https://github.com/pytorch/pytorch.git && cd pytorch
# 更新子模块
git submodule update --init --recursive
# 开启使用LMDB数据并编译安装caffe2
# USE_LMDB=ON是关键
USE_LMDB=ON python setup.py install
【(3) 安装完毕】
running install_scripts
# conda虚拟环境为：/home/xdq/anaconda3/envs/4pytorch/
Installing convert-caffe2-to-onnx script to /home/xdq/anaconda3/envs/4pytorch/bin
Installing convert-onnx-to-caffe2 script to /home/xdq/anaconda3/envs/4pytorch/bin
4 问题
4.1 import torch
【Problem】
import-im6.q16: not authorized `torch' @ error/constitute.c/WriteImage/1037
【Reason】
未在pytorch环境中启用torch
【Solve】
在pytorch环境中启用torch.
4.2 no module named ‘google’
【Problem】
File "/home/xdq/anaconda3/envs/4caffe2/lib/python3.7/site-packages/caffe2/proto/caffe2_pb2.py", line 6, in
from google.protobuf.internal import enum_type_wrapper
ModuleNotFoundError: No module named 'google'
【Reason】
缺少protobuf
【Solve】
安装protobuf
conda install -c https://conda.anaconda.org/anaconda protobuf
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/xdq/anaconda3/envs/4caffe2
added / updated specs:
- protobuf
The following packages will be downloaded:
package | build
---------------------------|-----------------
libprotobuf-3.6.1 | hd408876_0 4.1 MB anaconda
six-1.12.0 | py37_0 22 KB anaconda
protobuf-3.6.1 | py37he6710b0_0 615 KB anaconda
------------------------------------------------------------
Total: 4.7 MB
The following NEW packages will be INSTALLED:
libprotobuf: 3.6.1-hd408876_0 anaconda
protobuf: 3.6.1-py37he6710b0_0 anaconda
six: 1.12.0-py37_0 anaconda
Proceed ([y]/n)?
输入:y
4.3 no module named ‘past’
【Problem】
File "/home/xdq/anaconda3/envs/4caffe2/lib/python3.7/site-packages/caffe2/python/core.py", line 9, in
from past.builtins import basestring
ModuleNotFoundError: No module named 'past'
【Reason】
缺少future
【Solve】
安装future
conda install env_name future
5 总结
(1) Caffe2现已和Pytorch"融合",安装Caffe2和Pytorch中的任意一个,即可拥有两者的环境.
(2) 为在各自的conda环境中使用jupyter,需要在各自环境中先安装jupyter,然后启动想用的conda环境,在conda环境中启动jupyter.
(3) Caffe2和Pytorch均支持CPU和GPU两个版本,注意cuDNN和CUDA的版本,版本匹配可参考下面博客.
支持:GPU之CUDA&cuDNN&Tensorflow版本匹配
(5) 解决读取caffe2数据数据问题，使用源码安装方式，同时开启数据读取配置：USE_LMDB=ON;
[参考文献]
[1]https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=prebuilt
[2]https://pytorch.org/get-started/locally/
[3]https://www.cnblogs.com/huolifeng/p/6412183.html
[4]https://blog.csdn.net/u011534057/article/details/51557177
[5]https://github.com/pytorch/pytorch/issues/10119
[6]https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile