caffe2:A New Lightweight, Modular, and Scalable Deep Learning Framework
caffe是一个清晰,可读性高,快速的深度学习框架。作者是贾扬清,加州大学伯克利的ph.D,现就职于Facebook。caffe的官网是http://caffe.berkeleyvision.org/。
目前,贾扬清和他在Facebook的团队正在开发新一代框架Caffe2。今年4月18日,Facebook开源了Caffe2。Caffe2与Caffe的区别是什么?Caffe2更注重模块化,在移动端、大规模部署上表现卓越。如同TensorFlow,Caffe2使用C++Eigen库,支持ARM架构。
用一个实用脚本,Caffe上的模型可轻易地被转变到Caffe2上。Caffe设计的选择使得它处理视觉类型的难题时很完美。Caffe2延续了它对视觉类问题的支持,且增加了对自然语言处理、手写识别、时序预测有帮助的RNN和LSTM支持。
caffe2宣称 CODE ONCE, RUN ANYWHERE
注: 如今caffe2的安装将变得极为简单,通过Ubuntu的包管理器安装即可,可以直接下载 Pre-Built Binaries
,安装即可,这里不作说明,这里展示的是源码安装过程。
参考官网说明,选择平台(Platform)和安装类型(Install type),可以得到具体安装说明,本文选择:
Platform
: UbuntuInstall type
: Build From Source注:如果使用非Python原生环境,如使用anaconda,请参考:anaconda install
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
build-essential \
cmake \
git \
libgoogle-glog-dev \
libgtest-dev \
libiomp-dev \
libleveldb-dev \
liblmdb-dev \
libopencv-dev \
libopenmpi-dev \
libsnappy-dev \
libprotobuf-dev \
openmpi-bin \
openmpi-doc \
protobuf-compiler \
python-dev \
python-pip
# for Ubuntu 14.04
sudo apt-get install -y --no-install-recommends libgflags2
# for Ubuntu 16.04
sudo apt-get install -y --no-install-recommends libgflags-dev
sudo pip install \
future \
numpy \
protobuf
从 这里 (或终端输入命令 git clone --recursive https://github.com/caffe2/caffe2.git && cd caffe2
)下载 caffe2
源码并解压进入 caffe2
文件夹。
caffe需要自己配置,对于caffe2,使用 cmake
配置,可 参考下面的构建安装部分。
终端输入 sudo gedit ~/.bashrc
打开 “.bashrc” 文件,在文件末尾加入如下代码并保存:
#caffe2
#echo $PYTHONPATH
export PYTHONPATH=/usr/local:$PYTHONPATH
export PYTHONPATH=$PYTHONPATH:yourcaffe2rootdir/caffe2/build
#echo $LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
然后重新打开一个终端,或者输入 source ~/.bashrc
加载新的环境变量。
进入caffe2的根目录,终端执行如下命令,不报错误,代表安装成功!( -j$(nproc)
代表使用最大的线程编译,当然也可以手动指定,如 make -j4
)
# This will build Caffe2 in an isolated directory so that Caffe2 source is
# unaffected
mkdir build && cd build
# This configures the build and finds which libraries it will include in the
# Caffe2 installation. The output of this command is very helpful in debugging
cmake ..
# This actually builds and installs Caffe2 from makefiles generated from the
# above configuration step
sudo make install -j$(nproc)
执行完 cmake ..
终端输出:
$ echo $PYTHONPATH
/usr/local:yourcaffe2rootdir/caffe2/build
$ mkdir build && cd build
$ cmake ..
-- The CXX compiler identification is GNU 5.4.0
-- The C compiler identification is GNU 5.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test CAFFE2_LONG_IS_INT32_OR_64
-- Performing Test CAFFE2_LONG_IS_INT32_OR_64 - Success
-- Does not need to define long separately.
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE - Success
-- NUMA is available
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Success
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extention. Will build perfkernels.
-- Build type not set - defaulting to Release
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2: Found protobuf with old-style protobuf targets.
-- Caffe2 protobuf include directory:
-- Found Git: /usr/bin/git (found version "2.7.4")
-- The BLAS backend of choice:Eigen
-- Could NOT find NNPACK (missing: NNPACK_INCLUDE_DIR NNPACK_LIBRARY PTHREADPOOL_LIBRARY CPUINFO_LIBRARY)
-- Brace yourself, we are building NNPACK
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Found PythonInterp: /usr/bin/python (found version "2.7.12")
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Caffe2: Cannot find gflags automatically. Using legacy find.
-- Found gflags: /usr/include
-- Caffe2: Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
-- Caffe2: Cannot find glog automatically. Using legacy find.
-- Found glog: /usr/include
-- Caffe2: Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
-- git Version: v0.0.0
-- Version: 0.0.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT
-- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Found LMDB: /usr/include
-- Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
-- Found LevelDB: /usr/include
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
-- Found Snappy: /usr/include
-- Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so)
-- Found Numa: /usr/include
-- Found Numa (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnuma.so)
-- OpenCV found (/usr/share/OpenCV)
CMake Warning at cmake/Dependencies.cmake:270 (find_package):
By not providing "FindEigen3.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "Eigen3", but
CMake did not find one.
Could not find a package configuration file provided by "Eigen3" with any
of the following names:
Eigen3Config.cmake
eigen3-config.cmake
Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set
"Eigen3_DIR" to a directory containing one of the above files. If "Eigen3"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
CMakeLists.txt:101 (include)
-- Did not find system Eigen. Using third party subdirectory.
-- Found PythonInterp: /usr/bin/python (found suitable version "2.7.12", minimum required is "2.7")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7")
-- Found NumPy: /usr/local/lib/python2.7/dist-packages/numpy/core/include (found version "1.13.3")
-- NumPy ver. 1.13.3 found (include: /usr/local/lib/python2.7/dist-packages/numpy/core/include)
-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)
-- Found MPI_C: /usr/lib/openmpi/lib/libmpi.so
-- Found MPI_CXX: /usr/lib/openmpi/lib/libmpi_cxx.so;/usr/lib/openmpi/lib/libmpi.so
-- MPI support found
-- MPI compile flags:
-- MPI include path: /usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/include/usr/lib/openmpi/include/usr/lib/openmpi/include/openmpi
-- MPI LINK flags path: -Wl,-rpath -Wl,/usr/lib/openmpi/lib -Wl,--enable-new-dtags
-- MPI libraries: /usr/lib/openmpi/lib/libmpi_cxx.so/usr/lib/openmpi/lib/libmpi.so
CMake Warning at cmake/Dependencies.cmake:324 (message):
OpenMPI found, but it is not built with CUDA support.
Call Stack (most recent call first):
CMakeLists.txt:101 (include)
-- Found CUDA: /usr/local/cuda-8.0 (found suitable version "8.0", minimum required is "7.0")
-- Found CUDNN: /usr/local/cuda-8.0/include
-- Caffe2: CUDA detected: 8.0
-- Found cuDNN: v6.0.21 (include: /usr/local/cuda-8.0/include, library: /usr/local/cuda-8.0/lib64/libcudnn.so)
-- Automatic GPU detection failed. Building for all known architectures.
-- Added CUDA NVCC flags for: sm_20 sm_21 sm_30 sm_35 sm_50 sm_52 sm_60 sm_61
-- Could NOT find NCCL (missing: NCCL_INCLUDE_DIRS NCCL_LIBRARIES)
-- Could NOT find CUB (missing: CUB_INCLUDE_DIR)
-- Could NOT find Gloo (missing: Gloo_INCLUDE_DIR Gloo_LIBRARY)
-- MPI include path: /usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/usr/lib/openmpi/include/openmpi/opal/mca/event/libevent2021/libevent/include/usr/lib/openmpi/include/usr/lib/openmpi/include/openmpi
-- MPI libraries: /usr/lib/openmpi/lib/libmpi_cxx.so/usr/lib/openmpi/lib/libmpi.so
-- CUDA detected: 8.0
-- Found libcuda: /usr/local/cuda-8.0/lib64/stubs/libcuda.so
-- Found libnvrtc: /usr/local/cuda-8.0/lib64/libnvrtc.so
-- Found nccl: /home/liu/sfw/dlapp/caffe2/third_party/nccl/build/include
CMake Warning at cmake/Dependencies.cmake:457 (message):
mobile opengl is only used in android or ios builds.
Call Stack (most recent call first):
CMakeLists.txt:101 (include)
CMake Warning at cmake/Dependencies.cmake:533 (message):
Metal is only used in ios builds.
Call Stack (most recent call first):
CMakeLists.txt:101 (include)
-- GCC 5.4.0: Adding gcc and gcc_s libs to link line
-- Include NCCL operators
-- Including image processing operators
-- Excluding video processing operators due to no opencv
-- Excluding mkl operators as we are not using mkl
-- Include Observer library
-- Using lib/python2.7/dist-packages as python relative installation path
-- Automatically generating missing __init__.py files.
--
-- ******** Summary ********
-- General:
-- CMake version : 3.5.1
-- CMake command : /usr/bin/cmake
-- Git version : v0.8.1-1314-gd8770f8
-- System : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 5.4.0
-- Protobuf compiler : /usr/bin/protoc
-- Protobuf include path : /usr/include
-- Protobuf libraries : optimized;/usr/lib/x86_64-linux-gnu/libprotobuf.so;debug;/usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread
-- BLAS : Eigen
-- CXX flags : -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization
-- Build type : Release
-- Compile definitions :
--
-- BUILD_BINARY : ON
-- BUILD_DOCS : OFF
-- BUILD_PYTHON : ON
-- Python version : 2.7.12
-- Python includes : /usr/include/python2.7
-- BUILD_SHARED_LIBS : ON
-- BUILD_TEST : ON
-- USE_ATEN : OFF
-- USE_ASAN : OFF
-- USE_CUDA : ON
-- CUDA version : 8.0
-- CuDNN version : 6.0.21
-- CUDA root directory : /usr/local/cuda-8.0
-- CUDA library : /usr/local/cuda-8.0/lib64/stubs/libcuda.so
-- CUDA NVRTC library : /usr/local/cuda-8.0/lib64/libnvrtc.so
-- CUDA runtime library: /usr/local/cuda-8.0/lib64/libcudart.so
-- CUDA include path : /usr/local/cuda-8.0/include
-- NVCC executable : /usr/local/cuda-8.0/bin/nvcc
-- CUDA host compiler : /usr/bin/cc
-- USE_EIGEN_FOR_BLAS : 1
-- USE_FFMPEG : OFF
-- USE_GFLAGS : ON
-- USE_GLOG : ON
-- USE_GLOO : ON
-- USE_LEVELDB : ON
-- LevelDB version : 1.18
-- Snappy version : 1.1.3
-- USE_LITE_PROTO : OFF
-- USE_LMDB : ON
-- LMDB version : 0.9.17
-- USE_METAL : OFF
-- USE_MKL :
-- USE_MOBILE_OPENGL : OFF
-- USE_MPI : ON
-- USE_NCCL : ON
-- USE_NERVANA_GPU : OFF
-- USE_NNPACK : ON
-- USE_OBSERVERS : ON
-- USE_OPENCV : ON
-- OpenCV version : 2.4.9.1
-- USE_OPENMP : OFF
-- USE_PROF : OFF
-- USE_REDIS : OFF
-- USE_ROCKSDB : OFF
-- USE_THREADS : ON
-- USE_ZMQ : OFF
-- Configuring done
-- Generating done
-- Build files have been written to: yourcaffe2rootdir/caffe2/build
执行 make install -j$(nproc)
即开始编译构建,这个过程比较漫长,本人采用了30+线程,约10分钟。编译完出现如下提示信息:
[ 98%] Linking CXX executable ../bin/blob_test
CMakeFiles/blob_test.dir/core/blob_test.cc.o: In function `caffe2::(anonymous namespace)::ContentChunks_Serialization_Test::TestBody()':
blob_test.cc:(.text+0x3b3d6): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
[ 98%] Built target blob_test
[ 98%] Linking CXX shared module python/caffe2_pybind11_state_gpu.so
[ 98%] Built target caffe2_pybind11_state_gpu
Scanning dependencies of target caffe2_detectron_ops_gpu
[ 98%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/sample_as_op.cc.o
[ 98%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/group_spatial_softmax_op.cc.o
[ 98%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/sigmoid_focal_loss_op.cc.o
[ 98%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/select_smooth_l1_loss_op.cc.o
[ 98%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/smooth_l1_loss_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/batch_permutation_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/sigmoid_cross_entropy_loss_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/spatial_narrow_as_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/roi_pool_f_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/upsample_nearest_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/ps_roi_pool_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/softmax_focal_loss_op.cc.o
[ 99%] Building CXX object modules/detectron/CMakeFiles/caffe2_detectron_ops_gpu.dir/affine_channel_op.cc.o
[100%] Linking CXX shared library ../../lib/libcaffe2_detectron_ops_gpu.so
[100%] Built target caffe2_detectron_ops_gpu
Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:36 (file):
file cannot create directory: /usr/local/share/cmake/Caffe2. Maybe need
administrative privileges.
Makefile:104: recipe for target 'install' failed
make: *** [install] Error 1
可能会发现出现一个错误,是的,本人构建时执行的 make install -j$(nproc)
,因为没有权限,所以向 /usr/local/share/
目录写入文件时报错,是的,执行 sudo make install -j$(nproc)
即可,
...
-- Set runtime path of "/usr/local/bin/convert_db" to ""
-- Installing: /usr/local/bin/make_cifar_db
-- Set runtime path of "/usr/local/bin/make_cifar_db" to ""
-- Installing: /usr/local/bin/make_mnist_db
-- Set runtime path of "/usr/local/bin/make_mnist_db" to ""
-- Installing: /usr/local/bin/predictor_verifier
-- Set runtime path of "/usr/local/bin/predictor_verifier" to ""
-- Installing: /usr/local/bin/print_registered_core_operators
-- Set runtime path of "/usr/local/bin/print_registered_core_operators" to ""
-- Installing: /usr/local/bin/run_plan
-- Set runtime path of "/usr/local/bin/run_plan" to ""
-- Installing: /usr/local/bin/speed_benchmark
-- Set runtime path of "/usr/local/bin/speed_benchmark" to ""
-- Installing: /usr/local/bin/split_db
-- Set runtime path of "/usr/local/bin/split_db" to ""
-- Installing: /usr/local/bin/db_throughput
-- Set runtime path of "/usr/local/bin/db_throughput" to ""
-- Installing: /usr/local/bin/inspect_gpus
-- Set runtime path of "/usr/local/bin/inspect_gpus" to ""
-- Installing: /usr/local/bin/print_core_object_sizes
-- Set runtime path of "/usr/local/bin/print_core_object_sizes" to ""
-- Installing: /usr/local/bin/core_overhead_benchmark
-- Set runtime path of "/usr/local/bin/core_overhead_benchmark" to ""
-- Installing: /usr/local/bin/run_plan_mpi
-- Set runtime path of "/usr/local/bin/run_plan_mpi" to ""
-- Installing: /usr/local/bin/convert_encoded_to_raw_leveldb
-- Set runtime path of "/usr/local/bin/convert_encoded_to_raw_leveldb" to ""
-- Installing: /usr/local/bin/make_image_db
-- Set runtime path of "/usr/local/bin/make_image_db" to ""
-- Installing: /usr/local/bin/caffe2_benchmark
-- Set runtime path of "/usr/local/bin/caffe2_benchmark" to ""
-- Installing: /usr/local/bin/tutorial_blob
-- Set runtime path of "/usr/local/bin/tutorial_blob" to ""
caffe2默认被安装在根目录下/usr/local/
,当然你也可以在配置时修改安装目录。
参考 Test the Caffe2 Installation 验证安装。
为验证安装成功,终端输入 :
cd ~ && python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"
如果显示 Success
代表CPU版安装成功,如下图:
下面测试GPU,终端输入 python2 caffe2/python/operator_test/relu_op_test.py
却提示:
python2.7/dist-packages$ python2 caffe2/python/operator_test/relu_op_test.py
Traceback (most recent call last):
File "caffe2/python/operator_test/relu_op_test.py", line 22, in
from hypothesis import given
ImportError: No module named hypothesis
缺少库,pip install hypothesis
即可:
perator Relu.
I0312 14:27:49.096366 5580 operator.cc:178] Engine CUDNN is not available for operator ReluGradient.
.
----------------------------------------------------------------------
Ran 1 test in 1.888s
OK
输入如下命令,会显示显卡数量:
python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'
出现错误,参考 Troubleshooting 解决。
tools/extra
下在 tools/extra
下
参考 examples/notebooks/
文件夹下文件,如: