caffe compilation memo

Compile caffe from source for bottom-up

NCCL (nvidia.com has been blocked download local installer then apt update to install)

# https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html
# download local installer  https://developer.nvidia.com/nccl
sudo dpkg -i *.deb
sudo apt update
sudo apt install libnccl2 libnccl-dev

CuDNN

# download "cudnn library for linux"
tar xfv cudnn-10.0-linux-x64-v7.6.4.38.tgz
sudo cp cuda/include/* /usr/local/cuda/include
sudo cp cuda/lib64/* /usr/local/cuda/lib64

Dependencies

• The following steps are for ubuntu==16.04.
• For ubuntu>=17.04 , try sudo apt build-dep caffe-cuda to install dependencies)

sudo apt-get install libopenblas-dev # OpenBLAS

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install — no-install-recommends libboost-all-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo pip install protobuf
sudo apt-get install the python-dev

For bottom-up-attention

conda create -n py2caffe python=2.7 pip cython hdf5 pandas pyyaml
conda activate py2caffe
pip install protobuf json_lines jsonlines  # vilbert
pip install easydict opencv-python scikit-image 
cd bottom-up-attention/lib
make
cd ../caffe
make -j24 && make pycaffe

Make

cd ../caffe
cp Makefile.config.example Makefile.config

Make the following changes in Makefile.config file

WITH_PYTHON_LAYER := 1

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

CUDA_ARCH := -gencode arch=compute_61,code=sm_61 # GeForce GTX 1080 (Ti) , check out https://developer.nvidia.com/cuda-gpus to find compute capability
BLAS := open  # OpenBLAS is faster
ANACONDA_HOME := $(HOME)/anaconda3/envs/py2bottom
# Uncomment to use Python 3 (default is Python 2)
PYTHON_LIBRARIES := boost_python3 python3.7m
PYTHON_INCLUDE := $(ANACONDA_HOME)/include/python3.7m \
                  $(ANACONDA_HOME)/lib/python3.7/site-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) $(ANACONDA_HOME)/pkgs/hdf5-1.10.4-hb1b8bf9_0/include /usr/local/include /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) $(ANACONDA_HOME)/pkgs/hdf5-1.10.4-hb1b8bf9_0/lib /usr/local/lib /usr/lib usr/lib/x86_64-linux-gnu/hdf5/serial

USE_NCCL := 1

Make to run test

make all -j24
make test
make runtest

Reference:

https://medium.com/@atinesh/caffe-installation-on-ubuntu-18-04-lts-python-2-7-8e8c388ce51f

https://caffe.berkeleyvision.org/installation.html#compilation