Tensorflow 1.8 源码安装
Tensorflow源码安装
cuda 9.2 / cudnn 7.2 / python3.5 / tensorflow 1.8
1. anacanda
- 启动命令
root$ source activate python3.5
2. cuda
-
查看版本命令
root$cat /usr/local/cuda/version.txt -
下载链接：
https://developer.nvidia.com/cuda-92-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal -
下载版本
sudo sh cuda_9.2.148_396.37_linux.run -
安装指南：
https://developer.nvidia.com/cuda-92-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal -
验证方式参考：
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#verify-installation -
测试命令：
编译：
$cd ~/NVIDIA_CUDA-9.2_Sample;
$make;
测试：cd ～/NVIDIA_CUDA-9.2_Sample ; ./bin/x86_64/linux/release/deviceQuery -
验证结果：
测试通过deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.2, CUDA Runtime Version = 9.2, NumDevs = 8 Result = PASS -
结果：
PASS
3. cudnn
3.1 cudnn 7.1 安装 [失败]
-
查看版本命令
root#cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2 -
下载链接：
https://developer.nvidia.com/rdp/cudnn-archive -
安装文档：
https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html -
下载版本：
cuDNN v7.1.4 Runtime Library for Ubuntu16.04 (Deb) cuDNN v7.1.4 Developer Library for Ubuntu16.04 (Deb) cuDNN v7.1.4 Code Samples and User Guide for Ubuntu16.04 (Deb) -
安装命令：
dpkg -i cuDNN v7.1.4 Runtime Library for Ubuntu16.04 (Deb) dpkg -i cuDNN v7.1.4 Developer Library for Ubuntu16.04 (Deb) dpkg -i cuDNN v7.1.4 Code Samples and User Guide for Ubuntu16.04 (Deb)sudo apt update
-
环境变量：
export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64 -
校验：
`cd ~/cudnn_samples_v7/mnistCUDNN
make
./mnistCUDNN
Bus error (core dumped)`` -
结果：
FAILED
‘’‘7.1 失败了，重新下载 7.2’’’
3.2cudnn 7.2 安装
-
查看版本命令
root#cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2 -
下载链接：
https://developer.nvidia.com/rdp/cudnn-archive -
安装文档：
https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html -
下载版本：
cuDNN v7.2.1 Runtime Library for Ubuntu16.04 (Deb) cuDNN v7.2.1 Developer Library for Ubuntu16.04 (Deb) cuDNN v7.2.1 Code Samples and User Guide for Ubuntu16.04 (Deb) -
安装命令：
dpkg -i libcudnn7_7.2.1.38-1+cuda9.2_amd64.deb dpkg -i libcudnn7-doc_7.2.1.38-1+cuda9.2_amd64.deb dpkg -i libcudnn7-dev_7.2.1.38-1+cuda9.2_amd64.debsudo apt update
-
校验：
- 命令
cd ~/cudnn_samples_v7/mnistCUDNN make ./mnistCUDNN
- 命令
-
结果
Result of classification: 1 3 5 Test passed!-
拷贝
参考:
https://github.com/tensorflow/tensorflow/issues/2626
https://blog.csdn.net/xuezhisdc/article/details/48651003希望cudnn被tensorflow编译时发现，需要将生成的文件拷贝至
/usr/local/cuda/lib64/
以及/usr/local/cuda/include/-
查看：
step1:
先看一下/usr/lib/x86_64-linux-gnu/下文件是否完整(python3.\*) root@:/# ll /usr/lib/x86_64-linux-gnu/libcudnn* lrwxrwxrwx 1 root root 29 9月 27 09:29 /usr/lib/x86_64-linux-gnu/libcudnn.so -> /etc/alternatives/libcudnn_so lrwxrwxrwx 1 root root 17 7月 31 14:54 /usr/lib/x86_64-linux-gnu/libcudnn.so.7 -> libcudnn.so.7.2.1 -rw-r--r-- 1 root root 288585696 7月 31 14:54 /usr/lib/x86_64-linux-gnu/libcudnn.so.7.2.1 lrwxrwxrwx 1 root root 32 9月 27 09:29 /usr/lib/x86_64-linux-gnu/libcudnn_static.a -> /etc/alternatives/libcudnn_stlib -rw-r--r-- 1 root root 281810850 7月 31 14:54 /usr/lib/x86_64-linux-gnu/libcudnn_static_v7.astep2:
如果完整，将这些文件拷贝至/usr/local/cuda/lib64
cp /usr/lib/x86_64-linux-gnu/libcudnn* /usr/local/cuda/lib64/step3:
需要把cudnn.h拷贝到 cuda 路径下
`sudo cp cudnn.h /usr/local/cuda/include/``
理论上应重新下载 7.2.1的 cudnn.h 文件，但我保留了 7.1 的 cudnn.h 安装未见异常。
如果有问题，下载的话可以到
-
-
4. bazel install
- 路径： /home/royal/thirdparty/
- 文件： /home/royal/thirdparty/bazel-0.16.0-installer-linux-x86_64.sh
- 命令：
(python3.5) root$ chomd +x bazel-0.16.0-installer-linux-x86_64.sh
5. nccl 2
-
地址：
https://developer.nvidia.com/nccl/nccl-download -
版本：
nccl-repo-ubuntu1604-2.2.12-ga-cuda9.2_1-1_amd64.deb -
文档：
https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html#down -
命令：
$dpkg -i nccl-repo-ubuntu1604-2.2.12-ga-cuda9.2_1-1_amd64.deb $sudo apt update $sudo apt install libnccl2 libnccl-dev然后把libnccl.so.2 和nccl.h拷贝到对应路径。
$cp /usr/lib/x86_64-linux-gnu/libnccl.so.2 /usr/local/cuda-9.2/lib/libnccl.so.2 $cp /usr/include/nccl.h /usr/local/cuda-9.2/include/ -
完成。
6. tensorflow
-
文档：
https://www.tensorflow.org/install/source -
配置问题1：
configure 过程中出现找不到 nccl
重新安装了nccl，参考5 -
配置:
./configuer -
编译
$ bazel build --config=opt --config=cuda tensorflow/tools/pip_package:build_pip_package --action_env="LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" -
结果
看起来是成功了INFO: From Executing genrule //tensorflow/python/estimator/api:estimator_python_api_gen: tf.estimator package not installed. tf.estimator package not installed. Target //tensorflow/tools/pip_package:build_pip_package up-to-date: bazel-bin/tensorflow/tools/pip_package/build_pip_package INFO: Elapsed time: 668.037s, Critical Path: 240.01s INFO: 8021 processes: 8021 local. INFO: Build completed successfully, 10483 total actions -
错误
但运行下面命令出现错误：
python -c 'import tensorflow as tf; print(tf.__version__)'ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory看起来需要安装一下 cublas了。
libcublas
在设备上find / -name libcublas* ，发现在在libcublas 该有的都有：
/root/anaconda3/lib/libcublas.so.9.0
/root/anaconda3/lib/libcublas.so
/root/anaconda3/lib/libcublas.so.9.0.176
/root/anaconda3/pkgs/cudatoolkit-9.0-h13b8566_0/lib/libcublas.so.9.0
/root/anaconda3/pkgs/cudatoolkit-9.0-h13b8566_0/lib/libcublas.so
/root/anaconda3/pkgs/cudatoolkit-9.0-h13b8566_0/lib/libcublas.so.9.0.176
/usr/lib/x86_64-linux-gnu/libcublas_static.a
/usr/lib/x86_64-linux-gnu/libcublas.so.7.5.18
/usr/lib/x86_64-linux-gnu/stubs/libcublas.so
/usr/lib/x86_64-linux-gnu/libcublas_device.a
/usr/lib/x86_64-linux-gnu/libcublas.so
/usr/lib/x86_64-linux-gnu/libcublas.so.7.5
/usr/share/doc/libcublas7.5
/usr/share/lintian/overrides/libcublas7.5
/usr/share/man/man7/libcublas.so.7.gz
/usr/share/man/man7/libcublas.so.7
/usr/share/man/man7/libcublas.7.gz
/usr/share/man/man7/libcublas.7
/usr/local/cuda-9.2/doc/man/man7/libcublas.so.7
/usr/local/cuda-9.2/doc/man/man7/libcublas.7
/usr/local/cuda-9.2/lib64/libcublas_static.a
/usr/local/cuda-9.2/lib64/libcublas.so.9.2
/usr/local/cuda-9.2/lib64/stubs/libcublas.so
/usr/local/cuda-9.2/lib64/libcublas.so.9.2.148
/usr/local/cuda-9.2/lib64/libcublas_device.a
/usr/local/cuda-9.2/lib64/libcublas.so
/usr/local/cuda-9.0/doc/man/man7/libcublas.so.7
/usr/local/cuda-9.0/doc/man/man7/libcublas.7
/usr/local/cuda-9.0/lib64/libcublas_static.a
/usr/local/cuda-9.0/lib64/stubs/libcublas.so
/usr/local/cuda-9.0/lib64/libcublas.so.9.0
/usr/local/cuda-9.0/lib64/libcublas_device.a
/usr/local/cuda-9.0/lib64/libcublas.so
/usr/local/cuda-9.0/lib64/libcublas.so.9.0.176
/var/lib/dpkg/info.bak/libcublas7.5:amd64.shlibs
/var/lib/dpkg/info.bak/libcublas7.5:amd64.symbols
/var/lib/dpkg/info.bak/libcublas7.5:amd64.list
/var/lib/dpkg/info.bak/libcublas7.5:amd64.triggers
/var/lib/dpkg/info.bak/libcublas7.5:amd64.md5sums
于是思考，是不是环境没有正确加载到 libcublas* ,
参考这个链接 https://github.com/tensorflow/tensorflow/issues/15604
运行了这两条命令：
$bash -c "echo /usr/local/cuda/lib64/ >/etc/ld.so.conf.d/cuda.conf"
$ldconfig
然后试了一下
$python -c “import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))”
仍然报libcublas找不到的错误。
于是想了一下，我在anaconda下安装的，而这个env之前pip install tensorflow-gpu安装过一次，难道现在引入的是之前的 tensorflow ?
然后通过pip list 看了一下，果然有一个 tensorflow , 于是 source deactivate 之后 ,
运行了$python -c “import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))”,
发现在 anaconda之外是OK的，但版本是1.7？
所以，我再次激活了python3.5，通过pip uninstall tensorflow 卸载了之前的tf，又跑了一遍python -c 'import tensorflow as tf; print(tf.__version__)',发现正常了，但版本仍然是1.7。
表示现在1.7版本可用了，再用$python -c 'import tensorflow as tf; sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))'命令测了一下，现实8卡P100可用了。
2018-09-27 12:36:10.774651: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2018-09-27 12:36:43.384597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:88:00.0
totalMemory: 15.90GiB freeMemory: 360.88MiB
2018-09-27 12:36:43.643579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 1 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:8d:00.0
totalMemory: 15.90GiB freeMemory: 510.88MiB
2018-09-27 12:36:43.903969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 2 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:8e:00.0
totalMemory: 15.90GiB freeMemory: 510.88MiB
2018-09-27 12:36:44.170843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 3 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:8f:00.0
totalMemory: 15.90GiB freeMemory: 510.88MiB
2018-09-27 12:37:02.001743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 4 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:b2:00.0
totalMemory: 15.90GiB freeMemory: 510.88MiB
2018-09-27 12:37:02.286188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 5 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:b3:00.0
totalMemory: 15.90GiB freeMemory: 510.88MiB
2018-09-27 12:37:02.574992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 6 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:b5:00.0
totalMemory: 15.90GiB freeMemory: 510.88MiB
2018-09-27 12:37:02.578074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6
Thu Sep 27 12:36:42 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44 Driver Version: 396.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:88:00.0 Off | Off |
| N/A 27C P0 30W / 250W | 15637MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla P100-PCIE... Off | 00000000:8D:00.0 Off | Off |
| N/A 27C P0 29W / 250W | 15487MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla P100-PCIE... Off | 00000000:8E:00.0 Off | Off |
| N/A 29C P0 29W / 250W | 15487MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla P100-PCIE... Off | 00000000:8F:00.0 Off | Off |
| N/A 26C P0 29W / 250W | 15487MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 Tesla P100-PCIE... Off | 00000000:B2:00.0 Off | Off |
| N/A 25C P0 29W / 250W | 15487MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 Tesla P100-PCIE... Off | 00000000:B3:00.0 Off | Off |
| N/A 29C P0 31W / 250W | 15487MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 6 Tesla P100-PCIE... Off | 00000000:B5:00.0 Off | Off |
| N/A 28C P0 32W / 250W | 15487MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 63644 C python 15625MiB |
| 1 63644 C python 15475MiB |
| 2 63644 C python 15475MiB |
| 3 63644 C python 15475MiB |
| 4 63644 C python 15475MiB |
| 5 63644 C python 15475MiB |
| 6 63644 C python 15475MiB |
+-----------------------------------------------------------------------------+
OK，基本装好。
tensorflow 1.8
现在回头解决1.7版本的问题。 在未激活 anaconda 状态下是，查看版本是1.7，anaconda激活后，也是1.7，所以怀疑
是在未激活状态下，默认用pip 安装了 1.7 tensorflow-gpu。为了验证，我激活了python3.5 conda，然后重新编译
tensorflow。