windows10下theano启用gpu：CUDA + Anaconda

    最近在学习深度学习的相关内容，照着http://deeplearning.net/tutorial/中每个章节的例子一个个对着代码做。并且将部分代码重构了，代码地址在https://github.com/wangzhics/deeplearning里面了。

   deeplearning里面的代码是基于http://deeplearning.net/software/theano/tutorial/index.html#tutorial的，theano中我们可以定义变量，使用其内置的公式并组合，而不用关心具体的实现过程，能够加快我们的学习速度。theano中的代码可以在GPU上跑，前面为了快速上手学习算法，一直没有折腾这个，这两天花了些时间弄好了。

    其实官方文档http://deeplearning.net/software/theano/install_windows.html#gpu-windows已经比较全了，但是由于其是基于比较旧的软件版本进行介绍的，在用新版本时难免会遇到一些问题，下面是过程的一些记录

安装CUDA

    CUDA是nvidia提供可以使用C++进行GPU编程的接口，其官方下载地址为 https://developer.nvidia.com/cuda-downloads，最新版本为7.5。

    CUDA需要C++的编译器，Windows下可以使用Visual C++，我们可以直接下载其官网推荐的Visual Studio。 但是注意：CUDA 7.5不支持VS 2015， http://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/#axzz46v2MC6l8，我选择的是VS 2013 Express。

CUDA和VS2013直接下载安装就好，并不需要设置，安装好后先添加系统环境变量

CUDA在安装的时候可能会提醒找不到系统找不到VS环境，这个没关系，我们安装VS2013只是为了给CUDA编译环境。我们可以运行CUDA提供的例子来判断CUDA是否安装好了，打开C:\ProgramData\NVIDIA Corporation\CUDA Samples\v7.5\1_Utilities\deviceQuery

直接运行 deviceQuery.cpp，可以看到自己的显卡设备信息，这样CUDA就安装好了

安装Theano

    windows下有很多python科学计算的发行版工具包，我直接用的Anaconda ，其安装简单方便，官方下载地址为 https://www.continuum.io/downloads，我用的python3.x的版本（即Anaconda3），直接下载安装包安装就好。 注意：网上很多教程都是说直接运行

conda install mingw libpython

 来安装mingw和libpython，当使用python3.x的Anaconda时只能正常安装mingw，libpython并不能安装成功，会出现如下的错误

C:\Users\WangZ>conda install libpython
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: ....

The following specifications were found to be in conflict:
  - conda -> conda-env
  - conda -> menuinst
  - conda -> pycosat
  - conda -> python 2.7*|3.3*|3.4*
  - conda -> pyyaml
  - conda -> requests
  - conda-env (target=conda-env-2.4.5-py35_0.tar.bz2) -> python 2.7*|3.3*|3.4*|3.5*
  - libpython -> python 2.6*|2.7*|3.3*|3.4*
  - menuinst (target=menuinst-1.3.2-py35_0.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*|3.5*
  - pip (target=pip-8.1.1-py35_1.tar.bz2) -> python 2.7*|3.3*|3.4*
  - pycosat (target=pycosat-0.6.1-py35_0.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*|3.5*
  - python 3.5*
  - pyyaml (target=pyyaml-3.11-py35_3.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*
  - requests (target=requests-2.9.1-py35_0.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*|3.5*
  - setuptools (target=setuptools-20.3-py35_0.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*|3.5*
  - wheel (target=wheel-0.29.0-py35_0.tar.bz2) -> python 2.7*|3.3*|3.4*|3.5*
Use "conda info " to see the dependencies for each package.

这是因为libpython只支持到python3.4.x，而Anaconda3默认的环境是python3.5.1

C:\Users\WangZ>conda info libpython
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....

libpython 1.0 py26_0
--------------------
file name   : libpython-1.0-py26_0.tar.bz2
name        : libpython
version     : 1.0
build number: 0
build string: py26_0
channel     : defaults
size        : 29 KB
date        : 2014-08-22
license_family: Other
md5         : 11614b71090b7ad155e033c14f8216db
installed environments:
dependencies:
    mingw 4.7
    python 2.6*

...

libpython 1.0 py34_1
--------------------
file name   : libpython-1.0-py34_1.tar.bz2
name        : libpython
version     : 1.0
build number: 1
build string: py34_1
channel     : defaults
size        : 88 KB
date        : 2014-08-22
license_family: Other
md5         : b13c6d4f28d81e05f85cb50e0f6b63eb
installed environments:
dependencies:
    mingw 4.7
    python 3.4*

那咋办呢？Anaconda很强大我们可以创建指定版本的python环境，执行如下命令：

C:\Users\WangZ>cd /d E:\Anaconda3\Scripts

E:\Anaconda3\Scripts>conda create -n python34 python=3.4.4
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: .........

Package plan for installation in environment E:\Anaconda3\envs\python34:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    vs2010_runtime-10.00.40219.1|                0         1.1 MB
    python-3.4.4               |                4        31.7 MB
    setuptools-20.7.0          |           py34_0         767 KB
    wheel-0.29.0               |           py34_0         123 KB
    pip-8.1.1                  |           py34_1         1.6 MB
    ------------------------------------------------------------
                                           Total:        35.3 MB

The following NEW packages will be INSTALLED:

    pip:            8.1.1-py34_1
    python:         3.4.4-4
    setuptools:     20.7.0-py34_0
    vs2010_runtime: 10.00.40219.1-0
    wheel:          0.29.0-py34_0

Proceed ([y]/n)? y

Fetching packages ...
vs2010_runtime 100% |###############################| Time: 0:00:09 121.08 kB/s
python-3.4.4-4 100% |###############################| Time: 0:02:51 194.47 kB/s
setuptools-20. 100% |###############################| Time: 0:00:05 134.58 kB/s
wheel-0.29.0-p 100% |###############################| Time: 0:00:01  81.65 kB/s
pip-8.1.1-py34 100% |###############################| Time: 0:00:17  99.02 kB/s
Extracting packages ...
[      COMPLETE      ]|##################################################| 100%
Linking packages ...
[      COMPLETE      ]|##################################################| 100%
#
# To activate this environment, use:
# > activate python34
#

E:\Anaconda3\Scripts>

该指令会在env下创建一个python3.4.4的环境，文件名就是-n的参数。

然后我们需要把旧的系统参数指向我们新的Python3.4.4

改为

然后重启，一定要重启

改完后conda命令就不能用了，可以使用绝对路径引用，其位置在Anaconda3\Scripts下

然后运行“conda install libpython -n python34”即可，libpython依赖于mingw，mingw会自动安装

E:\Anaconda3\Scripts>conda install libpython -n python34
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: .........

Package plan for installation in environment E:\Anaconda3\envs\python34:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mingw-4.7                  |                1        56.1 MB
    libpython-1.0              |           py34_1          88 KB
    ------------------------------------------------------------
                                           Total:        56.2 MB

The following NEW packages will be INSTALLED:

    libpython: 1.0-py34_1
    mingw:     4.7-1

Proceed ([y]/n)? y

Fetching packages ...
mingw-4.7-1.ta 100% |###############################| Time: 0:04:28 218.86 kB/s
libpython-1.0- 100% |###############################| Time: 0:00:01  72.85 kB/s
Extracting packages ...
[      COMPLETE      ]|##################################################| 100%
Linking packages ...
[      COMPLETE      ]|##################################################| 100%

E:\Anaconda3\Scripts>

安装Theano

在Script下直接运行“pip install theano”会出错

E:\Anaconda3\Scripts>cd E:\Anaconda3\envs\python34\Scripts
E:\Anaconda3\envs\python34\Scripts>pip install theano
Collecting theano
Collecting scipy>=0.11 (from theano)
  Using cached scipy-0.17.0.tar.gz
    Complete output from command python setup.py egg_info:
    non-existing path in 'numpy\\distutils': 'site.cfg'
    Running from numpy source directory.
    C:\Users\WangZ\AppData\Local\Temp\easy_install-z7doyvws\numpy-1.11.0\setup.py:327: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      warnings.warn("Unrecognized setuptools command, proceeding with "
    
      
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\pkg_resources\__init__.py", line 1104, in obtain
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\setuptools\dist.py", line 380, in fetch_build_egg
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\setuptools\command\easy_install.py", line 640, in easy_install
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\setuptools\command\easy_install.py", line 670, in install_item
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\setuptools\command\easy_install.py", line 850, in install_eggs
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\setuptools\command\easy_install.py", line 1078, in build_and_install
      File "E:\Anaconda3\envs\python34\lib\site-packages\setuptools-20.7.0-py3.4.egg\setuptools\command\easy_install.py", line 1066, in run_setup
    distutils.errors.DistutilsError: Setup script exited with error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat).

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\WangZ\AppData\Local\Temp\pip-build-y5922frd\scipy\

E:\Anaconda3\envs\python34\Scripts>

这是因为我们没有python的编译环境，python依赖于VC2010，我们可以安装VC2010然后在命令行里运行VC2010下的vcvarsall.bat，但是Anaconda提供了更简单的办法

E:\Anaconda3\Scripts>conda install scipy -n python34
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
Solving package specifications: .........

Package plan for installation in environment E:\Anaconda3\envs\python34:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.11.0               |           py34_0         2.7 MB
    scipy-0.17.0               |      np111py34_1        10.3 MB
    ------------------------------------------------------------
                                           Total:        13.0 MB

The following NEW packages will be INSTALLED:
    mkl:   11.3.1-0
    numpy: 1.11.0-py34_0
    scipy: 0.17.0-np111py34_1

Proceed ([y]/n)? y
Fetching packages ...
numpy-1.11.0-p 100% |###############################| Time: 0:00:18 147.81 kB/s
scipy-0.17.0-n 100% |###############################| Time: 0:01:06 162.73 kB/s
Extracting packages ...
[      COMPLETE      ]|##################################################| 100%
Linking packages ...
[      COMPLETE      ]|##################################################| 100%
E:\Anaconda3\Scripts>

E:\Anaconda3\envs\python34\Scripts>pip install theano
Collecting theano
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.1 in e:\anaconda3\envs\python34\lib\site-packages (from theano)
Collecting six>=1.9.0 (from theano)
  Using cached six-1.10.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.11 in e:\anaconda3\envs\python34\lib\site-packages (from theano)
Installing collected packages: six, theano
Successfully installed six-1.10.0 theano-0.8.2
E:\Anaconda3\envs\python34\Scripts>

配置Theano

在个人主文件夹下新建一个“.theanorc.txt”的文档，例如我的在C:\Users\WangZ，该文档如何配置参考 http://deeplearning.net/software/theano/library/config.html

[global]
floatX = float32
device = gpu
[nvcc]
flags=-LE:\Anaconda3\envs\python34\libs
compiler_bindir=C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/bin

运行结果

参考 http://deeplearning.net/software/theano/tutorial/using_gpu.html里面的python测试脚本

from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

直接运行，会报错

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 11: invalid continuation byte

这是因为掉用nvcc返回的字符串是使用的cwindows默认的默认编码，把源码出错的地方改为 *.decode("GBK")就可以，当时改源码忘记截图了...

E:\Anaconda3\envs\python34\python.exe E:/ML/deeplearning/test_gpu.py
Using gpu device 0: GeForce GT 750M (CNMeM is disabled, cuDNN not available)
DEBUG: nvcc STDOUT mod.cu
   正在创建库 C:/Users/WangZ/AppData/Local/Theano/compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.4.4-64/tmp313wmfq0/meb404c8cd39208f6884dd773b584b7d7.lib 和对象 C:/Users/WangZ/AppData/Local/Theano/compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.4.4-64/tmp313wmfq0/meb404c8cd39208f6884dd773b584b7d7.exp

DEBUG: nvcc STDOUT mod.cu
   正在创建库 C:/Users/WangZ/AppData/Local/Theano/compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.4.4-64/tmpk8he1yq1/md9b7b5ef5d727b2e8e3d2cc34bb9d0eb.lib 和对象 C:/Users/WangZ/AppData/Local/Theano/compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.4.4-64/tmpk8he1yq1/md9b7b5ef5d727b2e8e3d2cc34bb9d0eb.exp

DEBUG: nvcc STDOUT mod.cu
   正在创建库 C:/Users/WangZ/AppData/Local/Theano/compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.4.4-64/tmp3mv9lkx8/mfeb646abb02bed7115bdc204940fcfff.lib 和对象 C:/Users/WangZ/AppData/Local/Theano/compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.4.4-64/tmp3mv9lkx8/mfeb646abb02bed7115bdc204940fcfff.exp

[GpuElemwise{exp,no_inplace}(), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 1.319867 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu

BINGO！

注意事项

1. CUBE 和 VS 的版本一定要选对，并将VS的路径加入环境变量
2. Anaconda只负责和python有关的，它能够安装那些在本机无法编译的模块，就是自动帮你管理了一个mingw，如果想要自己手动装一个mingw也是可以的
   
3. Theano的开发环境比较旧，但是原理还是python调用动态库，一步步来，出问题了多Google

a) 如果出现如下错误就是因为没有安装libpython，或者新建的python3.4.4的环境后没改环境变量并重启

C:\Users\WangZ\AppData\Local\Temp\ccU0pNP5.o: In function `_import_array':
E:/Anaconda3/lib/site-packages/numpy/core/include/numpy/__multiarray_api.h:1641: undefined reference to `__imp_PyExc_ImportError'
E:/Anaconda3/lib/site-packages/numpy/core/include/numpy/__multiarray_api.h:1647: undefined reference to `__imp_PyExc_AttributeError'
E:/Anaconda3/lib/site-packages/numpy/core/include/numpy/__multiarray_api.h:1652: undefined reference to `__imp_PyCapsule_Type'
E:/Anaconda3/lib/site-packages/numpy/core/include/numpy/__multiarray_api.h:1653: undefined reference to `__imp_PyExc_RuntimeError'

b) 网上有人在.theanorc.txt里面配置了gcc，其实并不需要，因为在Anaconda的环境变量里面已经有了