kaldi win10 vs2019 CPU和GPU下遇到的问题

1 需要组件

请按照顺序安装。

1.1 nvidia driver

nvidia win10 显卡型号和驱动下载地址直接双击exe安装
cuda_10.2.89_441.22_win10.exe下载地址直接双击exe安装
cudnn-10.2-windows10-x64-v7.6.5.32.zip下载地址
请将cudnn解压的库复制到C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2下即可。
【配置PATH环境变量】

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\libnvvp
C:\Program Files\NVIDIA Corporation\NVSMI

1.2 vs2019社区版

vs2019_community__1503913078.1588313013.exe vs社区版
C++桌面版全选即可

1.3 cmake

cmake-3.17.3-win64-x64.zip，解压到C:\ProgramFiles下
【或者】不用下载了，vs2019内置了make，将其添加到PATH中即可
【配置PATH环境变量】

C:\ProgramFiles\cmake-3.7.0-win64-x64\bin
# 或者
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin

1.4 git和TortoiseGit

请安装Git_V2.5.1_64_bit_setup.1441791170.exe
和TortoiseGit-1.8.15.0-64bit.msi
以及TortoiseGit-LanguagePack-1.8.15.0-64bit-zh_CN.msi中文语言包
【因为所有相关操作都在git bash上操作，windos命令行不支持mv cp等命令】

2 kaldi vs工程配置

2.1 kaldi

在这里可以看一下kaldi/windows文件夹的Reda

git clone https://github.com/kaldi-asr/kaldi.git C:\anjos\vs\kaldi-win

2.2 拷贝变量文件

在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【复制 variables.props.dev 一份为 variables.props】
(2) 【复制 kaldiwin_openblas.props 一份为 kaldiwin.props】
(3) 【复制 cuda_7.0.props 为 cuda_10.2.props】
(4) 【复制portaudio.props 为 portaudio.props.dev】

2.3 支持cuda GPU

因为安装了cudav10.2，所以发现cuda7.0已经过时，同时发现C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\BuildCustomizations下是**CUDA 10.2.props**。因此做如下变更：
在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【将 generate_solution.pl 文件中的所有cuda 7.0替换为cuda 10.2】
(2) 【将 generate_solution.pl 文件中的所有cuda_7.0替换为cuda_10.2】
(3) 【在 variables.props 的中增加】

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2
C:\Program FIles\NVIDIA Corporation\NvToolsExt\

(4) 【在variables.props的中修改】

    
      $(NVTOOLSDIR)
      true
    
    
      $(CUDADIR)
      true
    

(5)【在 cuda_10.2.props 子项下添加修改 cuda的include文件夹】

$(CUDADIR)\include;$(NVTOOLSDIR)\include;$(CUBDIR);%(AdditionalIncludeDirectories)

(6)【在 cuda_10.2.props 子项下添加修改 cuda的lib文件夹】

$(CUDADIR)\lib\x64;$(NVTOOLSDIR)\lib\x64;%(AdditionalLibraryDirectories)

(7)【在 cuda_10.2.props 子项下添加修改 cuda的依赖库】

cublas.lib;cuda.lib;cudart.lib;nvToolsExt64_1.lib;%(AdditionalDependencies)

最后样子：

  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(CUDADIR)\include;$(NVTOOLSDIR)\include;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
      <PreprocessorDefinitions>HAVE_CUDA;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(CUDADIR)\lib\x64;$(NVTOOLSDIR)\lib;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>cublas.lib;cusparse.lib;cudart.lib;curand.lib;cufft.lib;nvToolsExt64_1.lib;cusolver.lib;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>

2.4 支持vs 2019

在C:\anjos\vs\kaldi-win\windows下操作，将generate_solution.pl文件中增加vs2019相关：

my %TOOLS=( default=> "14.1",
            vs2015 => "14.0",
            vs2017 => "14.1",
			vs2019 => "v14.2"
            );
my %FORMAT=( default=> "14.10",
             vs2015 =>  "14.00",
             vs2017 =>  "14.10",
			 vs2019 => "v14.20"
             );
my %TOOLSET=( default=> "v141",
              vs2015 => "v140",
              vs2017 => "v141",
			  vs2019 => "v142"
              );

2.5 openfst

这时候配置openfst发现直接用源码其实是不ok的，kaldi用的openfst版本是1.6.2，不能直接拉openfst GitHub最新版。这里openfst官网有提供一个版本：

wget http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.2.tar.gz
wget https://pan.baidu.com/s/1o8SghqM
tar -xvf openfst-1.6.2.tar.gz
patch -b -p1 < openfst-1.6.2.patch

以vs2019方式打开C:/anjos/vs/kaldi-win/tools/openfst/openfst.sln，选择x64/Release & Debug配置分别进行编译即可，产生库在C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\x64.

在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在 variables.props 的 中修改】：

C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\
C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\x64

(2) 【在 openfstwin_debug.props 和 openfstwin_release.props 修改将子项下内容】：

libfst.lib;%(AdditionalDependencies)

(2) 【在 openfstwin_debug.props 和 openfstwin_release.props 修改将子项下内容】【这个要区分环境】：

$(OPENFSTLIB)\Release;%(AdditionalLibraryDirectories)

$(OPENFSTLIB)\Debug;%(AdditionalLibraryDirectories)

2.6 openblas

下载OpenBLAS-v0.2.14-Win64-int32.zip，解压至C:\anjos\vs\kaldi-win\tools\OpenBLAS-v0.2.14-Win64-int32
下载mingw64_dll.zip，解压至C:\anjos\vs\kaldi-win\tools\mingw64_dll
在C:\anjos\vs\kaldi-win\windows下操作：
【在variables.props的中修改】

C:\anjos\vs\kaldi-win\tools\OpenBLAS-v0.2.14-Win64-int32

2.7 PortAudio库

wget http://www.portaudio.com/archives/pa_stable_v190600_20161030.tgz -O C:\anjos\vs\kaldi-win\tools
wget http://www.steinberg.net/sdk_downloads/asiosdk2.3.zip -O C:\anjos\vs\kaldi-win\tools
#C:\anjos\vs\kaldi-win\tools\portaudio
tar -xvf pa_stable_v190600_20161030.tgz
#C:\anjos\vs\kaldi-win\tools\ASIOSDK2.3
unzip asiosdk.zip
# C:\anjos\vs\kaldi-win\tools\portaudio\src\hostapi\asio\ASIOSDK下
# 是C:\anjos\vs\kaldi-win\tools\ASIOSDK2.3的一系列文件
mv ASIOSDK2.3/* portaudio/src/hostapi/asio/ASIOSDK

msvc\portaudio.sln，选择x64/Release配置进行编译
【打开 C:\anjos\vs\kaldi-win\tools\portaudio\build\msvc\vportaudio.sln，分别在 x64|Debug 和 x64|Release 下编译生成项目】
在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在variables.props的中修改】

C:\anjos\vs\kaldi-win\tools\portaudio
C:\anjos\vs\kaldi-win\tools\portaudio\build\msvc\x64

(2)【在 variables.props 的 中修改】

    
      $(PORTAUDIO)
      true
    
      
      $(PORTAUDIOLIB)
      true
    

(3)【在 kaldiwin.props 子项下添加修改 PortAudio的include文件夹】

..\..\..\src;$(OPENBLASDIR)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common

(4)【在 kaldiwin.props 子项下添加修改 PortAudio的lib文件夹】

$(OPENBLASDIR)\lib\;%(AdditionalLibraryDirectories);$(PORTAUDIOLIB)\Debug

(5)【在 kaldiwin.props 子项下添加修改 PortAudio的依赖库】

libopenblas.dll.a;%(AdditionalDependencies);portaudio_x64.lib;pa_ringbuffer.obj

【对variables.props也做一样修改如下：】

portaudio.props最后样子：

  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(PORTAUDIO)\src\common;$(PORTAUDIO)\src\os\win;$(PORTAUDIO)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common</AdditionalIncludeDirectories>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(PORTAUDIOLIB);$(PORTAUDIOLIB)\Debug;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>portaudio_x64.lib;pa_ringbuffer.obj;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>

2.8 pthread-win

wget http://downloads.sourceforge.net/project/pthreads4w/pthreads-w32-2-9-1-release.zip
mkdir pthreads
cd pthreads
unzip C:\anjos\vs\kaldi-win\tools\pthreads-w32-2-9-1-release.zip

在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在 variables.props 的 中修改】：

C:\anjos\vs\kaldi-win\tools\pthreads-w32-2-9-1-release\Pre-built.2

2.9 cub库

cub-1.8.0下载地址
也可以直接用nvidia内嵌的cub版本C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include\thrust\system\cuda\detail（这里面有一个cub文件夹）：
在C:\anjos\vs\kaldi-win\windows下操作：
(1) 【在variables.props的中修改】

C:\anjos\vs\kaldi-win\tools\cub-1.8.0

(2) 【在kaldiwin.props的中增加变量】

..\..\..\src;$(OPENBLASDIR)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common;$(CUBDIR)

2.10 最后库界面

(1) 【kaldiwin.props】

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ImportGroup Label="PropertySheets" />
  <PropertyGroup Label="UserMacros">
    <!-- Change the following paths so they are correct on your machine -->
    <!-- Do not modify anything before this line -->
    <MKLDIR>C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\</MKLDIR>
    <OPENBLASDIR>C:\anjos\vs\kaldi-win\tools\OpenBLAS-v0.2.14-Win64-int32</OPENBLASDIR>
    <OPENFST>C:\anjos\vs\kaldi-win\tools\openfst-1.6.2</OPENFST>
    <OPENFSTLIB>C:\anjos\vs\kaldi-win\tools\openfst-1.6.2\x64</OPENFSTLIB>
    <CUBDIR>C:\anjos\vs\kaldi-win\tools\cub-1.8.0</CUBDIR>
    <PORTAUDIO>C:\anjos\vs\kaldi-win\tools\portaudio</PORTAUDIO>
    <PORTAUDIOLIB>C:\anjos\vs\kaldi-win\tools\portaudio\build\msvc\x64</PORTAUDIOLIB>
	<PTHREADW>C:\anjos\vs\kaldi-win\tools\pthreads-w32-2-9-1-release\Pre-built.2</PTHREADW>
	<NVTOOLSDIR>C:\Program FIles\NVIDIA Corporation\NvToolsExt</NVTOOLSDIR>
	<CUDADIR>C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2</CUDADIR>
    <!-- Do not modify anything after this line -->
  </PropertyGroup>
  <PropertyGroup />
  <ItemDefinitionGroup />
  <ItemGroup>
    <BuildMacro Include="OPENBLASDIR">
      <Value>$(OPENBLASDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="OPENFST">
      <Value>$(OPENFST)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="OPENFSTLIB">
      <Value>$(OPENFSTLIB)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="MKLDIR">
      <Value>$(MKLDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="CUBDIR">
      <Value>$(CUBDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="PORTAUDIO">
      <Value>$(PORTAUDIO)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
      <BuildMacro Include="PORTAUDIOLIB">
      <Value>$(PORTAUDIOLIB)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
    <BuildMacro Include="NVTOOLSDIR">
      <Value>$(NVTOOLSDIR)</Value>
      <EnvironmentVariable>true</EnvironmentVariable>
    </BuildMacro>
  </ItemGroup>
</Project>

(2)【kaldiwin.props】

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
    <LinkIncremental>
    </LinkIncremental>
  </PropertyGroup>
  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalOptions>/bigobj  %(AdditionalOptions)</AdditionalOptions>
      <AdditionalIncludeDirectories>..\..\..\src;$(OPENBLASDIR)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common;$(CUBDIR)</AdditionalIncludeDirectories>
      <PreprocessorDefinitions>HAVE_OPENBLAS;HAVE_LAPACK_CONFIG_H;LAPACK_COMPLEX_STRUCTURE;USE_ONLY_PRERECORDED;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(OPENBLASDIR)\lib\;%(AdditionalLibraryDirectories);$(PORTAUDIOLIB)\Debug</AdditionalLibraryDirectories>
      <AdditionalDependencies>libopenblas.dll.a;%(AdditionalDependencies);portaudio_x64.lib;pa_ringbuffer.obj</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
</Project>

(3)【cuda_10.2.props】

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
    <LinkIncremental>
    </LinkIncremental>
  </PropertyGroup>
  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(CUDADIR)\include;$(NVTOOLSDIR)\include;$(CUBDIR);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
      <PreprocessorDefinitions>HAVE_CUDA;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(CUDADIR)\lib\x64;$(NVTOOLSDIR)\lib\x64;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>cublas.lib;cuda.lib;cudart.lib;nvToolsExt64_1.lib;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
</Project>

(4) 【portaudio.props】

【这个要区分debug和release版本】，不然release下报错，这个bug我已经提交到github上了，已经merge到kaldi的master，详细见fix bug window10 release reprot error, and , debug normal.

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
    <LinkIncremental>
    </LinkIncremental>
  </PropertyGroup>
  <ItemDefinitionGroup>
    <ClCompile>
      <AdditionalIncludeDirectories>$(PORTAUDIO)\src\common;$(PORTAUDIO)\src\os\win;$(PORTAUDIO)\include;%(AdditionalIncludeDirectories);$(PORTAUDIO)\include;$(PORTAUDIO)\src\common</AdditionalIncludeDirectories>
    </ClCompile>
    <Link>
      <AdditionalLibraryDirectories>$(PORTAUDIOLIB);$(PORTAUDIOLIB)\Debug;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
      <AdditionalDependencies>portaudio_x64.lib;pa_ringbuffer.obj;%(AdditionalDependencies)</AdditionalDependencies>
    </Link>
  </ItemDefinitionGroup>
</Project>

3 kaldi vs工程生成

3.1 生成脚本

# 仅CPU
./generate_solution.pl --vsver vs2019 --enable-openblas
# 或也可以启用cuda GPU(这里我使用的是GPU)
./generate_solution.pl --vsver vs2019 --enable-cuda --enable-openblas
./get_version.pl

如果仅是启用CPU会出现错误信息，不用管：

ERROR?: file \c\anjos\vs\kaldi-win\windows\..\src\cudadecoder\batched-static-nnet3-kernels.cc not found - project kaldi-cudadecoder
ERROR?: file \c\anjos\vs\kaldi-win\windows\..\src\cudadecoder\cuda-decoder-kernels.cc not found - project kaldi-cudadecoder

3.2 最终编译

打开C:\anjos\vs\kaldi-win\kaldiwin_vs2019_OPENBLAS_CUDA\kaldiwin_vs2019.sln，在选择x64/Release & Debug配置分别进行编译即可

3.3 测试编译过程

【最后可以在batched-wav-nnet3-cuda模块（GPU）或 online2-wav-nnet3-latgen-faster 模块（CPU）】选择Debug|x64去编译

3.4 在此基础上新建自己工程

请参考Kaldi VS2019下新建工程

Reference

kaldi window install
Building of Kaldi VS2015 Project on Windows
Win10 VS 2019 preview+CUDA10.1+cudnn7.5.1 配置kaldi