Ubuntu16.04环境下Python下xlearn机器学习库的配置
一、xlearn的简介
参见：https://www.zhihu.com/question/37256015/answer/268151326，http://www.sohu.com/a/206728248_206784
在机器学习里，除了深度学习和树模型 (GBDT, RF) 之外，如何高效地处理高维稀疏数据也是非常重要的课题，Sparse LR, FM, FFM 这些算法被广泛运用在实际生产和kaggle比赛中。现有的开源软件例如 liblinear, libfm, libffm 都只能针对特定的算法，并且可扩展性、灵活性、易用性都不够友好。
相比于已有的软件，这款一款针对于海量数据处理任务的分布式机器学习系统xlearn（区别于360的深度学习调度平台 XLearning，xlearn来源于北大信科的肖臻课题组）的优势主要有：
1、通用性好，我们用统一的架构将主流的算法（lr, fm, ffm 等）全部囊括，用户不用再切换于不同软件之间。
2、性能好。xlearn由高性能c++开发，提供 cache-aware 和 lock-free learning，并且经过手工 SSE／AVX 指令优化。 在单机MacBook Pro上测试 xlearn 可以比 libfm 快13倍，比 libffm 和 liblinear 快5倍（基于Criteo CTR数据 bechmark）。
3、易用性和灵活性，xlearn 提供简单的 python 接口，并且集合了机器学习比赛中许多有用的功能，例如：cross-validation，early-stopping 等。除此之外，用户可以灵活选择优化算法（例如，SGD，AdaGrad, FTRL 等）。
4、可扩展性好。xlearn 提供 out-of-core 计算，利用外存计算可以在单机处理 1TB 数据。
二、xlearn的安装与配置（pip）
参见： http://xlearn-doc.readthedocs.io/en/latest/index.html
使用pip安装xlearn库：
yuhuiliu@sinclab-desktop:~$ sudo pip install xlearn
[sudo] password for yuhuiliu:
Collecting xlearn
Downloading https://files.pythonhosted.org/packages/1a/20/d2762ecfd0da63bf2f0ee95429c7cf8ad44ab8ad4adc48b405fa67a09848/xlearn-0.31a1.tar.gz (1.8MB)
100% |████████████████████████████████| 1.9MB 4.2MB/s
Building wheels for collected packages: xlearn
Running setup.py bdist_wheel for xlearn ... done
Stored in directory: /home/yuhuiliu/.cache/pip/wheels/9c/46/1a/e7682af4ef3320ad6e106c633aea0ee46ffb353aaf31723bab
Successfully built xlearn
tensorflow-gpu 1.7.0 requires numpy>=1.13.3, which is not installed.
h5py 2.7.1 requires numpy>=1.7, which is not installed.
tensorflow-tensorboard 1.5.0 requires numpy>=1.12.0, which is not installed.
torchvision 0.2.1 requires numpy, which is not installed.
pandas 0.22.0 requires numpy>=1.9.0, which is not installed.
patsy 0.5.0 requires numpy>=1.4, which is not installed.
matplotlib 2.1.2 requires numpy>=1.7.1, which is not installed.
keras 2.1.3 requires numpy>=1.9.1, which is not installed.
opencv-python 3.4.0.12 requires numpy>=1.11.1, which is not installed.
tensorflow 1.5.0 requires numpy>=1.12.1, which is not installed.
tensorboard 1.7.0 requires numpy>=1.12.0, which is not installed.
scipy 1.0.0 requires numpy>=1.8.2, which is not installed.
Installing collected packages: xlearn
Successfully installed xlearn-0.31a1显示有下面的库没有安装或者需要更新：
tensorflow-gpu 1.7.0 requires numpy>=1.13.3, which is not installed.
h5py 2.7.1 requires numpy>=1.7, which is not installed.
tensorflow-tensorboard 1.5.0 requires numpy>=1.12.0, which is not installed.
torchvision 0.2.1 requires numpy, which is not installed.
pandas 0.22.0 requires numpy>=1.9.0, which is not installed.
patsy 0.5.0 requires numpy>=1.4, which is not installed.
matplotlib 2.1.2 requires numpy>=1.7.1, which is not installed.
keras 2.1.3 requires numpy>=1.9.1, which is not installed.
opencv-python 3.4.0.12 requires numpy>=1.11.1, which is not installed.
tensorflow 1.5.0 requires numpy>=1.12.1, which is not installed.
tensorboard 1.7.0 requires numpy>=1.12.0, which is not installed.
scipy 1.0.0 requires numpy>=1.8.2, which is not installed.这里显示的tensorflow-gpu等库依赖的numpy版本过低，需要更新一下numpy：
yuhuiliu@sinclab-desktop:~$ sudo pip install tensorflow-gpu h5py numpy scipy scikit-learn -U
yuhuiliu@sinclab-desktop:~$ sudo pip install tensorflow-gpu h5py numpy scipy scikit-learn -U
Collecting tensorflow-gpu
Downloading https://files.pythonhosted.org/packages/f2/fa/01883fee1cdb4682bbd188edc26da5982c459e681543bb7f99299fca8800/tensorflow_gpu-1.8.0-cp35-cp35m-manylinux1_x86_64.whl (216.3MB)
100% |████████████████████████████████| 216.3MB 219kB/s
Requirement already up-to-date: h5py in /usr/local/lib/python3.5/dist-packages (2.7.1)
Requirement already up-to-date: numpy in /usr/local/lib/python3.5/dist-packages (1.14.3)
Requirement already up-to-date: scipy in /usr/local/lib/python3.5/dist-packages (1.1.0)
Requirement already up-to-date: scikit-learn in /usr/local/lib/python3.5/dist-packages (0.19.1)
Requirement not upgraded as not directly required: wheel>=0.26 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (0.31.0)
Requirement not upgraded as not directly required: termcolor>=1.1.0 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (1.1.0)
Requirement not upgraded as not directly required: six>=1.10.0 in /usr/lib/python3/dist-packages (from tensorflow-gpu) (1.10.0)
Requirement not upgraded as not directly required: astor>=0.6.0 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (0.6.2)
Requirement not upgraded as not directly required: protobuf>=3.4.0 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (3.5.1)
Collecting tensorboard<1.9.0,>=1.8.0 (from tensorflow-gpu)
Downloading https://files.pythonhosted.org/packages/59/a6/0ae6092b7542cfedba6b2a1c9b8dceaf278238c39484f3ba03b03f07803c/tensorboard-1.8.0-py3-none-any.whl (3.1MB)
100% |████████████████████████████████| 3.1MB 1.8MB/s
Requirement not upgraded as not directly required: gast>=0.2.0 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (0.2.0)
Requirement not upgraded as not directly required: grpcio>=1.8.6 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (1.11.0)
Requirement not upgraded as not directly required: absl-py>=0.1.6 in /usr/local/lib/python3.5/dist-packages (from tensorflow-gpu) (0.1.9)
Requirement not upgraded as not directly required: setuptools in /usr/local/lib/python3.5/dist-packages (from protobuf>=3.4.0->tensorflow-gpu) (39.1.0)
Requirement not upgraded as not directly required: werkzeug>=0.11.10 in /usr/local/lib/python3.5/dist-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu) (0.14.1)
Requirement not upgraded as not directly required: markdown>=2.6.8 in /usr/local/lib/python3.5/dist-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu) (2.6.11)
Requirement not upgraded as not directly required: bleach==1.5.0 in /usr/local/lib/python3.5/dist-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu) (1.5.0)
Requirement not upgraded as not directly required: html5lib==0.9999999 in /usr/local/lib/python3.5/dist-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu) (0.9999999)
Installing collected packages: tensorboard, tensorflow-gpu
Found existing installation: tensorboard 1.7.0
Uninstalling tensorboard-1.7.0:
Successfully uninstalled tensorboard-1.7.0
Found existing installation: tensorflow-gpu 1.7.0
Uninstalling tensorflow-gpu-1.7.0:
Successfully uninstalled tensorflow-gpu-1.7.0
Successfully installed tensorboard-1.8.0 tensorflow-gpu-1.8.0
显示xlearn成功安装：
yuhuiliu@sinclab-desktop:~$ sudo pip install xlearn
[sudo] password for yuhuiliu:
Requirement already satisfied: xlearn in /usr/local/lib/python3.5/dist-packages (0.31a1)
yuhuiliu@sinclab-desktop:~$ sudo pip list |grep xle
xlearn 0.31a1
三、使用conda创建新的环境
在上面可以看见使用pip install了xlearn，但是只能在系统自带的python环境下使用，如下图使用conda list|grep xlear 命令查看xlearn库的信息，就发现没有该库的信息：
yuhuiliu@sinclab-desktop:~$ conda list |grep xlea
yuhuiliu@sinclab-desktop:~$参照https://www.zhihu.com/question/58033789对conda这一工具的介绍和https://www.jianshu.com/p/7e4c29a26f29的解决方法，这时为了预防配置第三方库不慎导致的崩溃，一般创建新的环境进行配置。
1、先列出当前的环境列表：
yuhuiliu@sinclab-desktop:~$ conda env list
# conda environments:
#
base * /home/yuhuiliu/anaconda3
yuhuiliu@sinclab-desktop:~$ conda create -n ffm_baseline python=3
Solving environment: done
## Package Plan ##
environment location: /home/yuhuiliu/anaconda3/envs/ffm_baseline
added / updated specs:
- python=3
The following NEW packages will be INSTALLED:
ca-certificates: 2018.03.07-0 defaults
certifi: 2018.4.16-py36_0 defaults
libedit: 3.1.20170329-h6b74fdf_2 defaults
libffi: 3.2.1-hd88cf55_4 defaults
libgcc-ng: 7.2.0-hdf63c60_3 defaults
libstdcxx-ng: 7.2.0-hdf63c60_3 defaults
ncurses: 6.1-hf484d3e_0 defaults
openssl: 1.0.2o-h20670df_0 defaults
pip: 10.0.1-py36_0 defaults
python: 3.6.5-hc3d631a_2 defaults
readline: 7.0-ha6073c6_4 defaults
setuptools: 39.1.0-py36_0 defaults
sqlite: 3.23.1-he433501_0 defaults
tk: 8.6.7-hc745277_3 defaults
wheel: 0.31.0-py36_0 defaults
xz: 5.2.3-h5e939de_4 defaults
zlib: 1.2.11-ha838bed_2 defaults
Proceed ([y]/n)? Y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate ffm_baseline
#
# To deactivate an active environment, use:
# > source deactivate
#
yuhuiliu@sinclab-desktop:~$3、此时使用conda list命令查看如下，会发现多了一个ffm_baseline的环境：
yuhuiliu@sinclab-desktop:~$ conda env list
# conda environments:
#
base * /home/yuhuiliu/anaconda3
ffm_baseline /home/yuhuiliu/anaconda3/envs/ffm_baseline
4、source activate ffm_baseline一下进入ffm_baseline环境，使用conda list查看该环境的库：
yuhuiliu@sinclab-desktop:~$ source activate ffm_baseline
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ conda list
# packages in environment at /home/yuhuiliu/anaconda3/envs/ffm_baseline:
#
# Name Version Build Channel
ca-certificates 2018.03.07 0 defaults
certifi 2018.4.16 py36_0 defaults
libedit 3.1.20170329 h6b74fdf_2 defaults
libffi 3.2.1 hd88cf55_4 defaults
libgcc-ng 7.2.0 hdf63c60_3 defaults
libstdcxx-ng 7.2.0 hdf63c60_3 defaults
ncurses 6.1 hf484d3e_0 defaults
openssl 1.0.2o h20670df_0 defaults
pip 10.0.1 py36_0 defaults
python 3.6.5 hc3d631a_2 defaults
readline 7.0 ha6073c6_4 defaults
setuptools 39.1.0 py36_0 defaults
sqlite 3.23.1 he433501_0 defaults
tk 8.6.7 hc745277_3 defaults
wheel 0.31.0 py36_0 defaults
xz 5.2.3 h5e939de_4 defaults
zlib 1.2.11 ha838bed_2 defaults
四、第三方库xlearn在conda环境下的使用
在网上搜索之后，发现的原因是anaconda在托管了系统的python环境后，pip命令默认指向的是系统原始python环境里面的pip命令，因此使用pip install xlearn只能安装在系统环境的pip list中；而如果pip命令指向的是anconda托管环境中的pip命令时，可以pip install xlearn到conda的list中，作为对比，在另外一台电脑sinc-server上试验如下：
yuhuiliu@sinc-server:~$ pip install xlearn
Collecting xlearn
Downloading https://files.pythonhosted.org/packages/1a/20/d2762ecfd0da63bf2f0ee95429c7cf8ad44ab8ad4adc48b405fa67a09848/xlearn-0.31a1.tar.gz (1.8MB)
100% |████████████████████████████████| 1.9MB 620kB/s
Building wheels for collected packages: xlearn
Running setup.py bdist_wheel for xlearn ... done
Stored in directory: /home/yuhuiliu/.cache/pip/wheels/9c/46/1a/e7682af4ef3320ad6e106c633aea0ee46ffb353aaf31723bab
Successfully built xlearn
Installing collected packages: xlearn
Successfully installed xlearn-0.31a1
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
yuhuiliu@sinc-server:~$ conda list |grep xlearn
xlearn 0.31a1 yuhuiliu@sinc-server:~$ python
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xlearn
>>>这时，参考网上的说明，如果想在conda的环境中使用第三方库xlearn的话，附录网上的几种建议如下：
1、参照https://blog.csdn.net/sinat_39338078/article/details/78749724对第三方库xlearn进行安装：
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ sudo pip install xlearn
[sudo] password for yuhuiliu:
Requirement already satisfied: xlearn in /usr/local/lib/python3.5/dist-packages (0.31a1)
将原始安装于/usr/local/lib/python3.5/dist-packages/的库文件，
＃此处的文件大都是用pip安装的，也就是刚刚安装的xlearn也位于此处，但是还有一些依赖项＃
将这个文件夹下的所有文件复制到 对应的~/anaconda3/lib/python3.5/site-packages/目录下，有重复的选择替换
经过查看：
yuhuiliu@sinclab-desktop:~$ ls ~/anaconda3/lib/p
pkgconfig/ python3.6/
anaconda3/lib的目录下面环境为python3.6的库，强行合并python3.5下的库到python3.6的库下面感觉有点不靠谱~
2 、参照https://segmentfault.com/q/1010000012539647和https://www.zhihu.com/question/41974592中“史密斯”同学的解答，再根据上面的https://www.jianshu.com/p/7e4c29a26f29，先sudo pip uninstall xlearn卸载掉xlearn，直接执行/home/yuhuiliu/anaconda3/bin下的pip命令：
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ sudo pip uninstall xlearn
[sudo] password for yuhuiliu:
Uninstalling xlearn-0.31a1:
Would remove:
/usr/local/lib/python3.5/dist-packages/xlearn-0.31a1.dist-info/*
/usr/local/lib/python3.5/dist-packages/xlearn/*
Proceed (y/n)? Y
Successfully uninstalled xlearn-0.31a1
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ sudo /home/yuhuiliu/anaconda3/bin/pip install xlearn
Collecting xlearn
Using cached https://files.pythonhosted.org/packages/1a/20/d2762ecfd0da63bf2f0ee95429c7cf8ad44ab8ad4adc48b405fa67a09848/xlearn-0.31a1.tar.gz
Building wheels for collected packages: xlearn
Running setup.py bdist_wheel for xlearn ... done
Stored in directory: /home/yuhuiliu/.cache/pip/wheels/9c/46/1a/e7682af4ef3320ad6e106c633aea0ee46ffb353aaf31723bab
Successfully built xlearn
distributed 1.21.8 requires msgpack, which is not installed.
Installing collected packages: xlearn
Successfully installed xlearn-0.31a1
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ conda list |grep xlear
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ (ffm_baseline) yuhuiliu@sinclab-desktop:~$ conda env list
# conda environments:
#
base /home/yuhuiliu/anaconda3
ffm_baseline * /home/yuhuiliu/anaconda3/envs/ffm_baseline
(ffm_baseline) yuhuiliu@sinclab-desktop:~$ source activate base
(base) yuhuiliu@sinclab-desktop:~$ conda list |grep xlear
xlearn 0.31a1
(base) yuhuiliu@sinclab-desktop:~$ python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import xlearn as xl
>>> xl.hello()
----------------------------------------------------------------------------------------------
_
| |
__ _| | ___ __ _ _ __ _ __
\ \/ / | / _ \/ _` | '__| '_ \
> <| |___| __/ (_| | | | | | |
/_/\_\_____/\___|\__,_|_| |_| |_|
xLearn -- 0.31 Version --
----------------------------------------------------------------------------------------------
>>>
到这里，xlearn可以在conda创建的base环境下使用了。
3、解决xlearn的依赖问题：
(base) yuhuiliu@sinclab-desktop:~$ sudo /home/yuhuiliu/anaconda3/bin/pip install msgpack
[sudo] password for yuhuiliu:
Collecting msgpack
Downloading https://files.pythonhosted.org/packages/22/4e/dcf124fd97e5f5611123d6ad9f40ffd6eb979d1efdc1049e28a795672fcd/msgpack-0.5.6-cp36-cp36m-manylinux1_x86_64.whl (315kB)
100% |████████████████████████████████| 317kB 11kB/s
Installing collected packages: msgpack
Successfully installed msgpack-0.5.6
(base) yuhuiliu@sinclab-desktop:~$