XLearning 安装说明文档

一、版本说明

hadoop 2.7.3
TensorFlow 1.1.0
xlearning 1.1
java 1.8.0
pyton 2.7.5

二、系统环境安装

1. 安装java

下载jdk-8u73-linux-x64.tar.gz包，解压，并放置到指定目录（如:/opt/XLearning/software目录）下

tar -xzvf jdk-7u67-linux-x64.tar.gz -C /opt/XLearning/software

创建软链接

cd /opt/XLearning/software
ln -s jdk1.7.0_67 java

配置环境变量
vim /etc/profile.d/jdk.sh

    export JAVA_HOME=/opt/XLearning/software/jdk1.7.0_67
    export JRE_HOME=${JAVA_HOME}/jre
    export CLASSPATH=$CLASSPATH:${JAVA_HOME}/lib:${JRE_HOME}/lib
    export PATH=$PATH:${JAVA_HOME}/bin

环境变量生效

source /etc/profile

2.安装pip

下载pip-9.0.1.tar.gz解压，并安装

tar -xzvf pip-9.0.1.tar.gz
cd pip-9.0.1
python setup.py install

三、Hadoop部署

1.下载、解压

tar -xzvf hadoop-2.7.3.tar.gz
cd hadoop-2.7.3/etc/hadoop

2.修改配置文件

1. hadoop-env.sh文件

   vim hadoop-env.sh 
   JAVA_HOME = /opt/XLearning/software/java
   

2. yarn-env.sh文件

   vim yarn-env.sh
   JAVA_HOME = /opt/XLearning/software/java
   

3. mapred-env.sh文件

   vim mapred-env.sh
   JAVA_HOME = /opt/XLearning/software/java
   

4. core-site.xml文件

   vim core-site.xml
   

   修改_HOST，替换为主机名，如test-1
5. hdfs-site.xml文件

   vim hdfs-site.xml
   

   修改dfs.namenode.name.dir和dfs.datanode.data.dir属性

   
      dfs.namenode.name.dir
      file:/opt/XLearning/hadoop/data/dfs/name
      true
    
    
      dfs.datanode.data.dir
      file:/opt/XLearning/hadoop/data/dfs/data
      true
    
   

   创建目录

   mkdir /opt/XLearning/hadoop/data/dfs/name
   

6. mapred-site.xml文件

   mv mapred-site.xml.template mapred-site.xml
   vim mapred-site.xml
   

   修改_HOST，替换为主机名，如test-1
7. yarn-site.xml文件

   vim yarn-site.xml
   

   修改_HOST，替换为主机名，如test-1

   3. 配置HADOOP_HOME

   创建软链接

   cd /opt/XLearning/software
   ln -s hadoop-2.7.3 hadoop
   

   设置环境变量

   vim /etc/profile.d/hadoop.sh
   export HADOOP_HOME=/opt/XLearning/software/hadoop
   export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
   

   4.启动Hadoop各项服务

   格式化主节点命名空间

   cd hadoop
   ./bin/hadoop namenode -format
   

   启动存储服务和资源管理服务

   ./sbin/hadoop-daemon.sh start namenode
   ./sbin/hadoop-daemon.sh start datanode
   ./sbin/yarn-daemon.sh start resourcemanager
   ./sbin/yarn-daemon.sh start nodemanager
   ./sbin/mr-jobhistory-daemon.sh start historyserver
   

   查看各项服务的状态

   jps -ml
   

   四、安装TensorFlow

   利用pip安装

   pip install tensorflow-1.1.0-cp27-cp27mu-linux_x86_64.whl
   

   Python下验证

   >>>import tensorflow as tf
   >>>tf.__version__
   

   五、XLearning部署

   1. 编译、打包

   mvn package
   

   2. 解压

   tar -xzvf xlearning-1.1-dist.tar.gz
   

   3. 配置

   cd xlearning-1.1
   cd conf
   

   1 log4j.properties文件
   去掉注释

   # Settings the HistoryServer logs
   log4j.logger.net.qihoo.xlearning.jobhistory=DEBUG,RFA
   log4j.additivity.net.qihoo.xlearning.jobhistory=false
   log4j.appender.RFA=org.apache.log4j.RollingFileAppender
   log4j.appender.RFA.File=/tmp/XLearning/logs/XLearningHistoryServer.log
   log4j.appender.RFA.Encoding=UTF-8
   log4j.appender.RFA.Append=true
   log4j.appender.RFA.MaxFileSize=100MB
   log4j.appender.RFA.MaxBackupIndex=5
   log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
   

   2 xlearning-env.sh文件
   配置JAVA_HOME HADOOP_CONF_DIR

   export JAVA_HOME=/opt/XLearning/software/java
   export HADOOP_CONF_DIR=/opt/XLearning/software/hadoop/etc/hadoop
   

   3 xlearning-site.xml
   配置JobHistory信息，修改0.0.0.0为主机名，如test-1

   4. 启动XLearning JobHistoryServer服务

   cd xlearning-1.1
   sbin/start-history-server.sh
   

   5. XLearning客户端

   配置XLEARNING_HOME

   export XLEARNING_HOME=/opt/XLearning/xlearning-1.1
   

   创建目录

   hadoop fs -mkdir -p /tmp/XLearning/history
   hadoop fs -mkdir -p /tmp/XLearning/eventLog
   hadoop fs -mkdir -p /tmp/XLearning/staging
   

   上传数据到HDFS

   hadoop fs -put data /tmp/
   

   执行示例

   cd examples/tensorflow
   sh run.sh
   

   运行结果
   
   

更多精彩文章，详见 红象云腾社区