在领英TonY开源框架上运行tensorflow的demo

TonY : Tensorflow on YARN

1. 下载TonY官方代码

git clone https://github.com/linkedin/TonY

2. 进入代码文件夹根目录，打包TonY

./gradlew build -x test

jar文件保存在./tony-cli/build/libs/中：tony-cli-0.3.8-all.jar

3. 打包Python虚拟环境

wget https://files.pythonhosted.org/packages/33/bc/fa0b5347139cd9564f0d44ebd2b147ac97c36b2403943dbee8a25fd74012/virtualenv-16.0.0.tar.gz
tar xf virtualenv-16.0.0.tar.gz
python virtualenv-16.0.0/virtualenv.py venv
. venv/bin/activate
pip install tensorflow==1.13.1
pip install pandas==0.20.3
pip install scikit-learn==0.18.2
pip install scipy==0.19.1
pip install tensorflow-io==0.6.0
zip -r venv.zip venv

4. 编写配置文件 tony-test.xml

<configuration>
	<property>
 		 <name>tony.ps.instancesname>
  		<value>1value>
	property>
	<property>
  		<name>tony.worker.instancesname>
  		<value>2value>
	property>
	<property>
    	<name>tony.worker.memoryname>
    	<value>4gvalue>
  	property>
	<property>
    	<name>tony.ps.memoryname>
    	<value>3gvalue>
  	property>
configuration>

5. 将jar包、xml文件、venv压缩包放在同一目录下，在该目录下通过以下命令行运行tony-example中的tensorflow demo

java -cp `hadoop classpath`:tony-cli-0.3.8-all.jar com.linkedin.tony.cli.ClusterSubmitter --python_venv=venv.zip --src_dir=mnist-tensorflow --executes=mnist_distributed.py --task_params="--steps 100 --data_dir hdfs://xxx --working_dir hdfs://xxx" --conf_file=tony-test.xml --python_binary_path=venv/bin/python --shell_env=LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native:/usr/java/default/jre/lib/amd64/server

其中hdfs地址需更换为自己的数据地址