Storm On Yarn 安装部署

1. 安装 JDK7 和 Maven

2. 部署Hadoop2集群，并启动yarn
http://my.oschina.net/zc741520/blog/362824

3. 下载 Storm on Yarn
[grid@hadoop4 ~]$ wget https://github.com/yahoo/storm-yarn/archive/master.zip

4. 编译

[grid@hadoop4 ~]$ unzip master.zip
[grid@hadoop4 ~]$ cd storm-yarn-master
## 修改 pom.xml，将Hadoop的版本号改成对应的版本号
[grid@hadoop4 storm-yarn-master]$ vim pom.xml
    <properties>
        <storm.version>0.9.0-wip21</storm.version>
        <hadoop.version>2.5.2</hadoop.version>
        <!--hadoop.version>2.1.0.2.0.5.0-67</hadoop.version-->
    </properties>
## 编译
[grid@hadoop4 storm-yarn-master]$ mvn package -DskipTests

5. storm-yarn-master/lib/storm-0.9.0-wip21.zip 解压到上层目录storm-yarn-master中 

[grid@hadoop4 storm-yarn-master]$ cd lib
[grid@hadoop4 lib]$ unzip storm-0.9.0-wip21.zip -d ..
[grid@hadoop4 storm-yarn-master]$ ls
bin  CLA.pdf  create-tarball.sh  lib  LICENSE.txt  pom.xml  README.md  src  storm-0.9.0-wip21  target

6. 在HDFS中创建对应Storm目录
[grid@hadoop4 storm-yarn-master]$ hadoop fs -mkdir -p /lib/storm/0.9.0-wip21

7. 将storm.zip放到HDFS
##  根据实际需要，添加Storm工程需要的额外Jar包到storm-0.9.0-wip21的lib下，重新压缩成storm.zip文件，上传至HDFS的指定目录中（非常重要，集群中通过访问hdfs中的storm.zip获取工作环境）

[grid@hadoop4 storm-yarn-master]$ hadoop fs -put ./lib/storm.zip /lib/storm/0.9.0-wip21
[grid@hadoop4 storm-yarn-master]$ hadoop fs -ls /lib/storm/0.9.0-wip21
Found 1 items
-rw-r--r--   2 grid supergroup   17141078 2015-05-24 19:43 /lib/storm/0.9.0-wip21/storm.zip

8. 在安装Hadoop时已经设置好了hadoop的一些环境变量，现在再增加如下环境变量 

[grid@hadoop4 storm-yarn-master]$ vim ~/.bash_profile 
export PATH=$PATH:/home/grid/storm-yarn-master/storm-0.9.0-wip21/bin:/home/grid/storm-yarn-master/bin
[grid@hadoop4 storm-yarn-master]$ source ~/.bash_profile

9. 修改 storm-yarn-master/storm-0.9.0-wip21/conf/storm.yaml 配置文件，增加zookeeper的配置 

## zookeeper集群的安装与启动，在介绍Storm集群安装（ http://my.oschina.net/zc741520/blog/399233）时已经有过详细说明，这里就不再赘述

[grid@hadoop4 storm-yarn-master]$ vim storm-0.9.0-wip21/conf/storm.yaml 
storm.zookeeper.servers:
    - "hadoop4"
    - "hadoop5"
    - "hadoop6"

master.initial-num-supervisors: 1
master.container.size-mb: 1024

10. 启动 storm on yarn 环境
[grid@hadoop4 storm-yarn-master]$ storm-yarn launch storm-0.9.0-wip21/conf/storm.yaml

因为storm是作为一个yarn程序运行在集群上的，所以在YARN的集群管理页面中会有一个AppId
PS：第一次启动时失败了，最终发现是内存不足导致的，解决办法是在yarn-site.xml中设置yarn.nodemanager.vmem-check-enabled的值为false



 11. 查找nimbus节点

[grid@hadoop4 storm-yarn-master]$ storm-yarn getStormConfig -appId application_1432484548277_0001 -output ~/.storm/storm.yaml
[grid@hadoop4 storm-yarn-master]$ cat ~/.storm/storm.yaml | grep nimbus.host
nimbus.host: 192.168.0.107

12. 提交Topology，这里用Storm自带的WordCount例子程序

[grid@hadoop4 storm-yarn-master]$ storm jar lib/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology WordCountTopology -c nimbus.host=192.168.0.107

 13. 查看storm的UI监控界面（nimbus.host:7070）