Storm on yarn 安装
1：下载
[jifeng@feng02 storm]$ wget https://github.com/yahoo/storm-yarn/archive/master.zip
--2015-03-08 21:07:24-- https://github.com/yahoo/storm-yarn/archive/master.zip
正在解析主机 github.com... 192.30.252.130
正在连接 github.com|192.30.252.130|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://codeload.github.com/yahoo/storm-yarn/zip/master [跟随至新的 URL]
--2015-03-08 21:07:27-- https://codeload.github.com/yahoo/storm-yarn/zip/master
正在解析主机 codeload.github.com... 192.30.252.145
正在连接 codeload.github.com|192.30.252.145|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：未指定 [application/zip]
正在保存至: “master”
[ <=> ] 17,346,738 288K/s in 2m 0s
2015-03-08 21:09:30 (141 KB/s) - “master” 已保存 [17346738]
2：解压
[jifeng@feng02 storm]$ mv master storm-yarn.zip
[jifeng@feng02 storm]$ unzip storm-yarn.zip
Archive: storm-yarn.zip
e570e9665bac68396db09a9711875948cc63540e
creating: storm-yarn-master/
inflating: storm-yarn-master/CLA.pdf
inflating: storm-yarn-master/LICENSE.txt
inflating: storm-yarn-master/README.md
creating: storm-yarn-master/bin/
inflating: storm-yarn-master/bin/read-link
inflating: storm-yarn-master/bin/storm-yarn
inflating: storm-yarn-master/create-tarball.sh
creating: storm-yarn-master/lib/
inflating: storm-yarn-master/lib/storm-0.9.0-wip21.zip
inflating: storm-yarn-master/lib/storm-starter-0.0.1-SNAPSHOT.jar
linking: storm-yarn-master/lib/storm.zip -> storm-0.9.0-wip21.zip
inflating: storm-yarn-master/pom.xml
creating: storm-yarn-master/src/
creating: storm-yarn-master/src/main/
inflating: storm-yarn-master/src/main/genthrift.sh
creating: storm-yarn-master/src/main/java/
creating: storm-yarn-master/src/main/java/com/
creating: storm-yarn-master/src/main/java/com/yahoo/
creating: storm-yarn-master/src/main/java/com/yahoo/storm/
creating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Client.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Config.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/LaunchCommand.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/MasterClient.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/MasterServer.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormAMRMClient.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormMasterCommand.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormMasterServerHandler.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/StormOnYarn.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Util.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/Version.java
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/VersionCommand.java
creating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/generated/
inflating: storm-yarn-master/src/main/java/com/yahoo/storm/yarn/generated/StormMaster.java
creating: storm-yarn-master/src/main/resources/
inflating: storm-yarn-master/src/main/resources/logback.xml
inflating: storm-yarn-master/src/main/resources/master_defaults.yaml
inflating: storm-yarn-master/src/main/storm_master.thrift
creating: storm-yarn-master/src/test/
creating: storm-yarn-master/src/test/java/
creating: storm-yarn-master/src/test/java/com/
creating: storm-yarn-master/src/test/java/com/yahoo/
creating: storm-yarn-master/src/test/java/com/yahoo/storm/
creating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/EmbeddedZKServer.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestConfig.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestIntegration.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestStormCluster.java
inflating: storm-yarn-master/src/test/java/com/yahoo/storm/yarn/TestStormMaster.java
creating: storm-yarn-master/src/test/resources/
inflating: storm-yarn-master/src/test/resources/log4j.properties
inflating: storm-yarn-master/src/test/resources/logback-test.xml
finishing deferred symbolic links:
storm-yarn-master/lib/storm.zip -> storm-0.9.0-wip21.zip
[jifeng@feng02 storm]$ ls
storm-yarn-master storm-yarn.zip
[jifeng@feng02 storm]$ cd storm-yarn-master/
[jifeng@feng02 storm-yarn-master]$ ls
bin CLA.pdf create-tarball.sh lib LICENSE.txt pom.xml README.md src
3：编译
[jifeng@feng02 storm-yarn-master]$ mvn package -DskipTests [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building storm-yarn 1.0-alpha [INFO] ------------------------------------------------------------------------ Downloading: http://repo1.maven.org/maven2/storm/storm-core/0.9.0-wip21/storm-core-0.9.0-wip21.jar Downloading: http://repo1.maven.org/maven2/org/clojure/clojure/1.4.0/clojure-1.4.0.jar Downloading: http://repo1.maven.org/maven2/storm/libthrift7/0.7.0-2/libthrift7-0.7.0-2.jar Downloading: http://repo1.maven.org/maven2/clj-time/clj-time/0.4.1/clj-time-0.4.1.jar Downloading: http://repo1.maven.org/maven2/backtype/jzmq/2.1.0/jzmq-2.1.0.jar Downloading: http://repo1.maven.org/maven2/compojure/compojure/1.1.3/compojure-1.1.3.jar Downloading: http://repo1.maven.org/maven2/clout/clout/1.0.1/clout-1.0.1.jar Downloading: http://repo1.maven.org/maven2/ring/ring-core/1.1.5/ring-core-1.1.5.jar Downloading: http://repo1.maven.org/maven2/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar Downloading: http://repo1.maven.org/maven2/ring/ring-devel/0.3.11/ring-devel-0.3.11.jar Downloading: http://repo1.maven.org/maven2/clj-stacktrace/clj-stacktrace/0.2.2/clj-stacktrace-0.2.2.jar Downloading: http://repo1.maven.org/maven2/ring/ring-jetty-adapter/0.3.11/ring-jetty-adapter-0.3.11.jar Downloading: http://repo1.maven.org/maven2/ring/ring-servlet/0.3.11/ring-servlet-0.3.11.jar Downloading: http://repo1.maven.org/maven2/storm/carbonite/1.5.0/carbonite-1.5.0.jar Downloading: http://repo1.maven.org/maven2/storm/tools.cli/0.2.2/tools.cli-0.2.2.jar Downloading: http://repo1.maven.org/maven2/storm/jgrapht/0.8.3/jgrapht-0.8.3.jar Downloading: http://repo1.maven.org/maven2/com/google/guava/guava/13.0/guava-13.0.jar Downloading: http://repo1.maven.org/maven2/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.1.0-beta/hadoop-yarn-server-nodemanager-2.1.0-beta.jar Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar Downloading: http://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.1.0-beta/hadoop-yarn-server-nodemanager-2.1.0-beta.jar (452 KB at 46.8 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar Downloaded: http://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar (315 KB at 25.4 KB/sec) Downloading: http://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar Downloaded: http://repo1.maven.org/maven2/org/apache/avro/avro/1.5.3/avro-1.5.3.jar (258 KB at 18.0 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar (47 KB at 2.9 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.4/commons-compress-1.4.jar Downloaded: http://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.4/commons-compress-1.4.jar (234 KB at 9.4 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.1.0-beta/hadoop-mapreduce-client-core-2.1.0-beta.jar Downloaded: http://repo1.maven.org/maven2/com/google/guava/guava/13.0/guava-13.0.jar (1847 KB at 26.7 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.1.0-beta/hadoop-yarn-common-2.1.0-beta.jar Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-core/2.1.0-beta/hadoop-mapreduce-client-core-2.1.0-beta.jar (1341 KB at 15.1 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-client/2.1.0-beta/hadoop-yarn-client-2.1.0-beta.jar Downloaded: http://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar (973 KB at 7.8 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.1.0-beta/hadoop-yarn-api-2.1.0-beta.jar Downloaded: http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-common/2.1.0-beta/hadoop-yarn-common-2.1.0-beta.jar (1264 KB at 8.3 KB/sec) Downloading: http://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.3.3/zookeeper-3.3.3.jar
Downloading: https://clojars.org/repo/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar Downloaded: https://clojars.org/repo/storm/carbonite/1.5.0/carbonite-1.5.0.jar (58 KB at 8.9 KB/sec) Downloaded: https://clojars.org/repo/storm/libthrift7/0.7.0-2/libthrift7-0.7.0-2.jar (296 KB at 33.4 KB/sec) Downloaded: https://clojars.org/repo/storm/storm-netty/0.9.0-wip21/storm-netty-0.9.0-wip21.jar (42 KB at 4.7 KB/sec) Downloaded: https://clojars.org/repo/storm/jgrapht/0.8.3/jgrapht-0.8.3.jar (240 KB at 18.9 KB/sec) Downloaded: https://clojars.org/repo/storm/storm-core/0.9.0-wip21/storm-core-0.9.0-wip21.jar (5197 KB at 19.1 KB/sec) [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ storm-yarn --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ storm-yarn --- [INFO] Changes detected - recompiling the module! [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 13 source files to /home/jifeng/hadoop/storm/storm-yarn-master/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ storm-yarn --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ storm-yarn --- [INFO] Changes detected - recompiling the module! [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 5 source files to /home/jifeng/hadoop/storm/storm-yarn-master/target/test-classes [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default) @ storm-yarn --- [INFO] Building jar: /home/jifeng/hadoop/storm/storm-yarn-master/target/storm-yarn-1.0-alpha.jar [INFO] [INFO] --- maven-surefire-plugin:2.10:test (default-test) @ storm-yarn --- Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.pom Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.pom (3 KB at 0.8 KB/sec) Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.pom Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.pom (3 KB at 3.9 KB/sec) Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.pom Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.pom (4 KB at 6.8 KB/sec) Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.jar Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.jar Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.jar Downloading: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.1/plexus-utils-2.1.jar Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-booter/2.10/surefire-booter-2.10.jar (34 KB at 17.1 KB/sec) Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.10/surefire-api-2.10.jar (158 KB at 64.6 KB/sec) Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/maven-surefire-common/2.10/maven-surefire-common-2.10.jar (60 KB at 21.7 KB/sec) Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/2.1/plexus-utils-2.1.jar (220 KB at 16.2 KB/sec) [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ storm-yarn --- [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 24:18 min [INFO] Finished at: 2015-03-08T22:13:01+08:00 [INFO] Final Memory: 13M/32M [INFO] ------------------------------------------------------------------------ [jifeng@feng02 storm-yarn-master]$ ls bin CLA.pdf create-tarball.sh lib LICENSE.txt pom.xml README.md src target [jifeng@feng02 storm-yarn-master]$ cd target/ [jifeng@feng02 target]$ ls classes generated-test-sources maven-status test-classes generated-sources maven-archiver storm-yarn-1.0-alpha.jar [jifeng@feng02 target]$ cd ..
4：storm-0.9.0解压并移到storm-yarn-master 同级
[jifeng@feng02 storm-yarn-master]$ cd lib [jifeng@feng02 lib]$ unzip storm-0.9.0-wip21.zip Archive: storm-0.9.0-wip21.zip [jifeng@feng02 lib]$ mv storm-0.9.0-wip21 ../../ [jifeng@feng02 lib]$ cd .. [jifeng@feng02 storm-yarn-master]$ cd .. [jifeng@feng02 storm]$ ls storm-0.9.0-wip21 storm-yarn-master storm-yarn.zip
5 ：配置
[jifeng@feng02 storm-yarn-master]$ vi ~/.bash_profile后面加上：
export STORM_WORK=$HOME/hadoop/storm
export STORM_HOME=$STORM_WORK
export PATH=$PATH:$STORM_WORK/storm-yarn-master/bin:$STORM_WORK/storm-0.9.0-wip21/bin
[jifeng@feng02 lib]$ source ~/.bash_profile生效 6：上传storm.zip到hdfs
[jifeng@feng02 lib]$ hadoop fs -mkdir -p /lib/storm/0.9.0-wip21 [jifeng@feng02 lib]$ hadoop fs -moveFromLocal storm.zip /lib/storm/0.9.0-wip21 [jifeng@feng02 lib]$ hadoop fs -mkdir -p /user/storm [jifeng@feng02 lib]$ hadoop fs -chown storm /user/storm [jifeng@feng02 lib]$ cd .. [jifeng@feng02 storm-yarn-master]$ cd ..
7：启动
storm-yarn launch /home/jifeng/hadoop/storm/storm-0.9.0-wip21/conf/storm.yaml
[jifeng@feng02 storm-0.9.0-wip21]$ cd .. [jifeng@feng02 storm]$ storm-yarn launch /home/jifeng/hadoop/storm/storm-0.9.0-wip21/conf/storm.yaml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/03/14 17:35:23 INFO client.RMProxy: Connecting to ResourceManager at feng01/10.6.3.84:8032 15/03/14 17:35:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/14 17:35:25 INFO yarn.StormOnYarn: Copy App Master jar from local filesystem and add to local environment 15/03/14 17:35:27 INFO yarn.StormOnYarn: Set the environment for the application master 15/03/14 17:35:27 INFO yarn.StormOnYarn: YARN CLASSPATH COMMAND = [[yarn, classpath]] 15/03/14 17:35:27 INFO yarn.StormOnYarn: YARN CLASSPATH = [/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/etc/hadoop:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/hdfs/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/lib/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/mapreduce/*:/home/jifeng/hadoop/hadoop-2.4.1/contrib/capacity-scheduler/*.jar:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/*:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/yarn/lib/*] 15/03/14 17:35:27 INFO yarn.StormOnYarn: Using JAVA_HOME = [/home/jifeng/jdk1.7.0_45] 15/03/14 17:35:27 INFO yarn.StormOnYarn: Setting up app master command:[/home/jifeng/jdk1.7.0_45/bin/java, -Dstorm.home=./storm/storm-0.9.0-wip21/, -Dlogfile.name=<LOG_DIR>/master.log, com.yahoo.storm.yarn.MasterServer, 1><LOG_DIR>/stdout, 2><LOG_DIR>/stderr] 15/03/14 17:35:28 INFO impl.YarnClientImpl: Submitted application application_1425798050526_0001 application_1425798050526_0001 [jifeng@feng02 storm]$
8：关闭
[jifeng@feng02 storm]$ storm-yarn shutdown -appId application_1425798050526_0001 -output ~/hadoop/storm/storm.yaml SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/jifeng/hadoop/storm/storm-0.9.0-wip21/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/03/15 21:13:27 INFO client.RMProxy: Connecting to ResourceManager at feng01/10.6.3.84:8032 15/03/15 21:13:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/03/15 21:13:28 INFO yarn.StormOnYarn: application report for application_1425798050526_0001 :feng03:9000 15/03/15 21:13:28 INFO yarn.StormOnYarn: Attaching to feng03:9000 to talk to app master application_1425798050526_0001 15/03/15 21:13:29 INFO yarn.StormMasterCommand: org.apache.thrift7.transport.TTransportException