- .通过学习tutorial了解storm的整体架构(https://github.com/nathanmarz/storm/wiki/Tutorial)
- 通过学习Concepts了解storm的关键概念(https://github.com/nathanmarz/storm/wiki/Concepts)
- 通过学习Setting-up-a-Storm-cluster实际搭建一个storm cluster(https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
- 通过学习Running-topologies-on-a-production-cluster实际创建一个project并运行在storm cluster上(https://github.com/nathanmarz/storm/wiki/Running-topologies-on-a-production-cluster)
开始实战,确定你已经安装所有需要的软件:
====================
启动zookeeper
shawny@localhost:~/software/zookeeper-3.3.3/bin$ ./zkServer.sh start
JMX enabled by default
Using config: /home/shawny/software/zookeeper-3.3.3/bin/../conf/zoo.cfg
Starting zookeeper ...
STARTED
===========
启动storm nimbus (如果启动失败则查看zookeeper上的storm znode是否没有清除干净,如果非空则手动删除storm znode下面的所有子节点)
shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm nimbus
Running: java -server -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xmx1024m -Dlogfile.name=nimbus.log -Dlog4j.configuration=storm.log.properties -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/log4j:/home/shawny/software/storm-0.5.5-SNAPSHOT/conf backtype.storm.daemon.nimbus
===========
启动storm supervisor
shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm supervisor
Running: java -server -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xmx1024m -Dlogfile.name=supervisor.log -Dlog4j.configuration=storm.log.properties -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/log4j:/home/shawny/software/storm-0.5.5-SNAPSHOT/conf backtype.storm.daemon.supervisor
==========
启动storm UI
shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ bin/storm ui
Running: java -server -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xmx768m -Dlogfile.name=ui.log -Dlog4j.configuration=storm.log.properties -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/log4j:/home/shawny/software/storm-0.5.5-SNAPSHOT:/home/shawny/software/storm-0.5.5-SNAPSHOT/conf backtype.storm.ui.core
=================
通过“http://localhost:8080/”查看storm信息
Cluster Summary
Nimbus uptime
Supervisors
Used slots
Free slots
Total slots
Running tasks
4m 10s
1
0
4
4
0
Topology summary
Name
Id
Status
Uptime
Num workers
Num tasks
Supervisor summary
Host
Uptime
Slots
Used slots
localhost
3m 1s
4
0
=============
创建Storm_test工程
shawny@localhost:~/workspace/java/Storm_test$ tree
.
|-- bin
| |-- RandomSentenceSpout.class
| |-- resources
| | |-- splitsentence.py
| | |-- storm.py
| | `-- storm.rb
| |-- storm.yaml
| |-- WordCountTopology.class
| |-- WordCountTopology$SplitSentence.class
| `-- WordCountTopology$WordCount.class
|-- multilang
| `-- resources
| |-- splitsentence.py
| |-- storm.py
| `-- storm.rb
|-- src
| |-- RandomSentenceSpout.java
| `-- WordCountTopology.java
`-- storm_test.jar
NOTE:
1) 程序写好后打成jar包,注意export后面的library handling选择第二个
2) ~/.storm下面需要有storm.yaml, 里面包含storm的配置信息
===========
提交jar包给storm
shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm jar ~/workspace/java/Storm_test/storm_test.jar WordCountTopology test
Running: export STORM_JAR=/home/shawny/workspace/java/Storm_test/storm_test.jar; java -client -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/workspace/java/Storm_test/storm_test.jar:/home/shawny/.storm:/home/shawny/software/storm-0.5.5-SNAPSHOT/bin WordCountTopology test
0 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
25 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar /home/shawny/workspace/java/Storm_test/storm_test.jar to assigned location: /tmp/storm/nimbus/inbox/stormjar-a37d88eb-f4c1-4fa0-8621-27c00a8beaf3.jar
472 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /tmp/storm/nimbus/inbox/stormjar-a37d88eb-f4c1-4fa0-8621-27c00a8beaf3.jar
473 [main] INFO backtype.storm.StormSubmitter - Submitting topology test in distributed mode with conf {"topology.workers":3,"topology.debug":true}
747 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: test
====================
查看storm logs
shawny@localhost:~/software/storm-0.5.5-SNAPSHOT/logs$ tree
.
|-- nimbus.log
|-- supervisor.log
|-- ui.log
|-- worker-6700.log
|-- worker-6701.log
`-- worker-6702.log
===============
kill 提交的topology (test)
shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm kill test
Running: java -client -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/.storm:/home/shawny/software/storm-0.5.5-SNAPSHOT/bin backtype.storm.command.kill_topology test
0 [main] INFO backtype.storm.command.kill-topology - Killed topology: test