storm+kestrel+zookeeper

 
环境：
2台服务器：192.168.1.166
           192.168.1.167
系统：centos 5.6
 
部署：192.168.1.166：ui,nimbus,supervisor
      192.168.1.167:supervisor,kestrel,zookeeper
 
    因为公司网站对数据的实时性要求比较高，所以最近一直在研究storm，因为9月份刚开源，所以一些资料相对来说比较说，只能从官方wiki上去查，地址： https://github.com/nathanmarz/storm/wiki
    storm 是一个分布式的、容错的实时计算系统，它被托管在 GitHub 上，遵循 Eclipse Public License 1.0。Storm是由BackType开发的实时处理系统，BackType现在已在Twitter麾下。GitHub上的最新版本是Storm 0.5.2，基本是用 Clojure 写的。
 
1、安装zookeeper
 
zookeeper集群部署方式：
tar xvzf zookeeper-3.3.3.tar.gz
cd zookeeper-3.3.3
mv conf/zoo-sample.cfg conf/zoo.cfg
vim conf/zoo.cfg
 
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/var/lib/storm/zookeeper
dataLogDir=/var/log/zookeeper
clientPort=2181
server.1=192.168.1.166:2888:3888
server.2=192.168.1.167:2888:3888
 
保存退出。
mkdir /var/lib/storm/zookeeper
vim /var/lib/storm/zookeeper/myid
本机是166，所以这里输入1
保存退出。
 
另外2台机器同样的步骤，不过myid改为2,3即可。
注意：zookeeper的集群必须是单数的机器，也就是说要么3台做，要么单台做伪集群，2台做出来的话我个人测试是有问题的，这里也不是很确定，如果有哪位知道，麻烦告知下。
 
启动2台上面的服务：
/usr/local/zookeeper-3.3.3/bin/zkServer.sh start
 
#zookeeper单机部署：
同集群相同：只是配置文件略有不同：
#vim zoo.cfg
tickTime=2000
minSessionTimeout=2000
maxSessionTimeout=20000
dataDir=/var/lib/storm/zookeeper
dataLogDir=/var/log/zookeeper
clientPort=2181
保存退出
然后直接启动服务即可：/usr/local/zookeeper-3.3.3/bin/zkServer.sh start
 
 
1、安装zeromq
 安装依赖包
 yum -y install gcc-c++ e2fsprogs.x86_64 e2fsprogs-devel.x86_64
  
tar xvzf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7
./configure
make 
make install
 
2、安装jzmq
 
tar xvzf nathanmarz-jzmq-dd3327d.tar.gz
cd tar xvzf nathanmarz-jzmq-dd3327d
yum install pkgconfig libtool.x86_64
./autogen.sh
./configure
make
make install
 
 
3、安装python 2.6.6
 
tar jxvf Python-2.6.6.tar.bz2
./configure --bindir=/usr/bin --libdir=/usr/lib
make 
make install
 
4、安装kestrel
安装kestrel需要安装daemon
tar xvzf daemon-0.6.4.tar.gz
cd daemon-0.6.4
./config
make
make test
make install
make install-daemon-conf
make install-slack
 
tar xvzf kestrel-2.1.3.zip
mv kestrel-2.1.3 /usr/local/kestrel
vim /usr/local/kestrel/scripts/kestrel.sh
修改APP_HOME="/usr/local/$APP_NAME/current"为
APP_HOME="/usr/local/$APP_NAME"
保存
cp -rp /usr/local/kestrel/scripts/kestrel.sh /etc/init.d/kestrel
service kestrel start
 
5、安装storm
unzip storm-0.6.0.zip
cd storm-0.6.0
cd conf
vim strom.yaml
内容如下：
============================================================================
java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"
 
### storm.* configs are general configurations
# the local dir is where jars are kept
 storm.local.dir: "/var/lib/storm/data"
 storm.zookeeper.servers:
     - "192.168.1.166"
 storm.zookeeper.port: 2181
 storm.zookeeper.root: "/var/lib/storm/storm"
 storm.zookeeper.session.timeout: 20000
 storm.cluster.mode: "distributed" # can be distributed or local
 storm.local.mode.zmq: false
 
### nimbus.* configs are for the master
 nimbus.host: "192.168.1.166"
 nimbus.thrift.port: 6627
 nimbus.childopts: "-Xmx2048m"
 nimbus.task.timeout.secs: 30
 nimbus.supervisor.timeout.secs: 60
 nimbus.monitor.freq.secs: 10
 nimbus.task.launch.secs: 240
 nimbus.reassign: true
 nimbus.file.copy.expiration.secs: 600
 
 ui.port: 8080
 
 drpc.port: 3772
 
 
 supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703
 supervisor.childopts: "-Xmx2048m"
#how long supervisor will wait to ensure that a worker process is started
 supervisor.worker.start.timeout.secs: 240
#how long between heartbeats until supervisor considers that worker dead and tries to restart it
 supervisor.worker.timeout.secs: 30
#how frequently the supervisor checks on the status of the processes it's monitoring and restarts if necessary
 supervisor.monitor.frequency.secs: 3
#how frequently the supervisor heartbeats to the cluster state (for nimbus)
 supervisor.heartbeat.frequency.secs: 5
 supervisor.enable: true
 
### worker.* configs are for task workers
 worker.childopts: "-Xmx768m"
 worker.heartbeat.frequency.secs: 1
 
 task.heartbeat.frequency.secs: 3
 task.refresh.poll.secs: 10
 
 zmq.threads: 1
 zmq.linger.millis: 5000
 
 =======================================================================================================
日志路径修改：vim log4j/storm.log.properties
 修改：log4j.appender.A1.File = logs/${logfile.name}   为log4j.appender.A1.File = /var/log/${logfile.name}
 
 保存退出。启动服务。
 nimbus：nohup /opt/storm/storm-0.6.0/bin/storm nimbus &
         nohup /opt/storm/storm-0.6.0/bin/storm ui &
nohup /opt/storm/storm-0.6.0/bin/storm supervisor &
 supervisor:nohup /opt/storm/storm-0.6.0/bin/storm supervisor &
 
 ui访问：http://192.168.1.166:8080
 
说明：storm0.5.4的集群部署，我这边测试总是有问题，具体原因未知，也不知道是否为bug，不过6.0的总算成功，有兴趣的可以试试。