HTTPSQS &jafka & kestrel & beanstalkd

 HTTPSQS：
编译服务端
wget http://httpsqs.googlecode.com/files/libevent-2.0.12-stable.tar.gz
tar zxvf libevent-2.0.12-stable.tar.gz
cd libevent-2.0.12-stable/
./configure --prefix=/usr/local/libevent-2.0.12-stable/
make
make install
cd ../


wget http://httpsqs.googlecode.com/files/tokyocabinet-1.4.47.tar.gz
tar zxvf tokyocabinet-1.4.47.tar.gz
cd tokyocabinet-1.4.47/
./configure --prefix=/usr/local/tokyocabinet-1.4.47/
#注：在32位Linux操作系统上编译Tokyo cabinet，请使用./configure --enable-off64代替./configure，可以使数据库文件突破2GB的限制。
#./configure --enable-off64 --prefix=/usr/local/tokyocabinet-1.4.47/
make
make install
cd ../


遇到问题 configure: error: bzlib.h is required
解决办法：yum install bzip2-devel


wget http://httpsqs.googlecode.com/files/httpsqs-1.7.tar.gz
tar zxvf httpsqs-1.7.tar.gz
cd httpsqs-1.7/
make
make install
cd ../




启动
mkdir -p  /data0/queue
ulimit -SHn 65535 
httpsqs -d -p 1218 -x /data0/queue
请使用命令“killall httpsqs”、“pkill httpsqs”和“kill `cat /tmp/httpsqs.pid`”来停止httpsqs。


入队列
curl "http://10.16.27.33:1218/?name=your_queue_name&opt=put&data=myinputdata&auth=mypass123"


出队列
curl "http://10.16.27.33:1218/?charset=utf-8&name=your_queue_name&opt=get&auth=mypass123"
具体过程


[root@xen190v ~]# curl "http://10.16.27.33:1218/?name=your_queue_name&opt=put&data=myinputdata&auth=mypass123"
HTTPSQS_PUT_OK[root@xen190v ~]# 
[root@xen190v ~]# 
[root@xen190v ~]# curl "http://10.16.27.33:1218/?charset=utf-8&name=your_queue_name&opt=get&auth=mypass123"
myinputdata[root@xen190v ~]# 


查看队列状态
curl "http://10.16.27.33:1218/?name=your_queue_name&opt=status&auth=mypass123"
json方式
[root@xen190v ~]# curl "http://10.16.27.33:1218/?name=your_queue_name&opt=status_json&auth=mypass123"
{"name":"your_queue_name","maxqueue":1000000,"putpos":2,"putlap":1,"getpos":2,"getlap":1,"unread":0}
 


 php客户端类
 见 httpsqs_client.php
 test_client.php
##################################################################################
 jafka
 wget https://github.com/downloads/adyliu/jafka/jafka-1.0.tgz 


tar xzf jafka-1.0.tgz 


cd jafka-1.0 
启动 
[zengwenjie@xen189v ~/jafka-1.0]$ bash bin/server.sh conf/server.properties 
log4j:WARN No appenders could be found for logger (com.sohu.jafka.server.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
原因 log4j没有初始化
[zengwenjie@xen189v ~/jafka-1.0/conf]$ cp log4j.properties.in log4j.properties


[2013-01-30 11:29:42,408] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn$SendThread.java:1161)
java.net.ConnectException: Connection refused
原因：要先启动zookeeper


 bin/zookeeper-server.sh conf/zookeeper.properties 
bash bin/server.sh conf/server.properties
启动成功


生成消息
[root@xen189v ~/jafka-1.0]# bin/producer-console.sh --zookeeper localhost:2181 --topic demo
[2013-01-30 11:34:10,696] INFO Starting ZkClient event thread. (com.github.zkclient.ZkEventThread.java:65)
[2013-01-30 11:34:10,755] INFO Opening socket connection to server localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn$SendThread.java:1041)
[2013-01-30 11:34:10,769] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn$SendThread.java:949)
[2013-01-30 11:34:10,789] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13c89829f810001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn$SendThread.java:738)
[2013-01-30 11:34:10,792] INFO zookeeper state changed (SyncConnected) (com.github.zkclient.ZkClient.java:450)
[2013-01-30 11:34:10,863] INFO read all brokers count: 1 (com.sohu.jafka.producer.ZKBrokerPartitionInfo.java:157)
[2013-01-30 11:34:10,874] INFO Loading Broker id:1,creatorId:127.0.0.1-1359516722591,host:127.0.0.1,port:9092 (com.sohu.jafka.producer.ZKBrokerPartitionInfo.java:162)
[2013-01-30 11:34:10,886] INFO Creating sync producer for broker id = 1 at 127.0.0.1:9092 (com.sohu.jafka.producer.ProducerPool.java:123)
Enter you message and exit with empty string.
> ddddd
[2013-01-30 11:34:13,281] INFO Connected to 127.0.0.1:9092 for producing (com.sohu.jafka.producer.SyncProducer.java:146)
> xxxxxxxx
> ddddddddd
> xxxxxxxxx
> quit


消费消息
[root@xen189v ~/jafka-1.0]# bin/consumer-console.sh --zookeeper localhost:2181 --topic demo --from-beginning
[2013-01-30 11:34:52,470] INFO Connecting to zookeeper instance at localhost:2181 (com.sohu.jafka.consumer.ZookeeperConsumerConnector.java:327)
[2013-01-30 11:34:52,499] INFO Starting ZkClient event thread. (com.github.zkclient.ZkEventThread.java:65)
[2013-01-30 11:34:52,555] INFO Opening socket connection to server localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn$SendThread.java:1041)
[2013-01-30 11:34:52,568] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn$SendThread.java:949)
[2013-01-30 11:34:52,596] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13c89829f810002, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn$SendThread.java:738)
[2013-01-30 11:34:52,607] INFO zookeeper state changed (SyncConnected) (com.github.zkclient.ZkClient.java:450)
[2013-01-30 11:34:52,607] INFO Connected to zookeeper at localhost:2181 (com.sohu.jafka.consumer.ZookeeperConsumerConnector.java:329)
[2013-01-30 11:34:52,616] INFO starting auto committer every 10000 ms (com.sohu.jafka.consumer.ZookeeperConsumerConnector.java:178)
[2013-01-30 11:34:52,629] INFO Starting ZkClient event thread. (com.github.zkclient.ZkEventThread.java:65)
[2013-01-30 11:34:52,630] INFO Opening socket connection to server localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn$SendThread.java:1041)
[2013-01-30 11:34:52,631] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn$SendThread.java:949)
[2013-01-30 11:34:52,655] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13c89829f810003, negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn$SendThread.java:738)
[2013-01-30 11:34:52,656] INFO zookeeper state changed (SyncConnected) (com.github.zkclient.ZkClient.java:450)
[2013-01-30 11:34:52,680] INFO Terminate ZkClient event thread. (com.github.zkclient.ZkEventThread.java:83)
[2013-01-30 11:34:52,719] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn$EventThread.java:520)
[2013-01-30 11:34:53,483] INFO starting watcher executor thread for consumer console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3 (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:373)
[2013-01-30 11:34:53,612] INFO register consumer in zookeeper [/consumers/console-consumer-4734/ids/console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3] => [{"demo":1}] (com.sohu.jafka.consumer.ZookeeperConsumerConnector.java:733)
[2013-01-30 11:34:53,812] INFO begin rebalancing consumer console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3 try #0 (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:401)
[2013-01-30 11:34:53,860] INFO Clearing the current data chunk for this consumer iterator (com.sohu.jafka.consumer.ConsumerIterator.java:118)
[2013-01-30 11:34:53,860] INFO Releasing partition ownership (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:624)
[2013-01-30 11:34:53,861] INFO Consumer console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3 rebalancing the following partitions: [1-0] for topic demo with consumers: [console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3-0] (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:466)
[2013-01-30 11:34:53,862] INFO console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3-0 attempting to claim partition 1-0 (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:485)
[2013-01-30 11:34:53,917] INFO Disconnecting from /127.0.0.1:9092 (com.sohu.jafka.consumer.SimpleConsumer.java:103)
[2013-01-30 11:34:54,329] INFO Updating the cache (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:501)
[2013-01-30 11:34:54,333] INFO end rebalancing consumer console-consumer-4734_xen189v.ops.corp.qihoo.net-1359516892891-919e3fb3 try #0 (com.sohu.jafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.java:417)
[2013-01-30 11:34:54,335] INFO FetchRunnable- start fetching topic: demo:1-0: fetched offset = 0: consumed offset = 0, from id:1,creatorId:127.0.0.1-1359516722591,host:127.0.0.1,port:9092 (com.sohu.jafka.consumer.FetcherRunnable.java:86)
ddddd
xxxxxxxx
ddddddddd
xxxxxxxxx
quit


php形式客户端，未找到


找不到现成的php的kestrel客户端
########################################################################
kestrel
step 1: 安装sbt
wget http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-tools.sbt/sbt-launch/0.11.2/sbt-launch.jar
mv sbt-launch.jar /usr/local/bin/
echo 'java -Xmx512M -jar `dirname $0`/sbt-launch.jar "$@"' >> /usr/local/bin/sbt
chmod 777 /usr/local/bin/sbt




step 2: 安装scala
wget http://www.scala-lang.org/downloads/distrib/files/scala-2.9.2.rpm
rpm -ivh --force scala-2.9.2.rpm


step 3: 安装daemon
wget http://libslack.org/daemon/download/daemon-0.6.4.tar.gz
tar -zxvf daemon-0.6.4.tar.gz
cd daemon-0.6.4
./configure
make && make install




启动
nohup java -jar /usr/local/kestrel/kestrel_2.9.1-2.2.0.jar &
[root@xen190v daemon-0.6.4]# java
bash: java: command not found
[root@xen190v daemon-0.6.4]# yum install java


测试


进行如下操作： 
 [root@xen190v config]# telnet localhost 22133 
Trying 127.0.0.1...
Connected to xen190v.ops.corp (127.0.0.1).
Escape character is '^]'.
set x 0 0 5 
12345
STORED
set x 0 0 5 
56789
STORED


在另一个终端上获得该消息：  
[zengwenjie@xen190v ~]$ telnet localhost 22133 
Trying 127.0.0.1...
Connected to xen190v.ops.corp (127.0.0.1).
Escape character is '^]'.
get x
VALUE x 0 5
12345
END
get x
VALUE x 0 5
56789
END
################################################################
服务器端
libevent
wget http://cloud.github.com/downloads/libevent/libevent/libevent-1.4.14b-stable.tar.gz
cp libevent-1.4.14b-stable.tar.gz /usr/local/src/
tar zxvf libevent-1.4.14b-stable.tar.gz
cd  libevent-1.4.14b-stable
./configure --prefix=/usr/local/libevent
make
make install




tar zxvf beanstalkd-1.4.6.tar.gz 
cd  beanstalkd-1.4.6 
./configure --prefix=/usr/local/beanstalkd  --with-event=/usr/local/libevent
make 
make install 
启动
/usr/local/bin/beanstalkd   -d -l 127.0.0.1 -p 11300 
客户端
http://sourceforge.net/projects/beanstalk/ 下载php客户端代码 


<?php 
    require('./BeanStalk.class.php'); 
    $beanstalk = BeanStalk::open(array( 
        'servers'       => array( '127.0.0.1:11300' ), 
        'select'        => 'random peek' 
    ));   
    $beanstalk->use_tube('foo'); 
    $beanstalk->put(0, 0, 120, 'say hello world');          
    $beanstalk->watch('foo'); 
    $job = $beanstalk->reserve_with_timeout(); 
    echo $job->get();                  // Output: 'say hello world'    
    Beanstalk::delete($job);                    // Delete the job. 
?>