HTTPSQS说明安装维护

 

HTTPSQS is a Simple Queue Service based on HTTP GET/POST protocol.

This is free software, and you are welcome to modify and redistribute it under the New BSD License.

点击这里查看简体中文使用说明 (Click here to see the Simplified Chinese instructions)

1. Features

• Very simple

• Very fast, more than 10000 requests/sec

• High concurrency, support the tens of thousands of concurrent connections.

• Multiple queue

• A single queue length maximum support one billion (1,000,000,000).

• Low memory consumption, mass data storage, storage dozens of GB of data takes less than 100MB of physical memory buffer.

• Convenient to change the maximum queue length of per-queue.

• Queue status view

• Be able to view the contents of the specified queue ID.

• Multi-Character encoding support

• Less than 900 lines source code, easy to second development.

2. Benchmark Test

Using Apache AB for HTTPSQS Benchmark Test

3. Server Install

Operating System: CentOS Linux 5.4

ulimit -SHn 65535
 
wget http://httpsqs.googlecode.com/files/libevent-2.0.10-stable.tar.gz
tar zxvf libevent-2.0.10-stable.tar.gz
cd libevent-2.0.10-stable/
./configure --prefix=/usr/local/libevent-2.0.10-stable/
make
make install
cd ../
 
wget http://httpsqs.googlecode.com/files/tokyocabinet-1.4.47.tar.gz
tar zxvf tokyocabinet-1.4.47.tar.gz
cd tokyocabinet-1.4.47/
./configure --prefix=/usr/local/tokyocabinet-1.4.47/
#Note: In the 32-bit Linux operating system, compiler Tokyo cabinet, please use the ./configure --enable-off64 instead of ./configure to breakthrough the filesize limit of 2GB.
#./configure --enable-off64 --prefix=/usr/local/tokyocabinet-1.4.47/
make
make install
cd ../
 
wget http://httpsqs.googlecode.com/files/httpsqs-1.6.tar.gz
tar zxvf httpsqs-1.6.tar.gz
cd httpsqs-1.6/
make
make install
cd ../

4. Server Document

[root@xoyo ~]# httpsqs -h

 -l <ip_addr> interface to listen on, default is 0.0.0.0
 -p <num>       TCP port number to listen on (default: 1218)
 -x <path>      database directory (example: /opt/httpsqs/data)
 -t <second>    timeout for an http request (default: 3)
 -s <second>    the interval to sync updated contents to the disk (default: 5)
 -c <num>       the maximum number of non-leaf nodes to be cached (default: 1024)
 -m <size>      database memory cache size in MB (default: 100)
 -i <file>      save PID in <file> (default: /tmp/httpsqs.pid)
 -a <auth>      the auth password to access httpsqs (example: mypass123)
 -d             run as a daemon
 -h             print this help and exit

Example:

ulimit -SHn 65535
httpsqs -d -p 1218 -x /data0/queue

or with auth password:

ulimit -SHn 65535
httpsqs -d -p 1218 -x /data0/queue -a mypass123

Use command "killall httpsqs", "pkill httpsqs" and "kill cat /tmp/httpsqs.pid" to stop httpsqs.

Please note that don't use the command "pkill -9 httpsqs" and "kill -9 PID of httpsqs", otherwise, the data in memory has not been saved to disk will be lost.

5. Client Document

Important Notice: The message must be GET before the PUT position to catch up with the GET position in the circle.

(1). PUT text message into a queue

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=put&data=url_encoded_text_message&auth=mypass123"

or

HTTP POST protocol (Using curl for example):

curl -d "url_encoded_text_message" "http://host:port/?name=your_queue_name&opt=put&auth=mypass123"

Using google chrome browser for example:

If PUT successful, return:

HTTPSQS_PUT_OK

If an error occurs, return:

HTTPSQS_PUT_ERROR

If Queue full, return:

HTTPSQS_PUT_END

HTTPSQS add a line "Pos: xxx" in the HTTP header and sends that back to the client since version 1.2, the "xxx" is the put pos of the current queue. For example:

HTTP/1.1 200 OK
Content-Type: text/plain
Keep-Alive: 120
Pos: 19
Date: Thu, 18 Mar 2010 04:57:08 GMT
Content-Length: 14
 
HTTPSQS_PUT_OK

(2). GET text message from a queue

HTTP GET protocol (Using curl for example):

curl "http://host:port/?charset=utf-8&name=your_queue_name&opt=get&auth=mypass123"

curl "http://host:port/?charset=gb2312&name=your_queue_name&opt=get&auth=mypass123"

Using google chrome browser for example:

Return the queue contents.

If there is no unread queue message, return:

HTTPSQS_GET_END

HTTPSQS add a line "Pos: xxx" in the HTTP header and sends that back to the client since version 1.2, the "xxx" is the get pos of the current queue. For example:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Keep-Alive: 120
Pos: 7
Date: Thu, 18 Mar 2010 04:56:01 GMT
Content-Length: 7
 
message

The charset parameter (Example: /?charset=utf-8):

Documents transmitted with HTTP that are of type text, such as text/html, text/plain, etc., can send a charset parameter in the HTTP header to specify the character encoding of the document.

It is very important to always label Web documents explicitly. HTTP 1.1 says that the default charset is ISO-8859-1. But there are too many unlabeled documents in other encodings, so browsers use the reader's preferred encoding when there is no explicit charset parameter.

The line in the HTTP header typically looks like this:

Content-Type: text/plain; charset=utf-8

In theory, any character encoding that has been registered with IANA can be used, but there is no browser that understands all of them.

(3). View queue status

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=status&auth=mypass123"

Return (for example):

HTTP Simple Queue Service v1.3
------------------------------
Queue Name: xoyo
Maximum number of queues: 1000000
Put position of queue (1st lap): 45
Get position of queue (1st lap): 6
Number of unread queue: 39

or if the "Put position of queue" greater than the "Maximum number of queues", it will come back to the pos 1.

HTTP Simple Queue Service v1.3
------------------------------
Queue Name: xoyo
Maximum number of queues: 1000000
Put position of queue (2st lap): 4562
Get position of queue (1st lap): 900045
Number of unread queue: 104517

Using google chrome browser for example:

(4). View queue status in json

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=status_json&auth=mypass123"

Return (for example):

{"name":"xoyo","maxqueue":1000000,"putpos":45,"putlap":1,"getpos":6,"getlap":1,"unread":39}

or if the "Put position of queue" greater than the "Maximum number of queues", it will come back to the pos 1.

{"name":"xoyo","maxqueue":1000000,"putpos":4562,"putlap":2,"getpos":900045,"getlap":1,"unread":104517}

(5). View the contents of the specified queue pos (id)

HTTP GET protocol (Using curl for example):

curl "http://host:port/?charset=utf-8&name=your_queue_name&opt=view&pos=5&auth=mypass123"

curl "http://host:port/?charset=gb2312&name=your_queue_name&opt=view&pos=19&auth=mypass123"

pos >=1 and <= 1000000000

Return the contents of the specified queue pos.

(6). Reset the queue

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=reset&auth=mypass123"

If reset successful, return:

HTTPSQS_RESET_OK

If an error occurs, return:

HTTPSQS_RESET_ERROR

(7). Change the maximum queue length of per-queue

Default maximum queue length: 1000000

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=maxqueue&num=1000000000&auth=mypass123"

num >=10 and <= 1000000000

If change the maximum queue length successful, return:

HTTPSQS_MAXQUEUE_OK

This operation will be cancelled if "Put position of queue" less than "Get position of queue" (lap of PUT > lap of GET), will return:

HTTPSQS_MAXQUEUE_CANCEL

(8). Change the interval to sync updated contents to the disk

Default interval: 5 seconds or the parameter set by "httpsqs -s second"

HTTP GET protocol (Using curl for example):

curl "http://host:port/?name=your_queue_name&opt=synctime&num=10&auth=mypass123"

num >=1 and <= 1000000000

If change the interval successful, return:

HTTPSQS_SYNCTIME_OK

If "num" not between 1 to 1000000000, will return:

HTTPSQS_SYNCTIME_CANCEL

(9). Authentication failed

If password(/?auth=xxx) authentication failed, return:

HTTPSQS_AUTH_FAILED

(10). Global error

If an global error occurs, return:

HTTPSQS_ERROR

6. Client Document

(1). PHP Client

A、PHP ext: http://code.google.com/p/php-httpsqs-client/

B、PHP Class: View source code: httpsqs_client.php

Usage:

<?php
......
include_once("httpsqs_client.php");
$httpsqs = new httpsqs;
 
/* -----http connect without Keep-Alive----- */
/*
1. PUT text message into a queue.
   If PUT successful, return boolean: true
   If an error occurs, return boolean: false
*/
$result = $httpsqs->put($host, $port, $charset, $name, $data);
 
/*
2. GET text message from a queue.
   Return the queue contents.
   If there is no unread queue message, return text: HTTPSQS_GET_END
   If an error occurs, return boolean: false.
*/
$result = $httpsqs->get($host, $port, $charset, $name);
 
/*
3. GET text message and pos from a queue.
   Return example: array("pos" => 7, "data" => "text message")
   If there is no unread queue message, return: array("pos" => 0, "data" => "HTTPSQS_GET_END")
   If an error occurs, return boolean: false.
*/
$result = $httpsqs->gets($host, $port, $charset, $name);
 
/*
4. View queue status
*/
$result = $httpsqs->status($host, $port, $charset, $name);
 
/*
5. View queue status in json
   Return example: {"name":"queue_name","maxqueue":5000000,"putpos":130,"putlap":1,"getpos":120,"getlap":1,"unread":10}
*/
$result = $httpsqs->status_json($host, $port, $charset, $name);
 
/*
6. View the contents of the specified queue pos (id).
   Return the contents of the specified queue pos.
*/
$result = $httpsqs->view($host, $port, $charset, $name, $pos);
 
/*
7. Reset the queue.
   If reset successful, return boolean: true
   If an error occurs, return boolean: false
*/
$result = $httpsqs->reset($host, $port, $charset, $name);
 
/*
8. Change the maximum queue length of per-queue.
   If change the maximum queue length successful, return boolean: true
   If it be cancelled, return boolean: false
*/
$result = $httpsqs->maxqueue($host, $port, $charset, $name, $num);
 
/*
9. Change the interval to sync updated contents to the disk.
   If change the interval successful, return boolean: true
   If it be cancelled, return boolean: false
*/
$result = $httpsqs->synctime($host, $port, $charset, $name, $num);
 
/* -----http pconnect with Keep-Alive (Very fast in Command line mode)----- */
$result = $httpsqs->pput($host, $port, $charset, $name, $data);
$result = $httpsqs->pget($host, $port, $charset, $name);
$result = $httpsqs->pgets($host, $port, $charset, $name);
$result = $httpsqs->pstatus($host, $port, $charset, $name);
$result = $httpsqs->pstatus_json($host, $port, $charset, $name);
$result = $httpsqs->pview($host, $port, $charset, $name, $pos);
$result = $httpsqs->preset($host, $port, $charset, $name);
$result = $httpsqs->pmaxqueue($host, $port, $charset, $name, $num);
$result = $httpsqs->psynctime($host, $port, $charset, $name, $num);
?>

(2). Perl Client (Third Party, Developer: tonny0830)

Perl client file path: httpsqs-1.3/client/perl/

View source code: http://code.google.com/p/httpsqs/source/browse/trunk/client/perl/

(3). Java Client (Third Party, Developer: SuperSNY)

Download: http://httpsqs.googlecode.com/files/httpsqs4j-java-client-1.0.zip

New code: svn checkout http://httpsqs.googlecode.com/svn/trunk/client/httpsqs4j/ httpsqs4j

Documentation: http://blog.s135.com/book/httpsqs/client/httpsqs4j/

 

我已将此文档所需软件上传到这里，有兴趣的朋友们可以下载自行研究，沟通．

软件提取码为：MjAxMTA4MjIxMDU2MTQ1Njc1NjE3