【Gearman学习笔记】持久化存储队列

1.前言

这里是官网的文档：http://gearman.org/manual/job_server/#persistent_queues，说的并不是很清楚，所以自己做了个总结。

测试环境：

Linux-gentoo #1 SMP x86_64 Intel(R) Xeon(R) CPU E5645 @ 2.40GHz GenuineIntel GNU/Linux

2.为什么要做持久化？

从官网上找答案：

Inside the Gearman job server, all job queues are stored in memory. This means if a server restarts or crashes with pending jobs, they will be lost and are never run by a worker. Persistent queues were added to allow background jobs to be stored in an external durable queue so they may live between server restarts and crashes. The persistent queue is only enabled for background jobs because foreground jobs have an attached client. If a job server goes away, the client can detect this and restart the foreground job somewhere else (or report an error back to the original caller). Background jobs on the other hand have no attached client and are simply expected to be run when submitted.

翻译下：gearman的job server中的工作队列存储在内存中，这意味着一旦服务器在尚有等待处理的任务时重启或者宕机，那么这些任务就会丢失。持久化存储队列可以允许添加后台任务，并将其存储在外部的持久型队列里（比如mysql数据库），这样就没有上面提到的问题了。

3.gearman持久化配置流程

我使用mysql做任务持久化存储。

首先，建表：

create database gearman
   create table `gearman_queue` (
   `unique_key` varchar(64) NOT NULL,
   `function_name` varchar(255) NOT NULL,
   `priority` int(11) NOT NULL,
   `data` LONGBLOB NOT NULL,
   `when_to_run` INT, PRIMARY KEY  (`unique_key`)
)

注：gearman支持的持久化拓展有drizzle, mysql,  memcache, postgre, sqlite等

（我直接从gearman源码里贴出来了）：

 case ${PERSISTENT:-none} in
      drizzle)
        ;;
      mysql)
        GEARMAND_PARAMS="${GEARMAND_PARAMS} -q mysql"
            [ ${PERSISTENT_HOST}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --mysql-host=${PERSISTENT_HOST}"
            [ ${PERSISTENT_USER}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --mysql-user=${PERSISTENT_USER}"
            [ ${PERSISTENT_PASS}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --mysql-password=${PERSISTENT_PASS}"
            [ ${PERSISTENT_DB}     ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --mysql-db=${PERSISTENT_DB}"
            [ ${PERSISTENT_TABLE}  ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --mysql-table=${PERSISTENT_TABLE}"
            [ ${PERSISTENT_PORT}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --mysql-port=${PERSISTENT_PORT}"
        ;;
      memcache)
        [ ${PERSISTENT_SERVERLIST} ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} -q libmemcached --libmemcached-servers=${PERSISTENT_SERVERLIST}"
        ;;
      postgre)
        GEARMAND_PARAMS="${GEARMAND_PARAMS} -q libpq"
            [ ${PERSISTENT_HOST}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --libpq-host=${PERSISTENT_HOST}"
            [ ${PERSISTENT_USER}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --libpq-user=${PERSISTENT_USER}"
            [ ${PERSISTENT_PASS}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --libpq-password=${PERSISTENT_PASS}"
            [ ${PERSISTENT_DB}     ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --libpq-dbname=${PERSISTENT_DB}"
            [ ${PERSISTENT_PORT}   ] && GEARMAND_PARAMS="${GEARMAND_PARAMS} --libpq-port=${PERSISTENT_PORT}"
            [ ${PERSISTENT_TABLE}  ] && ewarn "Libpq doesn't recognise 'table' parameter."
            [ ${PERSISTENT_SOCKET} ] && ewarn "Libpq doesn't recognise 'socket' parameter. If no host is set, it automatically falls back to a socket."
        ;;
      tokyocabinet)
        GEARMAND_PARAMS="${GEARMAND_PARAMS} -q libtokyocabinet --libtokyocabinet-file=${PERSISTENT_FILE}"
        ;;
      sqlite)
        GEARMAND_PARAMS="${GEARMAND_PARAMS} -q libsqlite3 --libsqlite3-db=${PERSISTENT_FILE}"
        ;;
          none)
            ;;
      *)
            eerror "Wrong persistent queue store setting in /etc/conf.d/gearmand."
        return 1
        ;;
    esac

一般情况下，gearman的配置文件放在/etc/init.d/目录下，我这里放在了/etc/conf.d/gearmand里。

配置文件：（注意这里PERSISTENT="mysql"）

# /etc/conf.d/gearmand: config file for /etc/init.d/gearmand

# Persistent queue store
# The following queue stores are available:
# drizzle|memcache|mysql|postgre|sqlite|tokyocabinet|none
# If you do not wish to use persistent queues, leave this option commented out.
# Note that persistent queue mechanisms are mutally exclusive.
PERSISTENT="mysql"

# Persistent queue settings for drizzle, mysql and postgre
#PERSISTENT_SOCKET=""
PERSISTENT_HOST="localhost"
PERSISTENT_PORT="3306"
PERSISTENT_USER="gearman"
PERSISTENT_PASS="your-pass-word-here"
PERSISTENT_DB="gearman"
PERSISTENT_TABLE="gearman_queue"

# Persistent queue settings for sqlite
#PERSISTENT_FILE=""

# Persistent queue settings for memcache
#PERSISTENT_SERVERLIST=""

# General settings
#
# -j, --job-retries=RETRIES   Number of attempts to run the job before the job
#                             server removes it. Thisis helpful to ensure a bad
#                             job does not crash all available workers. Default
#                             is no limit.
# -L, --listen=ADDRESS        Address the server should listen on. Default is
#                             INADDR_ANY.
# -p, --port=PORT             Port the server should listen on. Default=4730.
# -r, --protocol=PROTOCOL     Load protocol module.
# -t, --threads=THREADS       Number of I/O threads to use. Default=0.
# -v, --verbose               Increase verbosity level by one.
# -w, --worker-wakeup=WORKERS Number of workers to wakeup for each job received.
#                             The default is to wakeup all available workers.
GEARMAND_PARAMS="-L 127.0.0.1 --verbose=DEBUG"

然后，在gearman的启动shell里，看源码：

 ebegin "Starting ${SVCNAME}"
        start-stop-daemon --pidfile /var/run/gearmand/gearmand.pid --start \
        --exec /usr/sbin/gearmand -- --pid-file=/var/run/gearmand/gearmand.pid \
        --user=gearmand --daemon \
        --log-file=/var/log/gearmand/gearmand.log ${GEARMAND_PARAMS}
 eend $?

启动gearman后，查看进程：

ps aux | grep gearman

可以看到，gearman和mysql数据库建立连接的语句：

gearmand 16746  0.0  0.3 423380  2176 ?        Ssl  02:54   0:00 /usr/sbin/gearmand --pid-file=/var/run/gearmand/gearmand.pid --user=gearmand --daemon --log-file=/var/log/gearmand/gearmand.log -L 127.0.0.1 --verbose=DEBUG -q mysql --mysql-host=localhost --mysql-user=gearman --mysql-password=DJW55RbdvazpmLuN --mysql-db=gearman --mysql-table=queue --mysql-port=3306

4.参考

1.http://gearman.org/manual/job_server/#persistent_queues

2.http://www.jaceju.net/blog/archives/1211/

3.http://blog.sina.com.cn/s/blog_54ef398901018l84.html