使用uWSGI做celery beat主备服务管理

什么是uWSGI

使用python的同学对uWSGI应该都比较熟悉，它一个Web服务器，它实现了WSGI协议、uwsgi、http等协议。同时它还具有配置、进程管理、socket创建、监控、日志记录、共享内存块、ipc、集群成员和 uWSGI订阅服务器的功能。
今天要介绍的是uWSGI在1.9版本后新加uWSGI Legion子系统，这个功能非常的惊喜。众所周知的在一个celery集群中只能同时存在一个beat进程，这么重要的一个进程却只能是单点，风险非常高，也调研过包括keepalive和zookeeper，但在使用和搭建上都比较复杂，关键天生的不太适合python。废话不多说，我们一起看看如何用uWSGI来做celery管理beat

服务配置

环境准备

这部分比较简单，我只说关键步骤

1. 需要两台linux服务器，我使用的ubuntu14.04, 地址分别是：192.168.1.161， 192.168.1.162
2. 分别在两台设备上安装virtualenv ，并创建虚拟环境。
3. 在虚拟环境下通过pip安装uWSGI和celery。
4. 安装redis。

测试代码和配置

创建celery测试项目

废话不多说，直接上代码，新建目录 uwsgi-beat，创建测试app
1、tasks.py

from celery_app import app

@app.task(queue='default')
def add(x, y):
    return x + y

@app.task(queue='default')
def sub(x, y):
    return x - y

2、celery_config.py

#-*-coding=utf-8-*-
from __future__ import absolute_import

from celery.schedules import crontab
# 中间件
BROKER_URL = 'redis://192.168.1.33:6379/7'
# 结果存储
CELERY_RESULT_BACKEND = 'redis://192.168.1.33:6379/8' 
# 默认worker队列
CELERY_DEFAULT_QUEUE = 'default'
# 异步任务
CELERY_IMPORTS = (
    "tasks"
)

from datetime import timedelta
# celery beat
CELERYBEAT_SCHEDULE = {
    'add':{
        'task':'tasks.add',
        'schedule':timedelta(seconds=3),
        'args':(1,12)
    }
}

3、celery_app.py

from __future__ import absolute_import
from celery import Celery

app = Celery('celery_app')
app.config_from_object('celery_config')

uWSGI配置

1、应用启动脚本
由于在uWSGI的配置中写命令比较复杂，所以我先写了一个进程管理的脚本:
daemon.sh

usage="Usage: daemon.sh (start|stop) app"

# if no args specified, show usage
if [ $# -le 1 ]; then
  echo $usage
  exit 1
fi

bin=`dirname "$0"`
bin=`cd "$bin"; pwd`
ROOT=`cd "${bin}/.."; pwd`

# get arguments
startStop=$1
shift
command=$1 #app
shift
umask 0000
stdout=/tmp/uwsgi-beat.log
pid=/tmp/uwsgi-beat.pid

case $startStop in
  (start)
    if [ -f $pid ]; then
      if ps -p `cat $pid` > /dev/null 2>&1; then
        echo $command running as process `cat $pid`.
        exit 1
      fi
    fi
    celery beat -A celery_app --loglevel=info --pidfile=$pid --logfile=$stdout --detach
    ;;
  (stop)
    if [ -f $pid ]; then
      if ps -p `cat $pid` > /dev/null 2>&1; then
        if ! kill -0 `cat $pid` > /dev/null 2>&1; then
          echo cannot stop $command with pid `cat $pid` - permission denied
        elif ${FORCEDKILL}; then
          kill -9 `cat $pid` > /dev/null 2>&1;
          sleep 1;
          echo $command killed
        else
          echo -n stopping $command
          kill `cat $pid` > /dev/null 2>&1
          while ps -p `cat $pid` > /dev/null 2>&1; do
            echo -n "."
            sleep 1;
          done
          echo
        fi
        if ! ps -p `cat $pid` > /dev/null 2>&1; then
          rm $pid
        fi
      else
        echo no $command to stop
      fi
    else
      echo no $command to stop
    fi
    ;;

  (*)
    echo $usage
    exit 1
    ;;

esac

2、uWSGI配置文件
编辑配置文件 celery.ini
需要注意，162与161上的legion-mcast的值可以一致，也可以不一致，默认是武力值比较大的会成为loader，下面配置中的85就是这个node的武力值

[uwsgi]
legion-mcast = clusterip 225.1.1.1:4242 85 bf-cbc:hello
legion-node = clusterip 225.1.1.1:4242

legion-lord = clusterip cmd:/srv/uwsgi-beat/daemon.sh start app
legion-unlord = clusterip cmd:/srv/uwsgi-beat/daemon.sh stop app
legion-death = clusterip cmd:/srv/uwsgi-beat/daemon.sh stop app
legion-join = clusterip log: Join the legion
legion-setup = clusterip log: Legion node setup

测试步骤

先在162上执行下面命令

celery worker --loglevel=info -A celery_app --logfile=/tmp/celery.log --pidfile=celeryd.pid

uwsgi celery.ini

然后在161上执行命令

uwsgi celery.ini

我配置的，161武力值是82，而162武力值是85

测试结果

先启动武力值小的节点

1、先启动161时，日志显示成为loader：

your memory page size is 4096 bytes
detected max file descriptor number: 65535
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
Python version: 3.4.3 (default, Nov 12 2018, 22:32:28)  [GCC 4.8.4]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x263d5d0
your mercy for graceful operations on workers is 60 seconds
*** Operational MODE: no-workers ***
spawned uWSGI master process (pid: 28370)
[uwsgi-mcast] joining multicast group: 225.1.1.1:4242
[uwsgi-legion] (phase: setup legion: clusterip) calling log: Legion node setup
 Legion node setup
WARNING: you are not using libuuid to generate Legions UUID
legion manager thread enabled
[uwsgi-legion] (phase: join legion: clusterip) calling log: Join the legion
 Join the legion
[uwsgi-legion] --- WE HAVE QUORUM FOR LEGION clusterip !!! (valor: 82 uuid: c7c5659c-5614d78f-9c6ac125-ae9ef4a1a checksum: 2618 votes: 1) ---
[uwsgi-legion] --- END OF QUORUM REPORT ---
[uwsgi-legion] attempting to become the Lord of the Legion clusterip
[uwsgi-legion] (phase: lord legion: clusterip) calling cmd:/srv/uwsgi-beat/daemon.sh start app
[uwsgi-legion] i am now the Lord of the Legion clusterip
Stale pidfile exists - Removing it.

2、接着启动162：

your memory page size is 4096 bytes
detected max file descriptor number: 163840
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
Python version: 3.4.3 (default, Nov 12 2018, 22:32:28)  [GCC 4.8.4]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x240d480
your mercy for graceful operations on workers is 60 seconds
*** Operational MODE: no-workers ***
spawned uWSGI master process (pid: 9207)
[uwsgi-mcast] joining multicast group: 225.1.1.1:4242
[uwsgi-legion] (phase: setup legion: clusterip) calling log: Legion node setup
 Legion node setup
legion manager thread enabled
[uwsgi-legion] node: cc valor: 82 uuid: c7c5659c-5614d78f-9c6ac125-ae9ef4a1a joined Legion clusterip
[uwsgi-legion] (phase: join legion: clusterip) calling log: Join the legion
 Join the legion
[uwsgi-legion] --- WE HAVE QUORUM FOR LEGION clusterip !!! (valor: 85 uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739 checksum: 5118 votes: 2) ---
[uwsgi-legion-node] node: cc valor: 82 uuid: c7c5659c-5614d78f-9c6ac125-ae9ef4a1a last_seen: 1563602448 vote_valor: 85 vote_uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739
[uwsgi-legion] --- END OF QUORUM REPORT ---
[uwsgi-legion] attempting to become the Lord of the Legion clusterip
[uwsgi-legion] (phase: lord legion: clusterip) calling cmd:/srv/uwsgi-beat/daemon.sh start app
[uwsgi-legion] i am now the Lord of the Legion clusterip

此时 162成为了loader， 161此时日志显示：

[uwsgi-legion] node: rabbitmq162 valor: 85 uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739 joined Legion clusterip
[uwsgi-legion] --- WE HAVE QUORUM FOR LEGION clusterip !!! (valor: 85 uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739 checksum: 5118 votes: 2) ---
[uwsgi-legion-node] node: rabbitmq162 valor: 85 uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739 last_seen: 1563602448 vote_valor: 85 vote_uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739
[uwsgi-legion] --- END OF QUORUM REPORT ---
[uwsgi-legion] a new Lord (valor: 85 uuid: a17a4ccb-4a7d-4551-9c16-b6c49be01739) raised for Legion clusterip...
[uwsgi-legion] (phase: unlord legion: clusterip) calling cmd:/srv/uwsgi-beat/daemon.sh stop app
app killed

celery beat 被停止了

先启动武力值大的节点

1、先启动162，此时162会是loader。
2、接着启动161，我们看此时161的日志

[uwsgi-legion] node: rabbitmq162 valor: 85 uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f joined Legion clusterip
[uwsgi-legion] (phase: join legion: clusterip) calling log: Join the legion
 Join the legion
[uwsgi-legion] --- WE HAVE QUORUM FOR LEGION clusterip !!! (valor: 85 uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f checksum: 4835 votes: 2) ---
[uwsgi-legion-node] node: rabbitmq162 valor: 85 uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f last_seen: 1563602724 vote_valor: 85 vote_uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f
[uwsgi-legion] --- END OF QUORUM REPORT ---

162成了legion中的一员，但并未成为loader，此时的loader依然是161。
如果此时我们停掉162，再来看161的日志

[uwsgi-legion] --- WE HAVE QUORUM FOR LEGION clusterip !!! (valor: 85 uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f checksum: 4835 votes: 2) ---
[uwsgi-legion-node] node: rabbitmq162 valor: 85 uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f last_seen: 1563602724 vote_valor: 85 vote_uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f
[uwsgi-legion] --- END OF QUORUM REPORT ---

[uwsgi-legion] node: rabbitmq162 valor: 85 uuid: a07f807d-46ce-4d37-bb18-7a2515b85c6f announced its death to Legion clusterip


[uwsgi-legion] --- WE HAVE QUORUM FOR LEGION clusterip !!! (valor: 82 uuid: 621e9fa3-917d7d4e-97026a35-5e2820821 checksum: 2325 votes: 1) ---
[uwsgi-legion] --- END OF QUORUM REPORT ---
[uwsgi-legion] attempting to become the Lord of the Legion clusterip
[uwsgi-legion] (phase: lord legion: clusterip) calling cmd:/srv/uwsgi-beat/daemon.sh start app
[uwsgi-legion] i am now the Lord of the Legion clusterip

161会立即成为新的loader

结论

从以上的测试情况看，uWSGI完全可以代替zookeeper或者keepalive来管理有竞争关系的应用，从而解决在服务必须单例部署时的热切问题。
更多配置请参考官网文档：https://uwsgi-docs.readthedocs.io/en/latest/Legion.html
原创文章，转载请注明出处