Linux Workflow Airflow CeleryExecutor 踩坑心酸历程
本篇所使用的系统环境为 Centos 6.8，Python 3.6.5，Airflow 1.9.0
Airflow 的搭建可以参考从0到1，AirFlow搭建踩坑历程
安装RabbitMQ
1.安装erlang
erlang地址：http://www.rabbitmq.com/releases/erlang/
使用 rpm -ihv 安装
2.安装RabbitMQ
rabbitmq地址：http://www.rabbitmq.com/releases/rabbitmq-server/
我所使用的是 rabbitmq-server-3.3.5-1.noarch.rpm
yum install -y rabbitmq-server-3.3.5-1.noarch.rpm
3.配置RabbitMQ
安装成功后使用命令 rabbitmqctl status 检查RabbitMQ 是否正常启动
# rabbitmqctl status
Status of node 'rabbit@ip-10-3-0-102' ...
[{pid,76168},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.3.5"},
{rabbit,"RabbitMQ","3.3.5"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.3.5"},
{webmachine,"webmachine","1.10.3-rmq3.3.5-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.3.5-git680dba8"},
{amqp_client,"RabbitMQ AMQP Client","3.3.5"},
{xmerl,"XML parser","1.2.10"},
{inets,"INETS CXC 138 49","5.7.1"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:48:48] [rq:48] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,109340632},
{connection_procs,5408},
{queue_procs,49248},
{plugins,282768},
{other_proc,9454312},
{mnesia,70952},
{mgmt_db,53224},
{msg_index,47728},
{other_ets,1168112},
{binary,6037264},
{code,17795196},
{atom,1621321},
{other_system,72755099}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,27013577113},
{disk_free_limit,50000000},
{disk_free,821105397760},
{file_descriptors,
[{total_limit,102542},
{total_used,5},
{sockets_limit,92285},
{sockets_used,1}]},
{processes,[{limit,1048576},{used,226}]},
{run_queue,0},
{uptime,83573}]
...done.打开和关闭服务
关闭服务：
# service rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.打开服务：
service rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.添加用户并配置权限
rabbitmqctl add_user admin 123456
rabbitmqctl set_user_tag admin administrator
rabbitmq add_vhost /
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" RabbitMQ UI访问，默认端口是15672：
http://localhost:15672
Username：admin
Password：123456
可以通过 rabbitmqctl -h 查看相应命令进行操作，也可以在 UI 中进行操作。
配置Airflow CeleryExecutor
pip3 install celery
配置CeleryExecutor (rabbitmq支持)
安装airflow的celery和rabbitmq组件
pip3 install airflow[celery,rabbitmq]配置 airflow.cfg
# 这里我将vhost 设置为 /
broker_url = pyamqp://admin:123456@localhost:5672//
celery_result_backend = db+mysql://airflow:airflow@locahost:3306/airflow
# 将 executor 设置为 CeleryExecutor
executor = CeleryExecutor
# 设置 default_queue
default_queue = airflow重新启动 airflow
airflow webserver
airflow worker(注意：用普通用户启动)
airflow flowerAirflow webserver UI 端口号为 8080
Airflow flower UI 端口号为 5555
测试 CeleryExecutor
遇到的坑：
1.queued messages 一直处于ready，没有消费者进行消费
执行命令 rabbitmqctl list_consumers 查看是否有消费者，发现消费者不处于正常队列中。
或者也可以在页面中点击你所创建的queue，判断是否有consumer。
如果没有，可以通过手动的方式进行 bind.
参考资料
1.Using RabbitMQ
2.Airflow Scaling Out with Celery