airflow高可用（HA）环境搭建

简述

前端时间尝试着搭建了airflow的高可用（HA）环境，为避免自己遗忘，现将安装过程整理成文档。公司环境没有外网，只是配置了本地yum源，安装之前将所有的用到的包都预先整理了一下。为了方便大家阅读我稍后会将包结构给出，大家在阅读的时候有些地方请根据自己的环境进行替换。
airflow_ha_install/
├── airflow-packages //该目录下主要存放了airflow依赖的三方包，没有外网只能先下载下来搞咯
├── airflow-scheduler-failover-controller-master
│ ├── CHANGELOG.md
│ ├── License.md
│ ├── README.md
│ ├── scheduler_failover_controller
│ ├── scripts
│ ├── setup.cfg
│ └── setup.py
├── pip-9.0.1.tar.gz
├── rabbitmq
│ ├── erlang-19.3.6.4-1.el7.x86_64.rpm //这个包是我编译之后的包，大家可以通过下载源码自行编译
│ └── rabbitmq-server-3.7.4-1.el7.noarch.rpm
└── systemd
├── airflow
├── airflow.conf
├── airflow-flower.service
├── airflow-kerberos.service
├── airflow-scheduler.service
├── airflow-webserver.service
├── airflow-worker.service
└── README

环境规划

host	IP	service
airflow-01	192.168.3.191	airflow-worker/webserver/ASFC/rabbitmq
airflow-02	192.168.3.192	airflow-worker/webserver/ASFC/rabbitmq
airflow-03	192.168.3.193	airflow-worker/webserver/Haproxy

准备环境

centos7
准备环境的操作需要以root用户完成

关闭防火墙和selinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0 
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

新建用户airflow 并赋予sudo权限

useradd airflow

安装依赖包

yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python-devel wget cyrus-sasl-devel.x86_64 libffi-devel python-psycopg2

安装pip

上传安装包到目标环境中的每台服务器
以root身份登录目标环境中的每台服务器执行如下命令

yum groupinstall "Development tools" -y
yum install python-setuptools python-devel -y
tar xzvf pip-9.0.1.tar.gz
cd pip-9.0.1
python setup.py install

安装rabbitmq_ha

单机上安装

分别在192.168.3.191 192.168.3.192 节点上安装rabbitmq server

1. Install erlang

   yum install airflow_ha_install/rabbitmq/erlang-19.3.6.4-1.el7.x86_64.rpm
   

2. Install RabbitMQ Server

   yum install airflow_ha_install/rabbitmq/rabbitmq-server-3.7.4-1.el7.noarch.rpm
   

3. 启动

   rabbitmq-server start
   

4. 设置开机自启动

   systemctl enable rabbitmq-server
   

5. 启用插件rabbitmq management

   rabbitmq-plugins enable rabbitmq_management
   输入http://ip:15672可以登录管理界面，默认账户guest/guest只能使用http://localhost:15672登录，要想远程登录，需要添加一个新的用户：
   # rabbitmqctl add_user admin admin
   #用户设置为administrator才能远程访问
   rabbitmqctl set_user_tags admin administrator 
   rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" //该命令使用户admin具有‘/’这个virtual host中所有资源的配置、写、读权限以便管理其中的资源,查看所有用户#rabbitmqctl list_users
   

集群搭建

#将192.168.3.191上的/var/lib/rabbitmq/.erlang.cookie复制到192     var/lib/rabbitmq/.erlang.cookie, 即服务器必须具有相同的cookie，如果不相同的话，无法搭建集群.
#192.168.3.192节点上分别执行命令，加入到集群
systemctl restart rabbitmq-server
rabbitmqctl stop_app
rabbitmqctl join_cluster  rabbit@airflow-01
rabbitmqctl start_app
#其中--ram代表是内存节点，如果希望是磁盘节点则不用加--ram，在rabbitmq集群中，至少需要一个磁盘节点
#查看集群的状态
rabbitmqctl cluster_status

设置成镜像队列
在192.168.3.191上执行

rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode": "automatic"}'

安装HAproxy

在192.168.3.191上安装HAproxy

yum install haproxy
vi /etc/haproxy/haproxy.cfg 
#1.修改默认的模式为tcp
#2.在文件末尾添加如下内容：
# port forwarding from 8080 to the airflow webserver on 8080
listen impala
 bind 0.0.0.0:8080
 balance roundrobin
 server airflow_webserver_1 auto-nn-01.embrace.com:8080 check
 server airflow_webserver_2 auto-nn-02.embrace.com:8080 check
 
listen rabbitmq-web-ui
 bind 0.0.0.0:15677
 balance roundrobin
 server rabbitmq_server_1 auto-cn-01.embrace.com:15672 check
 server rabbitmq_server_2 auto-nn-01.embrace.com:15672 check
 
listen rabbitmq-ui
 bind 0.0.0.0:5677
 balance roundrobin
 server rabbitmq_server_1 auto-cn-01.embrace.com:5672 check
 server rabbitmq_server_2 auto-nn-01.embrace.com:5672 check

# This sets up the admin page for HA Proxy at port 1936.
listen stats :1936
 mode http
 stats enable
 stats uri /
 stats hide-version
 stats refresh 30s

启动

systemctl start haproxy

关闭

systemctl stop haproxy

加入系统服务

systemctl enbale haproxy

安装配置airflow

安装

以root用户登录目标环境中的一台服务器，并执行如下命令进行安装

cd airflow_ha_install/airflow-packages
pip install --no-index --find-links . apache-airflow
pip install --no-index --find-links . apache-airflow[celery]
pip install --no-index --find-links . apache-airflow[crypto]

配置环境变量（AIRFLOW_HOME）并创建必要的目录

以airflow用户登录目标环境中的每一台服务器，执行如下命令

mkdir ~/airflow
# vi ~/.bash_profile 添加如下内容
export AIRFLOW_HOME=~/airflow

#载入环境变量
source ~/.bash_profile
#创建必须的目录

cd $AIRFLOW_HOME
mkdir dags
mkdir logs

配置airflow

该部分操作只需要在目标环境中的某一台服务器上执行即可，执行完成之后需将配置文件同步到其他服务器上

1. 初始化数据库

   airflow initdb
   

2.修改配置文件（$AIRFLOW_HOME/airflow.cfg）
* 修改执行器为CeleryExecutor
excutor=CeleryExcutor
* 修改数据库链接信息
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@$ip4addr:5432/airflow
* 设置dags初始化后为暂停状态
dags_are_paused_at_creation = True
* 不引用例子
load_examples = False
* 设置消息队列链接信息
broker_url = amqp://admin:[email protected]:5677/
* 设置celery执行pg
celery_result_backend = db+postgresql+psycopg2://postgres:[email protected]:5432/airflow
* 再次运行 airflow initdb，创建相关表结构
airflow initdb

3. 下发airflow.cfg
   将airflow.cfg到集群中的每台机器上

将airflow加入系统服务

以root用户登录目标环境中每一台服务器，执行如下命令

1. 拷贝service文件

   cd airflow_ha_install/systemd
   cp airflow*.service /usr/lib/systemd/system
   chown airflow:airflow /usr/lib/systemd/system/airflow*.service
   #确保文件中airflow命令的路径是正确的
   

2. copy the airflow.conf to /etc/tmpfiles.d/ or /usr/lib/tmpfiles.d/

   cd airflow_ha_install/systemd
   cp  airflow.conf  /etc/tmpfiles.d/
   

3. 确保 /run/airflow 存在，且所有者和所属组正确 (0755 airflow airflow)
4. 拷贝airflow至/etc/sysconfig/airflow 并修改 AIRFLOW_HOME

   cd airflow_ha_install/systemd
   cp airflow /etc/sysconfig/airflow
   

5. 加入到开机自启动

   systemctl enable airflow-webserver
   systemctl enable airflow-wroker
   systemctl enable airflow-flower
   

安装airflow-scheduler-failover-controller

1. Install ASFC

   cd airflow_ha_install/airflow-scheduler-failover-controller-master
   pip install -e . --no-index --find-links ../airflow-packages
   

2. 初始化

   scheduler_failover_controller init
   

3. 更新airflow.cfg

   scheduler_nodes_in_cluster = auto-nn-01,auto-nn-02
   #it is recommended that you use the value printed from the following command:scheduler_failover_controller get_current_host
   

4. 启动airflow免密登录

5. 测试联通行

   scheduler_failover_controller test_connection
   

6. 确保运行正常

   scheduler_failover_controller metadata
   

7. 加入到系统服务

   1. Login to each of the machines acting as Scheduler Failover Controllers

   2. Login as root

   3. Copy the scheduler_failover_controller.service file to the systemd directory. See the bellow list for the correct location based off your environment.
      ​ * /usr/lib/systemd/system/ for CentOS
      ​ * /lib/systemd/system/ for Ubuntu

   4. Edit the scheduler_failover_controller.service and change any configurations you would like
      ​ a. Change user and group options as needed to ensure that it is the same as the user and group the main airflow processes are running as

   5. Enable the service to be ran on startup of the machine
      ​ systemctl enable scheduler_failover_controller

   6. You're done!