airflow安装部署
准备工作：
–安装python36版本：
[root@master ~]# yum install python36u-pip python36 python36-libs -y
– 升级pip版本：
[root@master ~]# pip3.6 install --upgrade pip
[root@master ~]# pip --version
pip 20.3.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
[root@master ~]# pip list
Package Version
---------- -------
pip 20.3.3
setuptools 39.2.0
安装数据库：
[root@master ~]# yum -y install mariadb mariadb-server
修改数据库密码：
[root@master ~]# systemctl start mariadb
[root@master ~]# mysqladmin -uroot -p password '1213456';
–安装依赖的软件包：
[root@master ~]# yum install gcc gcc-c++ cyrus-sasl cyrus-sasl-devel cyrus-sasl-lib -y
– 登录数据库：
[root@master ~]# mysql -uroot -p1213456
mysql> create database airflow;
Query OK, 1 row affected (0.00 sec)
mysql> create user 'airflow'@'%' identified by 'airflow';
Query OK, 0 rows affected (0.00 sec)
mysql> create user 'airflow'@'localhost' identified by 'airflow';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on airflow.* to 'airflow'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on *.* to 'airflow'@'%';
Query OK, 0 rows affected (0.00 sec)
–查询airflow的文件：
[root@master ~]# pip show --files apache-airflow
–airflow安装的路径：
[root@master ~]# find / -name airflow
/var/lib/mysql/airflow
–创建airflow的配置文件
[root@master ~]# mkdir -p /etc/airflow
[root@master ~]# vim /etc/airflow/airflow.cfg
sql_alchemy_conn = mysql://airflow:airflow@localhost:3306/airflow
–设置airflow的文件路径：
[root@master ~]# vim /etc/profile
export AIRFLOW_HOME=/etc/airflow
export SLUGIFY_USES_TEXT_UNIDECODE=yes
–安装airflow软件：
[root@master ~]# pip install apache-airflow
[root@master ~]# pip install apache-airflow[devel]
[root@master ~]# pip install apache-airflow[celery]
[root@master ~]# pip install apache-airflow[jdbc]
[root@master ~]# pip install apache-airflow[mysql]
[root@master ~]# pip install apache-airflow[password]
[root@master ~]# pip install apache-airflow[rabbitmq]
[root@master ~]# pip install apache-airflow[redis]
–查询版本号：
[root@master ~]# pip list | grep -i airflow
apache-airflow 2.0.0
–初始化报错：
[root@master ~]# airflow db init
DB: sqlite:root/airflow/airflow.db
[2020-12-04 12:45:16,925] {db.py:678} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [unusual_prefix_619bc6aee900a08efa12755006b4aa599654a401_example_kubernetes_executor_config] Could not import DAGs in example_kubernetes_executor_config.py: No module named 'kubernetes'
WARNI [unusual_prefix_619bc6aee900a08efa12755006b4aa599654a401_example_kubernetes_executor_config] Install kubernetes dependencies with: pip install apache-airflow['cncf.kubernetes']
–解决问题：
[root@master ~]# pip install apache-airflow['cncf.kubernetes']
--再次初始化：
[root@master ~]# airflow db init
DB: sqlite:root/airflow/airflow.db
[2020-12-04 12:46:43,211] {db.py:678} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Initialization done
–启动sirflow服务：
[root@master ~]# airflow webserver -p 8080 &
[1] 23474
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2020-12-04 12:50:19,220] {dagbag.py:440} INFO - Filling up the DagBag from /dev/null
[2020-12-04 12:50:19,294] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[2020-12-04 12:50:37 +0800] [23608] [INFO] Starting gunicorn 19.10.0
[2020-12-04 12:50:37 +0800] [23608] [INFO] Listening at: http://0.0.0.0:8080 (23608)
[2020-12-04 12:50:37 +0800] [23608] [INFO] Using worker: sync
[2020-12-04 12:50:37 +0800] [23618] [INFO] Booting worker with pid: 23618
[2020-12-04 12:50:37 +0800] [23619] [INFO] Booting worker with pid: 23619
[2020-12-04 12:50:37 +0800] [23620] [INFO] Booting worker with pid: 23620
[2020-12-04 12:50:37 +0800] [23621] [INFO] Booting worker with pid: 23621
[2020-12-04 12:50:44,587] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:50:44,604] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:50:44,659] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:50:44,713] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
[2020-12-04 12:51:09 +0800] [23608] [INFO] Handling signal: ttin
[2020-12-04 12:51:09 +0800] [23820] [INFO] Booting worker with pid: 23820
[2020-12-04 12:51:11,327] {manager.py:727} WARNING - No user yet created, use flask fab command to do it.
