docker下载、启动、配置apache airflow

        mac下安装apache airflow，会遇到各种麻烦，如python版本不支持、网络下载包慢、各种配置等繁琐问题，幸好docker上有airflow资源。废话不多说，接下来记录我在本地搭建airflow项目的过程。

 

一、启动本地docker

二、访问docker hub上的airflow官网项目地址：https://hub.docker.com/r/apache/airflow。

$docker pull apache/airflow

三、在本地命令行工具执行docker pull apache/airflow，执行完之后，等待资源下载完成，之后重启docker容器，即可看到docker已经有airflow镜像。

四、运行airflow，在命令行工具执行以下命令（也可以在docker上对应的镜像镜像图像操作）       

$docker run -d -p 8080:8080 puckel/docker-airflow

五、如第四步没什么问题，即可访问到airflow控制台。

       访问地址：http://localhost:8080/admin/。（端口号在启动命令行里设置）

        注：还没初始化数据库之前，dag下的任务列表是为空的。

六、初始化数据库        

$airflow initdb

这时候应该会报以下错误：

$ airflow initdb
DB: sqlite:////usr/local/airflow/airflow.db
[2020-05-21 12:09:20,775] {{db.py:368}} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 91, in get_fernet
    for fernet_part in fernet_key.split(',')
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 91, in 
    for fernet_part in fernet_key.split(',')
  File "/usr/local/lib/python3.7/site-packages/cryptography/fernet.py", line 35, in __init__
    key = base64.urlsafe_b64decode(key)
  File "/usr/local/lib/python3.7/base64.py", line 133, in urlsafe_b64decode
    return b64decode(s)
  File "/usr/local/lib/python3.7/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in 
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1140, in initdb
    db.initdb(settings.RBAC)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 117, in initdb
    schema='default'))
  File "", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 127, in __init__
    self.extra = extra
  File "", line 1, in __set__
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 218, in set_extra
    fernet = get_fernet()
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 95, in get_fernet
    raise AirflowException("Could not create Fernet object: {}".format(ve))
airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

这问题是因为fernet key不正确，解决方法就是在docker命令行下生成fernet key，并配置到airflow.cfg的fernet_key配置项：

$python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"（回车）

>>>print(Fernet.generate_key().decode())"（回车）
>>>这里会得到一个字符串，并拷贝这个字符串

---------------------------------------------------

以上配置完成之后，重新执行：

$airflow initdb

如出现以下信息，则表明数据库初始化成功：

参考：https://www.jianshu.com/p/5349168dd346

 

七、重新刷新airflow控制台之后：

完！