安装airflow——踩坑

环境：centos6.5； mysql  Ver 14.14 Distrib 5.1.73,

主要参考这个文档……

https://blog.csdn.net/kk185800961/article/details/78431484

1、遇到的大坑：

[root@hadoopdn-04 ~]# /usr/local/python27/bin/airflow webserver -p 8080
[2017-04-21 13:56:19,229] {_init_.py:57} INFO - Using executor SequentialExecutor
____________ _____________
____ |_( )________ _/_ /________ __
____ /| |_ /__ __/ /_ __ /_ __ _ | /| / /
___ ___ | / _ / _ _/ _ / / // /_ |/ |/ /
// |// // // // __/__/|_/

/usr/local/python27/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-04-21 13:56:20,230] [3013]

{models.py:167}

INFO - Filling up the DagBag from /root/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
Traceback (most recent call last):
File "/usr/local/python27/bin/airflow", line 28, in
args.func(args)
File "/usr/local/python27/lib/python2.7/site-packages/airflow/bin/cli.py", line 791, in webserver
gunicorn_master_proc = subprocess.Popen(run_args)
File "/usr/local/python27/lib/python2.7/subprocess.py", line 711, in _init_
errread, errwrite)
File "/usr/local/python27/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

The solution is , PATH variable must be updated to have all the dependencies of airflow

即：确认/etc/profile里的配置，然后source /etc/profile

本来以为我的是这个问题，折腾了一天后现并不是！！终于……

solution post：

https://www.cnblogs.com/lwglinux/p/7100400.html

找到了解决方法……现在的airflow真是太不友好了，只说找不到文件找不到文件，不说找不到那个文件啊……

2、又一个因为版本问题的坑

[root@hadoop-07 bin]# ./airflow initdb
[2018-07-26 15:14:33,354] {__init__.py:57} INFO - Using executor SequentialExecutor
DB: mysql://******
[2018-07-26 15:14:34,553] {db.py:287} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
Traceback (most recent call last):
……………………
File "/usr/local/python/lib/python2.7/site-packages/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py", line 36, in upgrade
    op.alter_column(table_name='dag', column_name='last_scheduler_run', type_=mysql.DATETIME(fsp=6))

…………

  File "/usr/local/python/lib/python2.7/site-packages/alembic/ddl/mysql.py", line 67, in alter_column
    else existing_autoincrement
  File "/usr/local/python/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
………………
  File "/usr/local/python/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/python/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
sqlalchemy.exc.ProgrammingError: (_mysql_exceptions.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(6) NULL' at line 1") [SQL: u'ALTER TABLE dag MODIFY last_scheduler_run DATETIME(6) NULL'] (Background on this error at: http://sqlalche.me/e/f405)

solution post：

https://www.phpyuan.com/article/1519.html

需要变更

/usr/local/python/lib/python2.7/site-packages/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py

将 mysql.DATETIME(fsp=6) 全部改为 mysql.DATETIME()或将mysql升级到5.7或以上的版本。

3、mysql和airflow没在同一台机器引起的（我认为是这样）

sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1292, "Incorrect datetime value: 
'2019-04-28 04:02:34.333702+00:00' for column 'last_scheduler_run' at row 1") [SQL: 
u'INSERT INTO dag (dag_id, is_paused, is_subdag, is_active, last_scheduler_run, last_pickled,
last_expired, scheduler_lock, pickle_id, fileloc, owners, description, default_view, schedule_interval) 
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: 
('example_skip_dag', 1, 0, 1, datetime.datetime(2019, 4, 28, 4, 2, 34, 333702, tzinfo=),
 None, None, None, None, '/usr/lib/python2.7/site-packages/airflow/example_dags/example_skip_dag.py',
 u'airflow', u'', None, '{"type": "timedelta", "attrs": {"seconds": 0, "days": 1, 
"microseconds": 0}}')] (Background on this error at: http://sqlalche.me/e/e3q8)

在安装airflow的机器上安装mysql的一些工具。

pip install apache-airflow[mysql]
yum install mysql-devel