Airflow-工作流中文问题解决

问题

实际应用中，使用 airflow 运行 BashOperator 操作时，由于业务日志包含中文导致 airflow task log 卡住，任务其实是正常运行，但是没办法 hook task status,问题原因是 python 编码的问题，work 日志中会出现编码错误，如下：

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py", line 95, in _read_task_logs
    self.logger.info('Subtask: {}'.format(line.rstrip('\n')))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u6ce8' in position 55: ordinal not in range(128)

解决办法

1. 打开 base_task_runner.py 文件，vi /usr/lib/python2.7/site-packages/airflow/task_runner/base_task_runner.py

2. 将 self.logger.info('Subtask: {}'.format(line.rstrip('\n'))) 做如下修改

修改前：

 def _read_task_logs(self, stream):
        while True:
            line = stream.readline().decode('utf-8')
            if len(line) == 0:
                break
            self.logger.info('Subtask: {}'.format(line.rstrip('\n')))

修改后

 def _read_task_logs(self, stream):
        while True:
            line = stream.readline().decode('utf-8')
            if len(line) == 0:
                break
            self.logger.info(u'Subtask: {}'.format(line.rstrip('\n')))