Azkaban-3.x two-server 模式搭建

---

环境准备

• Azkaban安装包
  azkaban-db-0.1.0-SNAPSHOT.tar.gz
  azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
  azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
  参考Azkaban-3.x 源码编译
• JDK 1.8+
• MySQL，可以参考以下两篇文章进行安装：
  CentOS-7(64位minimal版)安装MySQL-5.7(64位glibc版)
  Linux中安装MySQL

---

1. two-server模式简介

web-server和executor-server是两个单独的进程，但都运行在同一个机器上，调度任务只能分配到该机器上执行，元数据存储在MySQL中，MySQL可以在其他机器上。

2. two-server模式安装

(1) web-server 安装

<1> 数据库准备

首先解压azkaban-db-0.1.0-SNAPSHOT.tar.gz安装包，改名为azkaban-db，里面存放着初始化数据库的SQL脚本

MariaDB [(none)]> CREATE DATABASE azkaban;
MariaDB [(none)]> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
MariaDB [(none)]> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> USE azkaban;

# 执行的create-all.sql在azkaban-db.tar.gz解压后的目录中
MariaDB [azkaban]> source /opt/beh/core/azkaban-3.76.0/azkaban-db/create-all-sql-0.1.0-SNAPSHOT.sql

MariaDB [azkaban]> show tables;
+--------------------------+
| Tables_in_azkaban        |
+--------------------------+
| QRTZ_BLOB_TRIGGERS       |
| QRTZ_CALENDARS           |
| QRTZ_CRON_TRIGGERS       |
| QRTZ_FIRED_TRIGGERS      |
| QRTZ_JOB_DETAILS         |
| QRTZ_LOCKS               |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE     |
| QRTZ_SIMPLE_TRIGGERS     |
| QRTZ_SIMPROP_TRIGGERS    |
| QRTZ_TRIGGERS            |
| active_executing_flows   |
| active_sla               |
| execution_dependencies   |
| execution_flows          |
| execution_jobs           |
| execution_logs           |
| executor_events          |
| executors                |
| project_events           |
| project_files            |
| project_flow_files       |
| project_flows            |
| project_permissions      |
| project_properties       |
| project_versions         |
| projects                 |
| properties               |
| triggers                 |
+--------------------------+
29 rows in set (0.000 sec)

执行完create-all.sql脚本后，检查一下有没有创建失败的表，要注意的是在sql脚本中，有一些字段varchar类型长度超过了mysql的限制，所以创建表的时候可能报错（不是一定的，与MySQL以及Azkaban的版本都有关系），如果报错，把varchar字段长度修改为128即可。

<2> 生成 keystore 和 truststore 文件

参考 Azkaban-3.x 生成 keystore 和 truststore 文件

解压azkaban-web-server-0.1.0-SNAPSHOT.tar.gz安装包，改名为azkaban-web-server，在其中创建key目录，将生成的3个文件放到key目录下

<3> 修改 conf/azkaban.properties

修改azkaban.properties(无论是web-server的还是exec-server的)，都要注意以下几点

1. 每行的末尾不要留空格
2. 时区一定是Asia/Shanghai
3. 所有的路径写成绝对路径
4. 各个配置项参考Azkaban-3.x 配置信息说明

azkaban.name=azkaban-3.76.0
azkaban.label=two-server-mode
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
azkaban.project.dir=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/projects
azkaban.jobtype.plugin.dir=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/plugins/jobtypes
web.resource.dir=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/web
default.timezone.id=Asia/Shanghai
viewer.plugin.dir=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/plugins/viewer
cache.directory=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/cache
velocity.dev.mode=false

azkaban.use.multiple.executors=true

user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/conf/azkaban-users.xml
lockdown.create.projects=false
lockdown.upload.projects=false
project.temp.dir=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/temp

executor.port=12321

jetty.port=8081
jetty.use.ssl=true
jetty.ssl.port=8443
jetty.maxThreads=25
jetty.keystore=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/key/keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/key/truststore
jetty.trustpassword=azkaban
jetty.connector.stats=true

database.type=mysql
mysql.port=3306
mysql.host=beh07
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

# mail.sender=
# mail.host=
# mail.user=
# mail.password=
# job.failure.email=
# job.success.email=

说明：azkaban.use.multiple.executors=true这一行必须配置，因为3.76.0版本(我不确认是从哪个版本开始)已经不支持two-server模式了，不加这一行启动web-server会报错：

2019/08/23 15:08:44.437 +0800 ERROR [StdOutErrRedirect] [Azkaban] java.lang.IllegalArgumentException: azkaban.use.multiple.executors must be true. Single executor mode is not supported any more.

我们虽然开启了多executor模式，但是只启动一个executor，并且executor和web-server在同一台机器上，这也就相当于是two-server模式了。

<4> 修改 conf/log4j.properties

将其中的存放日志的路径由相对路径改为绝对路径，logs目录手动创建

log4j.appender.server.File=/opt/beh/core/azkaban-3.76.0/azkaban-web-server/logs/azkaban-webserver.log

(2) exec-server 安装

解压azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz安装包，改名为azkaban-exec-server

<1> 修改conf/azkaban.properties

azkaban.webserver.url=https://beh07:8443
azkaban.jobtype.plugin.dir=/opt/beh/core/azkaban-3.76.0/azkaban-exec-server/plugins/jobtypes
azkaban.project.dir=/opt/beh/core/azkaban-3.76.0/azkaban-exec-server/projects
default.timezone.id=Asia/Shanghai

database.type=mysql
mysql.port=3306
mysql.host=beh07
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

executor.port=12321
executor.connector.stats=true
executor.maxThreads=50
executor.flow.threads=30
executor.global.properties=/opt/beh/core/azkaban-3.76.0/azkaban-exec-server/conf/global.properties

<2> 修改 conf/log4j.properties

将其中的存放日志的路径由相对路径改为绝对路径，logs目录手动创建

log4j.appender.server.File=/opt/beh/core/azkaban-3.76.0/azkaban-exec-server/logs/azkaban-execserver.log

3. 测试

(1) 必选先启动exec-server

[hadoop@beh07 azkaban-exec-server]$ bin/start-exec.sh

(2) 查看数据库的executors表

MariaDB [azkaban]> select * from executors;
+----+-------+-------+--------+
| id | host  | port  | active |
+----+-------+-------+--------+
|  1 | beh07 | 12321 |      0 |
+----+-------+-------+--------+

# 这里必须手动把active的值设置为1
MariaDB [azkaban]> UPDATE executors SET active = 1 WHERE id = 1;

(3) 启动web-server

[hadoop@beh07 azkaban-web-server]$ bin/start-web.sh

(4) 查看进程

[hadoop@beh07 azkaban-web-server]$ jps
13096 AzkabanWebServer
13130 Jps
13051 AzkabanExecutorServer

(5) 访问 WEB-UI

注意URL是https://hostname:8443

使用azkaban用户登录，密码是azkaban

(6) 提交一个测试任务

<1> 创建一个文件，名为hello.job，其中的内容如下：

type=command
command=echo "hello azkaban"

<2> 将文件打成.zip包

<3> 进行以下操作

<4> 查看调度页面

发现任务总是运行失败：

去看executor的日志发现：

2019/08/23 15:43:30.713 +0800 ERROR [ExecutorServlet] [Azkaban] executor became inactive before setting up the flow 2
azkaban.executor.ExecutorManagerException: executor became inactive before setting up the flow 2
        at azkaban.execapp.FlowRunnerManager.createFlowRunner(FlowRunnerManager.java:397)
        at azkaban.execapp.FlowRunnerManager.submitFlow(FlowRunnerManager.java:341)
        at azkaban.execapp.ExecutorServlet.handleAjaxExecute(ExecutorServlet.java:288)
        at azkaban.execapp.ExecutorServlet.handleRequest(ExecutorServlet.java:136)
        at azkaban.execapp.ExecutorServlet.doPost(ExecutorServlet.java:93)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

executor又变成未激活状态了，我们只能手动激活：

hadoop@beh07 azkaban-3.76.0]$ curl http://beh07:12321/executor?action=activate
{"status":"success"}

然后再查看调度页面：

发现任务已经再持续成功地调度了，关于手动激活executor的问题，官网的解释是：

We don't update prior to web-server start.

Let me introduce how we do a regular Executor deployment. Both old webserver and old executor(3.a) are up initially, and old executor is running jobs (we can not interrupt it). Executor Deployment starts:

install new Executor (3.b) on our box without activating.
Run job tests, which targets 3.b and make sure 3.b is working.
Activate 3.b, and deactivate 3.a. Even though 3.a is deactivated, jobs still keep runnig there.
Deployment completes.
We would better run tests against new executor before activating it. Is it a bit clear now?

也就是说，我们只能手动的激活Executor。

(7) 取消测试的调度任务，关闭服务

hadoop@beh07 azkaban-exec-server]$ bin/shutdown-exec.sh 
Killing executor. [pid: 13051], attempt: 1
shutdown succeeded
[hadoop@beh07 azkaban-web-server]$ bin/shutdown-web.sh 
Killing web-server. [pid: 13194], attempt: 1
shutdown succeeded

至此，Azkaban-3.x two-server 模式就搭建成功了！