Azkaban 简介及安装(3.70)
Azkaban 简介及安装(3.70)
1.Azkaban 简介
官网：https://azkaban.github.io/
Azkaban 一个批量工作流任务调度器，使用Java语言开发。用于在一个工作流内以一个特定的顺序运行一组工作和流程。Azkaban使用job配置文件建立任务之间的依赖关系，并提供一个易于使用的web用户界面维护和跟踪你的工作流。
Azkaban 的优点
-
提供功能清晰，简单易用的Web UI界面
-
提供job配置文件快速建立任务和任务之间的依赖关系
-
提供模块化和可插拔的插件机制，原生支持command、Java、Hive、Pig、Hadoop
-
基于Java开发，代码结构清晰，易于二次开发
Azkaban 安装模式
Azkaban 有三种部署方式：单服务模式、2个服务模式、分布式多服务模式
-
solo server model（单服务模式）：该模式中 webServer 和 executorServer 运行在同一个进程中，进程名是AzkabanSingleServer。可以使用自带的H2数据库或者配置mysql数据。该模式适用于小规模的使用。此模式合适个人试用使用。它也可以用在小规模的使用案例中。
-
two server model（2个服务模式）：数据库为mysql，采用主从设置进行备份，管理服务器（webServer）和执行服务器（executorServer）在不同进程中运行，这种模式下，管理服务器和执行服务器互不影响。适用在较重的生成环境中。
-
multiple-executor（分布式多服务模式）：存放元数据的数据库为mysql，采用主从设置进行备份，管理服务器（webServer）和执行服务器（executorServer）在不同进程中运行。多个 Executor 模式为最重的生产环境。
2.Azkaban 3.x 安装(two-server 模式)
安装前准备
- 安装JDK1.8
- 安装MySQL
下载源码
git clone https://github.com/azkaban/azkaban.git
编译源码
cd azkaban; ./gradlew build installDist
编译报错：
> Task :az-crypto:test
azkaban.crypto.DecryptionTest > testV1_1 FAILED
java.lang.RuntimeException: java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
at azkaban.crypto.Crypto.decrypt(Crypto.java:76)
at azkaban.crypto.DecryptionTest.testV1_1(DecryptionTest.java:35)
Caused by:
java.lang.RuntimeException: org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:57)
at azkaban.crypto.Crypto.decrypt(Crypto.java:74)
... 1 more
Caused by:
org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:1050)
at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:725)
at azkaban.crypto.CryptoV1_1.decrypt(CryptoV1_1.java:55)
... 2 more
azkaban.crypto.EncryptionTest > testEncryption FAILED
org.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine
at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:1073)
at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.encrypt(StandardPBEByteEncryptor.java:924)
at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.encrypt(StandardPBEStringEncryptor.java:642)
at azkaban.crypto.CryptoV1_1.encrypt(CryptoV1_1.java:42)
at azkaban.crypto.Crypto.encrypt(Crypto.java:58)
at azkaban.crypto.EncryptionTest.testEncryption(EncryptionTest.java:28)
5 tests completed, 2 failed
解决方法：
1.下载jre：https://www.oracle.com/technetwork/cn/java/javase/downloads/jce8-download-2133166-zhs.html
2.在/usr/java/jdk1.8.0_141-cloudera/jre/lib/security目录下替换原有的
3.重新编译 ./gradlew build installDist
启停单机版的服务
cd azkaban-solo-server/build/install/azkaban-solo-server
bin/start-solo.sh
bin/shutdown-solo.sh
数据库设置
# 建用户 建库 授权 刷新权限
CREATE DATABASE azkaban_zxl;
CREATE USER 'azkaban_zxl'@'%' IDENTIFIED BY 'azkaban_zxl';
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban_zxl.* to 'azkaban_zxl'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
# 在/etc/my.cnf中配置Mysql Packet Size的大小
[mysqld]
...
max_allowed_packet=1024M
# 重启mysql
/sbin/service mysqld restart
# 创建Azkaban表
source /root/apps/create-all-sql.sql
配置web-server
vi /root/azkaban/azkaban-web-server/conf/azkaban.properties
default.timezone.id=Asia/Shanghai
#database.type=h2
#h2.path=./h2
#h2.create.tables=true
database.type=mysql
mysql.port=3306
mysql.host=10.xxx.xxx.xx
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
executor.port=12321
azkaban.use.multiple.executors=true
将azkaban-web-server/conf/azkaban.properties 同步到 /root/azkaban/azkaban-exec-server/conf下
启动分布式多服务模式
多服务模式是多个executorServer分布在 不同服务器上，只需要将/root/azkaban/azkaban-exec-server拷贝到不同机器上即可组成分布式。
先启动executorServer bin/start-exec.sh, 再启动webServer bin/start-web.sh。
**Note:**启动executorServer时会将executorServer的host,port同步到mysql的executors表中;
root@localhost|azkaban_zxl>select * from executors;
+----+---------------------------------------+-------+--------+
| id | host | port | active |
+----+---------------------------------------+-------+--------+
| 1 | 10.xxx.xx.xx | 12321 | 0 |
| 2 | 10.xxx.xx.xx | 12321 | 0 |
+----+---------------------------------------+-------+--------+
启动webServer时会如下错误，是因为自动同步到executors表中的executor默认是未激活，需要在数据库中update：
2019/03/22 10:00:47.210 +0800 INFO [ExecutorManager] [Azkaban] Initializing executors from database.
2019/03/22 10:00:47.418 +0800 ERROR [ExecutorManager] [Azkaban] No active executors found
2019/03/22 10:00:47.418 +0800 ERROR [StdOutErrRedirect] [Azkaban] Exception in thread "main"
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban] azkaban.executor.ExecutorManagerException: No active executors found
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban] at azkaban.executor.ActiveExecutors.setupExecutors(ActiveExecutors.java:52)
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban] at azkaban.executor.ExecutorManager.setupExecutors(ExecutorManager.java:231)
2019/03/22 10:00:47.419 +0800 ERROR [StdOutErrRedirect] [Azkaban] at azkaban.executor.ExecutorManager.initialize(ExecutorManager.java:155)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban] at azkaban.executor.ExecutorManager.start(ExecutorManager.java:169)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban] at azkaban.webapp.AzkabanWebServer.launch(AzkabanWebServer.java:235)
2019/03/22 10:00:47.420 +0800 ERROR [StdOutErrRedirect] [Azkaban] at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:228)
root@localhost|azkaban_zxl>update executors set active=1 where id=1;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
root@localhost|azkaban_zxl>update executors set active=1 where id=2;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
executor重启方式
1. 停止webServer bin/shutdown-web.sh
2. 停止executor bin/shutdown-exec.sh
3. 删除 xxx/azkaban-exec-server-3.65.0/currentpid 文件
4. 启动executor bin/start-exec.sh
5. 去数据库中确认 executor状态为1
6. 启动webServer bin/start-web.sh
shylin