azkaban 安装与使用 （阅过 无数网友博客 后总结）

一、环境

1. centos7、hadoop2.5.0  、jdk1.8  、hive 0.13.1、 sqoop_146_h204、mysql  5.7.24 ....

2. 其中 jdk必须用jdk1.8或更新

3. azkaban executor 和 webserver 在同一个节点

二、azkaban 安装部署

1. 安装git: yum -y install git

2. wget https://services.gradle.org/distributions/gradle-4.6-all.zip

3. 获取azkaban安装包，wget https://github.com/azkaban/azkaban/archive/master.zip

4. 解压后执行下列命令：注意当前所在目录

先更改这个文件属性： gradle/wrapper/gradle-wrapper.properties

最后一项 distributionUrl=gradle-4.6-all.zip

./gradlew clean
./gradlew distTar

5. 编译完成后，获得

azkaban-db.tar.gz

 webserver.tar.gz

 executor.tar.gz

这是经过我改名后的。

6. 解压到azkaban-soft 文件夹

7.mysql 中执行： 

CREATE DATABASE azkaban;
CREATE USER 'azkaban_User_cys'@'％'IDENTIFIED BY '321456';    #创建用户，授权用户
use azkaban

source /opt/modules/azkaban/azkaban-soft/db/create-all-sql-0.1.0-SNAPSHOT.sql;

设置所有node的状态为1
update executors set active=1;

8. web安装目录下执行：keytool -keystore keystore -alias jetty -genkey -keyalg RSA

设置时区 ，这一步很重要keytool工具在jdk包中，不用安装。

执行命令，keytool -keystore keystore -alias jetty -genkey -keyalg RSA

输入密码

9. 修改配置参数

先executor 

conf文件夹下：azkaban.properties

# Azkaban Personalization Settings
# 可以将其他的属性注释掉或者删除，这里只配置时区这一项
default.timezone.id=Asia/Shanghai

# Azkaban UserManager class
# Loader for projects
executor.global.properties=/opt/modules/azkaban/azkaban-soft/executor/conf/global.properties  #绝对路径
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
# mail settings
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30
# 这里添加一个端口配置
executor.port=12321
azkaban.execution.dir=/opt/modules/azkaban/azkaban-soft/executor/executions#绝对路径
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/opt/modules/azkaban/azkaban-soft/executor/plugins/jobtypes#绝对路径
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=192.168.135.135  #你mysql的节点 ip
mysql.database=azkaban #mysql中 你创建的数据库database
mysql.user=root
mysql.password=321456
mysql.numconnections=100

10 再配置webserver

conf文件夹下：azkaban.properties

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/opt/modules/azkaban/azkaban-soft/webserver/conf/azkaban-users.xml
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=MyAzkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/opt/modules/azkaban/azkaban-soft/webserver/web/
default.timezone.id=Asia/Shanghai
# Loader for projects
executor.global.properties=/opt/modules/azkaban/azkaban-soft/webserver/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=true
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=/opt/modules/azkaban/azkaban-soft/webserver/keystore
jetty.password=321456
jetty.keypassword=321456
jetty.truststore=/opt/modules/azkaban/azkaban-soft/webserver/keystore
jetty.trustpassword=321456

project.temp.dir=/opt/modules/azkaban/azkaban-soft/webserver/temp
project.version.retention=3

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
azkaban.jobtype.plugin.dir=/opt/modules/azkaban/azkaban-soft/webserver/plugins/jobtypes
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=321456
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
 

==========================

 附加安装：yum install -y nodejs

 

在azkaban库中添加executors的ip/域名和端口        
insert into executors(host,port)values("192.168.135.135",3666);
insert into executors(host,port)values("192.168.135.145",3667);
insert into executors(host,port)values("192.168.135.155",3668);

 

/opt/install/hadoop-2.5.0/share/hadoop/common/hadoop-common-2.5.0.jar

11. 启动和关闭

webserver/bin/shutdown-web.sh
executor/bin/shutdown-exec.sh

executor/bin/start-exec.sh
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo
webserver/bin/start-web.sh

请记住通过调用以下方法激活执行程序：
cd azkaban-exec-server/build/install/azkaban-exec-server
curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo  每次启动都要执行

update executors set active=1 where id=2;

更改时区
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

遇见故障：执行flow 一直处于preparing 状态
可能原因：

在准备执行flow时，没有指定 useExecutor。
并且多个执行服务器上的内存小于6G或者CPU使用率超过了95%
当指定了useExecutor时，并且设置了内存不检查则可以执行的，小心不要写错了单词哦，也不可以吧这个int类型的留空或者字符串java.lang.NumberFormatException。
查看这个问题的原因，处理解决完上面的三个步骤
还要把MySQL中active_executing_flows表的update_time=’0’ 数据删除。

并且execution_flows表中executor_id=null的也需要删除。

在 flow params中设置”useExecutor” = EXECUTOR_ID即可。

EXECUTOR_ID为executors表的主键。

delete from active_executing_flows where exec_id = 11;