Azkaban安装手册
一、安装前准备
请确保upsql(mysql)、nginx安装成功
二、正式安装
1、上传安装包
将编译好的azkaban分别上传至服务器/APS/usr/vdsappas目录下
- 1）azkaban-db-0.1.0-SNAPSHOT.zip
- 2）azkaban-exec-server-0.1.0-SNAPSHOT.zip
- 3）azkaban-web-server-0.1.0-SNAPSHOT.zip
CDH客户端服务器列表：172.21.51.79 172.21.51.80
2、解压文件
unzip azkaban-db-0.1.0-SNAPSHOT.zip
unzip azkaban-exec-server-0.1.0-SNAPSHOT.zip
unzip azkaban-web-server-0.1.0-SNAPSHOT.zip
解压之后如下图
3、创建Azkaban数据库
创建azkaban数据库，并授权
# Example database creation command, although the db name doesn't need to be 'azkaban'
mysql> CREATE DATABASE azkaban;
# Example database creation command. The user name doesn't need to be 'azkaban'
mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
# Replace db, username with the ones created by the previous steps.
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
注意：如果你上传的任务包过大，这需要修改mysql的配置如下：
To configure this in linux, open /etc/my.cnf. Somewhere after mysqld, add the following:
[mysqld]
...
max_allowed_packet=1024M
重启数据库
创建azkaban表
将azkaban sql表结构导入mysql
mysql> use azkaban;
mysql> source /APS/usr/vdsappas/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql
创建好之后azkaban中的表如下图所示：
4、配置azkaban exec-server和web-server
分别在172.21.51.79和172.21.51.80的机器上配置azkaban的配置文件
cd /APS/usr/vdsappas/azkaban-exec-server-0.1.0-SNAPSHOT/conf
配置azkaban.properties
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
#jetty.use.ssl=false
#jetty.maxThreads=25
#jetty.port=8081
#jetty.maxThreads=25
#jetty.ssl.port=8443
#jetty.port=8081
#jetty.keystore=keystore
#jetty.password=123456
#jetty.keypassword=123456
#jetty.truststore=keystore
#jetty.trustpassword=123456
# Where the Azkaban web server is located
#web访问地址，现用nginx代理
azkaban.webserver.url=http://172.21.51.81:80
# mail settings
##用来配置邮件的发送
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
azkaban.native.lib=false
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/APS/usr/vdsappas/azkaban-exec-server-0.1.0-SNAPSHOT/plugins/jobtypes
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=60001
mysql.host=172.21.51.81
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30
executor.port=12321
cd /APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/conf
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/APS/usr/vdsappas/azkaban-web-server-0.1.0-SNAPSHOT/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
#jetty.use.ssl=false
#jetty.maxThreads=25
#jetty.port=8081
#jetty.maxThreads=25
#jetty.ssl.port=8443
#jetty.port=8081
#jetty.keystore=keystore
#jetty.password=123456
#jetty.keypassword=123456
#jetty.truststore=keystore
#jetty.trustpassword=123456
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
mail.user=
mail.password=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
azkaban.native.lib=false
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=60001
mysql.host=172.21.51.81
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
配置完成之后
进入mysql的数据库
insert into executors(host,port,active) values("172.21.51.79",12321,1);
insert into executors(host,port,active) values("172.21.51.80",12321,1);
# insert into executors(host,port) values("EXECUTOR_HOST",EXECUTOR_PORT);
一，首先启动的是
172.21.51.79 Azkaban Executor Server
172.21.51.80 Azkaban Executor Server
/APS/usr/vdsappas/azkaban-exec-server-0.1.0-SNAPSHOT /bin start-exec.sh
二，然后是
172.21.51.79 Azkaban Web Server
172.21.51.80 Azkaban Web Server
/APS/usr/vdsappas/azkaban-exec-server-0.1.0-SNAPSHOT /bin start-web.sh
5、激活Executor
curl http://172.21.51.79:12321/executor?action=activate
curl http://172.21.51.80:12321/executor?action=activate
6、配置nginx代理
以root用户进入nginx，修改niginx配置
vim /etc/nginx/nginx.conf
client_max_body_size 2G;
upstream azkaban_https {
ip_hash;
server 172.21.51.79:8443;
server 172.21.51.80:8443;
}
server {
listen 80;
server_name 172.21.51.81;
location /{
proxy_pass http://azkaban_https;
}
}
配置好之后，重启nginx
nginx -s reload
在浏览器输入http://172.21.51.81 如下图所示，输入用户名和密码azkaban/azkaban
问题：
解决 Azkaban Executor 启动后不会自动激活问题
2019/05/21 12:11:03.158 +0800 ERROR [ExecutorManager] [Azkaban] Rolling back executor assignment for execution id:5
azkaban.executor.ExecutorManagerException: java.io.IOException: executor became inactive before setting up the flow 5
at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:80)
at azkaban.executor.ExecutorApiGateway.callWithExecutable(ExecutorApiGateway.java:43)
at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1051)
at azkaban.executor.ExecutorManager.access$600(ExecutorManager.java:65)
at azkaban.executor.ExecutorManager$QueueProcessorThread.selectExecutorAndDispatchFlow(ExecutorManager.java:1282)
at azkaban.executor.ExecutorManager$QueueProcessorThread.processQueuedFlows(ExecutorManager.java:1254)
at azkaban.executor.ExecutorManager$QueueProcessorThread.run(ExecutorManager.java:1192)
Caused by: java.io.IOException: executor became inactive before setting up the flow 5
at azkaban.executor.ExecutorApiGateway.callForJsonObjectMap(ExecutorApiGateway.java:108)
at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:78)
... 6 more
2019/05/21 12:11:03.163 +0800 WARN [ExecutorManager] [Azkaban] Executor otest01:12321 (id: 5), active=true responded with exception for exec: 5
azkaban.executor.ExecutorManagerException: azkaban.executor.ExecutorManagerException: java.io.IOException: executor became inactive before setting up the flow 5
at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1057)
at azkaban.executor.ExecutorManager.access$600(ExecutorManager.java:65)
at azkaban.executor.ExecutorManager$QueueProcessorThread.selectExecutorAndDispatchFlow(ExecutorManager.java:1282)
at azkaban.executor.ExecutorManager$QueueProcessorThread.processQueuedFlows(ExecutorManager.java:1254)
at azkaban.executor.ExecutorManager$QueueProcessorThread.run(ExecutorManager.java:1192)
Caused by: azkaban.executor.ExecutorManagerException: java.io.IOException: executor became inactive before setting up the flow 5
at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:80)
at azkaban.executor.ExecutorApiGateway.callWithExecutable(ExecutorApiGateway.java:43)
at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1051)
... 4 more
Caused by: java.io.IOException: executor became inactive before setting up the flow 5
at azkaban.executor.ExecutorApiGateway.callForJsonObjectMap(ExecutorApiGateway.java:108)
at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:78)
... 6 more
各种方法都使用了但是也没有用，最后发现一篇大神写的博客后就解决了：
通过翻看 Azkaban Executor Server 源码,发现 Executor 提供了 rest 接口,来手动激活 Executor.
为什么不在 Executor 启动时自动激活呢….官方 issue110 解释如下:
We don't update prior to web-server start.
Let me introduce how we do a regular Executor deployment. Both old webserver and old executor(3.a) are up initially, and old executor is running jobs (we can not interrupt it). Executor Deployment starts:
install new Executor (3.b) on our box without activating.
Run job tests, which targets 3.b and make sure 3.b is working.
Activate 3.b, and deactivate 3.a. Even though 3.a is deactivated, jobs still keep runnig there.
Deployment completes.
We would better run tests against new executor before activating it. Is it a bit clear now?
解决办法（手动激活 Executor）：
curl http://${executorHost}:${executorPort}/executor?action=activate