azkaban集群安装配置

azkaban集群安装配置

官方文档地址https://azkaban.readthedocs.io/en/latest/

1 安装

1.1 下载源码

https://github.com/azkaban/azkaban/releases

1.2 编译

Azkaban构建使用Gradle，并且需要Java8或更高版本。

#运行其自带打包工具但需要先cd到文件夹内
cd /opt/azkaban-3.79.0/
#运行自带方法 Build Azkaban
./gradlew build

编译完成后, 各组件的build目录下, 会生成相应的.tar.gz文件. 例如, ./azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz.
组件列举如下:

文件夹	功能
azkaban-exec-server	核心之一, (分布式)执行器
azkaban-web-server	核心之二, 网页服务
azkaban-db	数据库工具
azkaban-solo-server	单机版, 用于学习基本操作
azkaban-common	基本的依赖包
azkaban-spi	azkaban存储接口以及exception类
azkaban-hadoop-security-plugin	hadoop 有关kerberos插件

为了便于操作接下分别把压缩包移到/opt/下新创建的azkaban文件夹里

mkdir /opt/azkaban-3.79.0
 
mv  azkaban-3.79.0/azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz  /opt/azkaban-3.79.0/
 
mv  azkaban-3.79.0/azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz  /opt/azkaban-3.79.0/
 
mv  azkaban-3.79.0/azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz  /opt/azkaban-3.79.0/
 
mv  azkaban-3.79.0/azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz  /opt/azkaban-3.79.0/

注意编译过程需要联网

2 配置Mysql

Azkaban的元数据是交给Mysql进行管理的，我们需要在Mysql中建立专用的数据库用来
保存Azkaban产生的元数据。具体步骤如下：

2.1 建库

#使用root权限进入数据库
mysql -uroot -p

#创建azkaban数据库
create database azkaban

#进入到azkaban数据库
use azkaban

#创建Azkaban数据库专用的用户
CREATE USER 'azkaban'@'%' IDENTIFIED BY 'Azkaban123'; 

#授权
GRANT ALL ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION; 

#刷新权限
FLUSH PRIVILEGES; 

2.2 导入数据

首先解压azkaban-db-0.1.0-SNAPSHOT.tar.gz包，该包下包含一些mysql数据库的建表语句

#进入azkaban数据库
use azkaban
#导入数据
source /opt/azkaban-3.79.0/azkaban-db/create-all-sql-0.1.0-SNAPSHOT.sql

3 配置WebServer:

3.1 解压

解压azkaban-web-server-0.1.0-SNAPSHOT.tar.gz包到指定服务器和指定目录并且重命名为azkaban-web-server-3.62.0-7，webServer目录结构为:

drwxr-xr-x. 3 root root   65 Nov  4 11:41 bin
drwxr-xr-x. 2 root root  106 Nov  4 11:41 conf
drwxr-xr-x. 2 root root 4096 Nov  4 11:41 lib
drwxr-xr-x. 6 root root   73 Nov  4 11:42 web

3.2 配置azkaban.properties

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=linux01
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

3.3 配置用户

配置conf/azkaban-user.xml

4 配置ExecutorServer

4.1 解压

解压azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz到指定目录下，并且重命名为：azkaban-exec-server-3.62.0-7它的目录结构为：

[root@linux01 azkaban-exec-server-0.1.0-SNAPSHOT]# ll
total 4
drwxr-xr-x. 3 root root   67 Nov  4 11:43 bin
drwxr-xr-x. 2 root root   81 Nov  4 11:43 conf
drwxr-xr-x. 2 root root 4096 Nov  4 11:43 lib
drwxr-xr-x. 3 root root   22 Nov  4 11:43 plugins

4.2 修改conf/azkaban.properties

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Where the Azkaban web server is located
azkaban.webserver.url=http://localhost:8081
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=linux01
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30

5 启动Azkaban

5.1 启动顺序

azkaban集群模式必须先启动executor-server,再启动webServer,不然一定无法运行

5.2 启动exec-server

./bin/start-exec.sh
#注意azkaban在启动时默认不会激活executor
#需要手动激活
#执行以下命令激活executor
[root@linux01 azkaban-exec-server]# curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo
{"status":"success"}
#或者进入到Mysql数据库中  更改executors表中的active字段为1 进行激活
./bin/start-web.sh

5.3 启动web-server

cd /soft/azkaban/azkaban-web-server/
./bin/start-web.sh

登陆 web服务器地址:8081进行登陆,至此azkaban的搭建完成。