Azkaban安装步骤
1.下载Azkaban的源码
wget https://github.com/azkaban/azkaban/archive/3.38.0.tar.gz
tar -zxvf azkaban-3.38.0.tar.gz -C /opt/source/
cd azkaban-3.38.0/
$>wget https://services.gradle.org/distributions/gradle-4.1-all.zip
包已下载
2.Azkaban的部署模式
2.1模式介绍
1.solo server mode,依赖自带的H2数据库,web server和executor server在同一个线程。测试或者小规模的案例可以用。
2.two sercer mode,需要mysql作为元数据库,web server和executor server在不同一个线程
3.multiple executor mode,2模式的扩展,可以有多个executor server
2.2安装使用第2种模式
2.3解压
解压azkaban-3.38.0.tar.gz到/opt/source,解压后目录结构说明如下所示
目录结构:
azkaban-common 常用的工具类(基础架构)
azkaban-db 与数据库相关的
azkaban-exec-server 执行的server
azkaban-hadoop-security-plugin hadoop安全的插件(不用去管它)
azkaban-solo-server 我们所要部署的模式
azkaban-spi 存储一些接口相关的东西(异常处理这些)
azkaban-web-server web-server(ui上的东西)
3.编译
3.1编译
3.x是需要编译,官方没有提供直接的安装包
Cp /gradle-4.1-all.zip /opt/source/azkaban-3.38.0/gradle/wrapper
$>cd /opt/source/azkaban-3.38.0/gradle/wrapper
$>vi gradle-wrapper.properties
distributionUrl=gradle-4.1-all.zip
#distributionUrl=https\://services.gradle.org/distributions/gradle-4.1-all.zip
3.2.build
编译,时间会比较慢
./gradlew installDist
如果不是第一次编译,最好先clean一下
./gradlew clean
编译完以后,我们需要四个目录,
azkaban-db,存放建表的语句
azkaban-exec-server,执行服务
azkaban-solo-server,solo模式使用
azkaban-web-server,web服务
4.创建Azkaban元数据库
sql脚本在azkaban-db文件中
mysql> use azkaban;
mysql> source [sql脚本目录]。有一个全的,名字里带all
5.配置 keystore
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Enter keystore password: 123456
what is your first and last name: fibonacci
what is the name of your organizational unit: xxx
what is the name of your organization: xxx
what is the name of your City or Locality: beijing
what is the name of your State or Province: beijing
what is the two_letter country code for this unit: CN
将生成的keystone文件拷贝到web-server的安装目录下,和conf等目录同级
6.创建必要目录
azkaban-exec-server 、Azkaban-web-server必须有以下目录
bin 启动脚本存放目录
conf 配置文件存放目录(没有的话从solo-server的目录中拷贝过来)
lib 依赖jar包存放目录extlib 附加jar包存放目录(没有的话手动创建)
plugins 插件安装目录web (没有的话从solo-server的目录中拷贝过来)
web 资源文件
logs 日志文件(手动创建)
2.Azkaban-exec-server
bin 启动脚本存放目录conf 配置文件存放目录(没有的话从solo-server的目录中拷贝过来)lib 依赖jar包存放目录extlib 附加jar包存放目录(没有的话手动创建)plugins 插件安装目录
以上的目录,没有的话从azkaban-solo-server里面拷贝
7.配置文件
7.1.azkaban-web-server azkaban.properties
路径/opt/source/azkaban-3.38.0/azkaban-web-server/build/install/azkaban-web-server/conf
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=h2
h2.path=./h2
h2.create.tables=true
# Velocity dev mode
velocity.dev.mode=false
# db
database.type=mysql
mysql.port=3306
mysql.host=10.15.107.12
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8058
# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
7.2 azkaban.properties配置描述
参考于网络
# Azkaban Personalization Settings
azkaban.name=Allin # 服务器UI名称,界面显示
azkaban.label=My Local Azkaban # 描述
azkaban.color=#FF3601 # 界面颜色
azkaban.default.servlet.path=/index
web.resource.dir=/azkaban-web-server/web/ # 最好用绝对路径,不然界面可能会诡异的丑
default.timezone.id=Asia/Shanghai
# 权限
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/azkaban-web-server/conf/azkaban-users.xml # 绝对路径,不然可能报错
# Loader for projects
executor.global.properties=/azkaban-web-server/conf/global.properties # 绝对路径,不然可能报错
azkaban.project.dir=/azkaban-web-server/projects # 绝对路径,不然可能报错
# Velocity dev mode
velocity.dev.mode=false
# jerry服务器的配置
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8443
jetty.keystore=//azkaban-web-server/conf/keystore # 用的是步骤5生成的文件
jetty.password=123456 # 密码都设一样的,也好记
jetty.keypassword=123456
jetty.truststore=//azkaban-web-server/conf/keystore
jetty.trustpassword=123456
# Azkaban Executor settings
executor.port=12321 # 执行服务器端口
# 邮件配置,job失败的话,会发邮件提醒,我没配置
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/azkaban-web-server/plugins/jobtypes
# mysql配置
database.type=mysql
mysql.port=3306
mysql.host=hadoop02
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
2.azkaban-web-server azkaban-users.xml
一看就知道怎么配置了
3.azkaban-exec-server azkaban.properties
# Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai
# Loader for projects
executor.global.properties=/azkaban-exec-server/conf/global.properties
azkaban.project.dir=/azkaban-exec-server/projects
azkaban.jobtype.plugin.dir=/azkaban-exec-server/plugins/jobtypes
database.type=mysql
mysql.port=3306
mysql.host=hadoop02
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50 # 最大线程数
executor.port=12321 # 端口号
executor.flow.threads=30 # 流式执行的线程数
4.log4j的配置
log4j.rootLogger=DEBUG,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
7.3 azkaban-exec-server azkaban.properties
路径:/opt/source/azkaban-3.38.0/azkaban-exec-server/build/install/azkaban-exec-server
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=h2
h2.path=./h2
h2.create.tables=true
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# db
database.type=mysql
mysql.port=3306
mysql.host=10.15.107.12
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=***@66.com
mail.host=10.8.***.***
mail.user=***@66.com
mail.password=***
mail.port=25
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes
8.启动
8.1 exec-server
先启动exec-server
bin目录下,azkaban-executor-start.sh
cd /opt/source/azkaban-3.38.0/azkaban-exec-server/build/install/azkaban-exec-server/
执行 bin/ azkaban-executor-start.sh
8.2 web-server
bin目录下,azkaban-web-start.sh
cd /opt/source/azkaban-3.38.0/azkaban-web-server/build/install/azkaban-web-server/
执行 bin/ azkaban-web-start.sh
8.3检查进程
Jsp
8.4.访问
http://10.15.107.153:8058