Azkaban安装过程分享-附《Azkaban技术分享手册》

以azkaban2.5为例:

1、MySQL安装与配置
sudo service mysqld start
mysql -u root -p

a、为azkaban单独创建一个数据库:
CREATE DATABASE azkaban;

b、单独创建一个数据库用户
为了简单,我直接用root/hadoop%123,在生产上大家最好单独建一个用户

c、为azkaban建表:
下载azkaban-sql-script-2.5.0.tar.gz,解压缩
执行create-all-sql-2.5.0.sql
mysql -u root -phadoop%123 -Dazkaban

Azkaban Jetty server properties.

jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=root%123
jetty.keypassword=root%123
jetty.truststore=keystore
jetty.trustpassword=root%123

c、Setting up the DB
修改conf/azkaban.properties

database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=hadoop%123
mysql.numconnections=100

d、Setting up the UserManager
修改conf/azkaban.properties
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml

e、Running Web Server

bin/azkaban-web-start.sh
bin/azkaban-web-shutdown.sh

https://10.128.7.241:8443/

3、 配置Azkaban Executor Server
a、安装 Azkaban Executor Server
下载azkaban-executor-server-2.5.0.tar.gz,解压缩即可
b、Setting up the DB
修改conf/azkaban.properties

database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=hadoop%123
mysql.numconnections=100

c、Configuring AzabanWebServer and AzkabanExecutorServer clients
The Executor server needs to be setup with a port, and the AzabanWebServer will need to know what this port is.

修改conf/azkaban.properties

Azkaban Executor settings

executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

这里要注意以下,两种不同的模式,配置是不一样的:

Single Executor Mode:
executor.port=12321就可以拉(只要跟AzabanWebServer的azkaban.properties下的executor.port保持一致即可)

d、Running Executor Server

bin/azkaban-exec-start.sh
bin/azkaban-exec-shutdown.sh

4、多Executor Server模式:
a、启动Multiple Executor Mode
修改AzabanWebServer的azkaban.properties的以下属性:
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
b、同时,你还得在多个节点上运行Executor Server

c、把多个Executor Server的host和port写到数据库里

注意:当前版本还没有UI管理接口去管理executor,只能把多个executor写到数据库的executors表,webserver就能找到他们拉。
insert into executors(host,port) values(“EXECUTOR_HOST”,EXECUTOR_PORT);

5、配置Azkaban插件
阿兹卡班设计的思路的是使非核心功能基于插件,所以核心功能非常轻便,安装升级方便,而且很容易扩展到不同的操作系统。

web server插件:

viewer plugins:
that enable custom web pages to add features to Azkaban. Some of the known implementations include HDFS filesystem viewer, and Reportal.
trigger plugins:
that enable custom triggering methods.
user manager plugin :
that enables custom user authentication methods. For instance, in LinkedIn we have LDAP based user authentication.
alerter plugins :
that enable different alerting methods to users, in addition to email based alerting.

executor server插件:
pluggable job type executors on AzkabanExecutorServer, such as job types for hadoop ecosystem components.

a、User Manager Plugins

修改webserver的azkaban.properties:
user.manager.class=MyUserManagerClass

把插件包含的jar放到plugins directory。
需要Azkaban技术分享手册的,点击以下链接下载
http://www.dajiangtai.com/community/18218.do

你可能感兴趣的:(Hadoop)