1.下载 azkaban3.38.0压缩包:https://github.com/azkaban/azkaban/releases/tag/3.38.0
2.解压后文件夹重命名为 azkaban
3.在 azkaban 目录下build
./gradlew clean
./gradlew distTar
4.在 azkaban 文件夹同级目录下新建 azkaban3.38.0文件夹,并分别将 azkaban-db、azkaban-exec-server、azkaban-web-server中,/build/distributions下的 tar.gz 拷贝到 azkaban3.38.0文件夹中,并用 tar -zxvf 命令解压
5.将azkaban-solo-server/build/distributions下的 tar.gz 文件解压,并将temp 和 plugins 文件夹用 cp -r 命令拷贝到 azkaban3.38.0/azkaban-exec-server和azkaban3.38.0/azkaban-web-server文件夹下
*******之后全对 azkaban3.38.0文件夹下文件操作*******
6.1修改 exec 和 web 文件夹下 conf 文件夹中的 azkaban.properties 文件
# database.type=h2
# h2.path=./h2
# h2.create.tables=true
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100
6.2两个文件夹都增加 log4j.properties
log4j.rootLogger=INFO,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
7.在 web/conf 下添加keystore
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
然后随意
8.在 exec和 web 路径下新建 extlib 文件夹,把 lib 文件夹下的 mysql jar 包拷贝到 extlib 中
9.mysql 新建用户,配置信息和 azkaban.properties中的配置统一,并赋权限
10.在 azkaban 数据库中,执行 source /path/create-all-sql-0.1.0-SNAPSHOT.sql
11.先执行 exec-start.sh,后开启 web-start.sh,注意都使用 bin/*-start.sh 命令,*是什么都懂的
12.页面打开 localhost:8081
遇到问题:
1.没找到 hbase 的 jar 包???
解决:不要 source /etc/profile,或者别写 HBASE_HOME
2.在页面新建任务后,上传 zip 包,失败
原因:azkaban 把上传的 zip 包扔到 mysql 库里了,很蠢
解决:用 /etc/my.cnf 修改 mysql 最大上传包大小,mac 下一般没有这个文件,自己 vim 一个
# Example MySQL config file for medium systems.
#
# This is for a system with little memory (32M - 64M) where MySQL plays
# an important part, or systems up to 128M where MySQL is used together with
# other programs (such as a web server)
#
# MySQL programs look for option files in a set of
# locations which depend on the deployment platform.
# You can copy this option file to one of those
# locations. For information about these locations, see:
# http://dev.mysql.com/doc/mysql/en/option-files.html
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.
# The following options will be passed to all MySQL clients
[client]
default-character-set=utf8
#password = your_password
port = 3306
socket = /tmp/mysql.sock
# Here follows entries for some specific programs
# The MySQL server
[mysqld]
character-set-server=utf8
init_connect='SET NAMES utf8
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 999M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
character-set-server=utf8
init_connect='SET NAMES utf8'
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking
# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin
# binary logging format - mixed recommended
binlog_format=mixed
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=, MASTER_PORT=,
# MASTER_USER=, MASTER_PASSWORD= ;
#
# where you replace , , by quoted strings and
# by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host =
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user =
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port =
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /usr/local/mysql/data
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /usr/local/mysql/data
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 999M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
default-character-set=utf8
[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
max_allowed_packet的值自己根据情况改,然后重启 mysql
修改命令:set global max_allowed_packet = 2*1024*1024*100;
查看命令:show VARIABLES like '%max_allowed_packet%';
3.在页面如何用 flow 处理多个任务
解决:写多个 job 文件,然后用 dependencies 属性,指定这个 job 的前提任务,然后将几个.job 文件还有打好的spark jar 包都打到一个 zip 包里,提交到页面上一个 project 里