Hadoop生态圈-Azkaban实战之Command类型多job工作流flow

　　　　　　　　　　　Hadoop生态圈-Azkaban实战之Command类型多job工作流flow

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

 　　

　　Azkaban内置的任务类型支持command、java。

 

一.Command类型单一job示例

1>.创建job描述文件

 

2>.将job资源文件打包成zip文件

 

3>.通过azkaban的web管理平台创建project并上传job压缩包（首先创建project）

4>.上传zip包

 

5>.启动执行该job

6>.设置周期任务并开始执行job任务

 7>.工程变为绿色说明job执行成功

 

 8>.查看job list

 

二.Command类型多job工作流flow

1>.创建有依赖关系的多个job描述

[yinzhengjie@s101 ~]$ more foo.job 
# foo.job
type=command
command=echo foo
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ more bar.job 
# bar.job
type=command
dependencies=foo
command=echo bar
[yinzhengjie@s101 ~]$ 

2>.将所有job资源文件打到一个zip包中

[yinzhengjie@s101 ~]$ yum -y install zip
Loaded plugins: fastestmirror
You need to be root to perform this command.
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ 
[yinzhengjie@s101 ~]$ sudo yum -y install zip
[sudo] password for yinzhengjie: 
Loaded plugins: fastestmirror
base                                                                                                                                                                                                                | 3.6 kB  00:00:00     
epel/x86_64/metalink                                                                                                                                                                                                | 6.3 kB  00:00:00     
epel                                                                                                                                                                                                                | 3.2 kB  00:00:00     
extras                                                                                                                                                                                                              | 3.4 kB  00:00:00     
updates                                                                                                                                                                                                             | 3.4 kB  00:00:00     
epel/x86_64/primary            FAILED                                                                   0% [                                                                                             ]  0.0 B/s |    0 B  --:--:-- ETA 
http://mirrors.aliyun.com/epel/7/x86_64/repodata/d214b3e89f48b6e4033ee71f60676d1c912c1a2726241f62de9181243dacaca1-primary.xml.gz: [Errno 14] HTTP Error 404 - Not Found                                  ]  0.0 B/s |    0 B  --:--:-- ETA 
Trying other mirror.
To address this issue please refer to the below knowledge base article 

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/

(1/3): epel/x86_64/updateinfo                                                                                                                                                                                       | 925 kB  00:00:07     
(2/3): updates/7/x86_64/primary_db                                                                                                                                                                                  | 2.7 MB  00:00:23     
(3/3): epel/x86_64/primary                                                                                                                                                                                          | 3.5 MB  00:00:21     
Determining fastest mirrors
 * base: mirror.bit.edu.cn
 * epel: mirrors.tongji.edu.cn
 * extras: mirror.bit.edu.cn
 * updates: mirror.bit.edu.cn
epel                                                                                                                                                                                                                           12601/12601
Resolving Dependencies
--> Running transaction check
---> Package zip.x86_64 0:3.0-11.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================================================================================================================
 Package                                               Arch                                                     Version                                                       Repository                                              Size
===========================================================================================================================================================================================================================================
Installing:
 zip                                                   x86_64                                                   3.0-11.el7                                                    base                                                   260 k

Transaction Summary
===========================================================================================================================================================================================================================================
Install  1 Package

Total download size: 260 k
Installed size: 796 k
Downloading packages:
zip-3.0-11.el7.x86_64.rpm                                                                                                                                                                                           | 260 kB  00:00:03     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : zip-3.0-11.el7.x86_64                                                                                                                                                                                                   1/1 
  Verifying  : zip-3.0-11.el7.x86_64                                                                                                                                                                                                   1/1 

Installed:
  zip.x86_64 0:3.0-11.el7                                                                                                                                                                                                                  

Complete!
[yinzhengjie@s101 ~]$ 

安装zip软件包（[yinzhengjie@s101 ~]$ yum -y install zip）

[yinzhengjie@s101 data]$ cat /home/yinzhengjie/bar.job 
# bar.job
type=command
dependencies=foo
command=echo bar
[yinzhengjie@s101 data]$ cat /home/yinzhengjie/foo.job 
# foo.job
type=command
command=echo foo
[yinzhengjie@s101 data]$ 
[yinzhengjie@s101 data]$ zip yinzhengjie.zip /home/yinzhengjie/bar.job /home/yinzhengjie/foo.job 
  adding: home/yinzhengjie/bar.job (deflated 9%)
  adding: home/yinzhengjie/foo.job (deflated 15%)
[yinzhengjie@s101 data]$ ll | grep yinzhengjie.zip 
-rw-rw-r-- 1 yinzhengjie yinzhengjie       460 Jun 24 20:45 yinzhengjie.zip
[yinzhengjie@s101 data]$ 
[yinzhengjie@s101 data]$ sz yinzhengjie.zip 
[yinzhengjie@s101 data]$ 

3>.在azkaban的web管理界面创建工程并上传zip包（过程和上面的上传步骤类似）

 

 4>.在web和后台查看日志信息