oozie 任务调度处理
标签（空格分隔）： 协作框架
- 一：oozie example 运行任务调度案例
- 二：oozie 运行自定的mapreduce 的jar 包
- 三：oozie 调度shell 脚本
- 四：oozie 的coordinator 周期性调度当前任务
一： 运行oozie example 案例
1.1 解压exmaple包
解压example 包
tar -zxvf oozie-examples.tar.gz
cd /home/hadoop/yangyang/oozie/examples/apps/map-reduce
job.properties --定义job相关的属性，比如目录路径、namenode节点等。
--定义workflow的位置
workflow.xml --定义工作流相关的配置（start --end --kill）（action）
--mapred.input.dir
--mapred.output.dir
lib --目录，存放job任务需要的资源（jar包）1.2 更改job.properties
nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/hadoop/${examplesRoot}/apps/map-reduce/workflow.xml
outputDir=map-reduce1.3 配置workflow.xml 文件：
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
mapred.mapper.class
org.apache.oozie.example.SampleMapper
mapred.reducer.class
org.apache.oozie.example.SampleReducer
mapred.map.tasks
1
mapred.input.dir
/user/${wf:user()}/${examplesRoot}/input-data/text
mapred.output.dir
/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}
Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
1.3 上传example 目录到hdfs 上面
hdfs dfs -put example example
1.4 运行oozie 调度任务
bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config examples/apps/map-reduce/job.properties -run
查看状态：
输出目录
二：oozie 运行自定的mapreduce 的jar 包
2.1 在hdfs 上创建上传目录
cd /home/hadoop/yangyang/oozie/
hdfs dfs -mkdir oozie-apps2.2 新建本地的文件用作上传的目录
mkdir oozie-apps
cd /home/hadoop/yangyang/oozie/examples/apps
cp -ap map-reduce /home/hadoop/yangyang/oozie/oozie-apps/
cd /homme/hadoop/yangyang/oozie/oozie-appps/map-reduce
mkdir input-data
2.3 拷贝运行的jar包与要运行的job 任务的文件
cp -p mr-wordcount.jar yangyang/oozie/oozie-apps/map-reduce/lib/
cp -p /home/hadoop/wc.input ./input-data2.4 配置job.properties 文件和workflow.xml
vim job.properties
nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=oozie-apps/map-reduce
oozie.wf.application.path=${nameNode}/user/hadoop/${examplesRoot}/workflow.xml
outputDir=oozie-reduce
vim workflow.xml
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
mapred.mapper.new-api
true
mapred.reducer.new-api
true
mapred.input.dir
/user/hadoop/${examplesRoot}/input-data
mapreduce.job.map.class
org.apache.hadoop.wordcount.WordCountMapReduce$WordCountMapper
mapreduce.map.output.key.class
org.apache.hadoop.io.Text
mapreduce.map.output.value.class
org.apache.hadoop.io.IntWritable
mapreduce.job.reduce.class
org.apache.hadoop.wordcount.WordCountMapReduce$WordCountReducer
mapreduce.job.output.key.class
org.apache.hadoop.io.Text
mapreduce.job.output.value.class
org.apache.hadoop.io.IntWritable
mapred.output.dir
/user/hadoop/${examplesRoot}/output-data/${outputDir}
Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
2.6 上传文件到hdfs 上面：
hdfs dfs -put map-reduce oozie-apps2.7 执行oozie 命令运行job 处理
bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config oozie-apps/map-reduce/job.properties -run2.8 在浏览器上面查看测试结果
三：oozie 调度shell 脚本
3.1 生成配置文件：
cd /home/hadoop/yangyang/oozie/examples/apps
cp -ap shell/ ../../oozie-apps/
mv shell mem-shell3.2 书写shell 脚本：
cd /home/hadoop/yangyang/oozie/oozie-apps/mem-shellvim meminfo.sh
#!/bin/bash
/usr/bin/free -m >> /tmp/meminfo3.3 配置job.properties 文件和workflow.xml
vim job.properties
nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=oozie-apps/mem-shell
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/workflow.xml
EXEC=meminfo.sh
vim workflow.xml
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
${EXEC}
/user/hadoop/oozie-apps/mem-shell/${EXEC}#${EXEC}
Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
3.4 上传配置文件到hdfs 上面
cd /home/hadoop/yangyang/oozie/oozie-apps
hdfs dfs -put mem-shell oozie-apps
3.5 执行oozie 调度 shell脚本
bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config oozie-apps/mem-shell/job.properties -run
四：oozie 的coordinator 周期性调度当前任务
4.1 配置时区 更改oozie 的配置文件
cd /home/hadoop/yangyang/oozie/conf
vim oozie-site.xml 增加：
oozie.processing.timezone
GMT+0800
oozie.service.coord.check.maximum.frequency
false
4.2 更改本地 时间
使用root 账户 配置
cp -p /etc/localtime /etc/localtime.bak
rm -rf /etc/localtime
cd /usr/share/zoneinfo/Asia/
cp -p Shanghai /etc/localtime
4.3 更改oozie-consle.js 文件
cd /home/hadoop/yangyang/oozie/oozie-server/webapps/oozie
vim oozie-console.js
function getTimeZone() {
Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
return Ext.state.Manager.get("TimezoneId","GMT+0800");
}
4.4 从新启动oozie 服务
bin/oozie-stop.sh
bin/oozie-start.sh 4.5 查看oozie 的当前时间
4.6 配置job.properties 文件和workflow.xml
cd /home/hadoop/yangyang/oozie/examples/apps
cp -ap cron ../../oozie-apps/
cd cron
rm -rf job.properties workflow.xml
cd /home/hadoop/yangyang/oozie/oozie-apps/mem-shell
cp -p * ../cron
配置job.properties
vim job.properties
---
nameNode=hdfs://namenode01.hadoop.com:8020
jobTracker=namenode01.hadoop.com:8032
queueName=default
examplesRoot=oozie-apps/cron
oozie.coord.application.path=${nameNode}/user/hadoop/${examplesRoot}/
start=2016-06-6T16:57+0800
end=2016-06-6T20:00+0800
workflowAppUri=${nameNode}/user/hadoop/${examplesRoot}/
EXEC=meminfo.sh
配置workflow.xml
vim workflow.xml
---
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
${EXEC}
/user/hadoop/oozie-apps/cron/${EXEC}#${EXEC}
Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
配置coordinator.xml
vim coordinator.xml
---
${workflowAppUri}
jobTracker
${jobTracker}
nameNode
${nameNode}
queueName
${queueName}
EXEC
${EXEC}
4.7 上传配置文件到hdfs 上面：
hdfs dfs -put cron oozie-apps4.8 执行 oozie 命令 运行job
bin/oozie job -oozie http://namenode01.hadoop.com:11000/oozie -config oozie-apps/cron/job.properties -run4.9 从web浏览job的相关问题
