Oozie的使用
一：Oozie运行examples。
1、解压示例包
$ tar -zxf oozie-examples.tar.gz -C ./
2、$ bin/hdfs dfs -put /opt/cdh5/oozie-4.0.0-cdh5.3.6/examples/
3、编写job.properties
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=examples
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml
outputDir=map-reduce${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml 对应的路径
hdfs://bigdata-senior01.ibeifeng.com:8020/user/beifeng/examples/apps/map-reduce/workflow.xml
4、运行 (job.properties 运行的是linux本地的job文件 报错的话看 logs/oozie.log的日志文件查看)
bin/oozie job -oozie http://bigdata-01.yushu.com:11000/oozie -config examples/apps/map-reduce/job.properties -run
5、修改oozie-site.xml
-》重启oozie bin/oozied.sh stop bin/oozied.sh start
6、oozie机制
默认一个workflow会先启动一个MR-》对WF进行封装，初始化工作
二：oozie自定义MapReduce WorkFlow
1、拷贝官方提供的example的示例
2、将Java代码打包 放在lib目录下
3、修改job.properties
#
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/mr-wf
oozie.wf.application.path=${nameNode}/${examplesRoot}/workflow.xml
inputDir=input
outputDir=output-oozie4、修改 workflow.xml
${jobTracker}
${nameNode}
mapreduce.job.queuename
${queueName}
mapred.mapper.new-api
true
mapred.reducer.new-api
true
mapreduce.job.map.class
com.bigdata.mapreduce.WCMapReduce$WCMapper
mapreduce.job.reduce.class
com.bigdata.mapreduce.WCMapReduce$WCReducer
mapreduce.map.output.key.class
org.apache.hadoop.io.Text
mapreduce.map.output.value.class
org.apache.hadoop.io.IntWritable
mapreduce.job.output.key.class
org.apache.hadoop.io.Text
mapreduce.job.output.value.class
org.apache.hadoop.io.IntWritable
mapreduce.input.fileinputformat.inputdir
${nameNode}/${inputDir}
mapreduce.output.fileoutputformat.outputdir
${nameNode}/${outputDir}
Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
5、放jar包
6、上传目录到HDFS
7、运行
export OOZIE_URL="http://bigdata-01.yushu.com:11000/oozie"
bin/oozie job -config apps/app/mrwf/job.properties -run
三、oozie自定义Shell WorkFlow
job.properties配置文件内容：
#
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/shell-wf
oozie.wf.application.path=${nameNode}/${examplesRoot}/workflow.xml
EXEC=oozie.sh
workflow.xml文件内容：
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
${EXEC}
${EXEC}#${EXEC}
Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
oozie.sh文件内容：
#!/bin/bash
HIVE_HOME=/opt/cdh5/hive-0.13.1-cdh5.3.6
$HIVE_HOME/bin/hive -e "show databases" >>/opt/datas/oozie-hive.txt最后执行命令：bin/oozie job -config application/apps/shell-wf/job.properties -run
四、oozie修改时区
1、时间调度
-》coordinate
-》基于时间的调度
-》基于数据可用性的调度
-》起始时间
-》结束时间
-》时间频率
2、时区
3、oozie支持从UTC和GMT
4、将oozie的默认时区指定为GMT
5、命令：date -R
-》如果命令没有输出+0800的显示
-》删除时区文件/etc/localtime
-》删除后重新生成，创建一个软连接ln -s
/usr/share/zoneinfo/Asia/shanghai
6、修改oozie时区
修改oozie-site.xml
-》重启oozie
-》修改oozie-server/webapps/oozie/oozie-console.js代码
function getTimeZone() {
Ext.state.Manager.setProvider(new Ext.state.CookieProvider());
return Ext.state.Manager.get("TimezoneId","GMT+0800");
}
五、oozie Coordinator讲解
-》修改oozie-site.xml
* * * * *
job.properties文件内容：
nameNode=hdfs://bigdata-01.yushu.com:8020
jobTracker=bigdata-01.yushu.com:8032
queueName=default
examplesRoot=user/ds/application/apps/cron-wf
oozie.coord.application.path=${nameNode}/${examplesRoot}/coordinator.xml
start=2017-06-26T19:50+0800
end=2017-06-26T19:58+0800
workflowAppUri=${nameNode}/${examplesRoot}/workflow.xmlcoordinator.xml文件内容：frequency="${coord:minutes(2)} 表示每隔两分钟执行
${workflowAppUri}
jobTracker
${jobTracker}
nameNode
${nameNode}
queueName
${queueName}
workflow.xml文件内容：
参考文献：http://blog.csdn.net/nsrainbow/article/details/43746111
参考文献2：https://www.zybuluo.com/tsing1226/note/263232
参考文献3：http://shiyanjun.cn/archives/684.html oozie Coordinator Job