用海豚调度器对Hive数仓各层数据库的SQL任务进行定时调度。比如,DWD层脱敏清洗表的动态插入数据、DWS层指标表的动态插入数据
1、海豚调度器:apache-dolphinscheduler-2.0.5-bin.tar.gz
2、Hive:apache-hive-3.1.2-bin.tar.gz
#! /bin/bash
source /etc/profile
hive -e "
use hurys_dc_dws;
set hive.vectorized.execution.enabled=false;
set hive.auto.convert.join=false;
set mapreduce.map.memory.mb=10150;
set mapreduce.map.java.opts=-Xmx6144m;
set mapreduce.reduce.memory.mb=10150;
set mapreduce.reduce.java.opts=-Xmx8120m;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
set hive.exec.parallel=true;
set hive.support.concurrency=false;
set mapreduce.map.memory.mb=4128;
set hive.vectorized.execution.enabled=false;
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions.pernode=1000;
set hive.exec.max.dynamic.partitions=1500;
insert overwrite table dws_queue_dynamic_statistics_30min partition(day)
select
t1.device_no,
t1.create_time,
case when minute(create_time ) >= 30 then
concat(substr(create_time, 1, 14), '30:00') else
concat(substr(create_time, 1, 14), '00:00')end as start_time,
t1.lane_no,
t2.name,
t3.direction,
t1.queue_count,
t1.queue_len,
t1.day
from hurys_dc_dwd.dwd_queue_dynamic as t1
right join hurys_dc_dws.dws_device_name as t2
on t2.device_no = t1.device_no
right join hurys_dc_dws.dws_device_direction as t3
on t3.device_no = t1.device_no
where t1.create_time is not null
group by t1.device_no, t1.create_time,t1.lane_no, t2.name, t3.direction, t1.queue_count, t1.queue_len, t1.day
"
注意:最后一条SQL不需要加分号;
从新建的空白表到工作流运行后查询表中有数据,说明海豚调度HiveSQL任务成功!!!
到这里,用海豚调度器定时调度HiveSQL脚本任务的演示就结束了!
第一点:海豚对于HiveSQL的SQL调度能力似乎不是特别好,尤其是对一些复杂SQL的调度。因此海豚调度的是HiveSQL的脚本任务,而不是直接调度HiveSQL
第二点:SHELL节点脚本中最后一条SQL不要加分号;
乐于奉献共享,帮助你我他!!!