【黑马甄选离线数仓day06_核销主题域开发】
1. 核销主题_DWD和DWM层
1.0 ODS层
操作数据存储层: ==Operate Data Store==
核心理念: 几乎和源数据保持一致,粒度相同
注意事项: ==同步方式(全量同步,全量覆盖,增量仅新增,增量新增和更新)+ 内部表 + 分区表(部分) + 指定字符分隔符 +orc + zlib==
第二天的时候已经完成了从mysql以及sqlserver抽取数据到ods层的过程
1.1 DWD层
数据仓库细节层: ==Data Warehouse Detail==
为DWM层提前合并部分数据
注意事项: ==同步方式(全量同步,全量覆盖,增量仅新增,增量新增和更新)+ 内部表 + 分区表(部分) + 指定字符分隔符 +orc + snappy==
核心理念:
清洗 转换
区分事实表和维度表
降维拉宽
1- 将相关的表进行合并操作:
事实表中多个事实表之间有关联, 一般是必须合并的
事实表和维度表合并操作: 和有关联的维度表进行合并
2- 如果事实表和其他的表没有关联, 此时尝试和维度表合并即可
注意: 如果维度表都没有, 无需合并, 进行清洗过滤即可
3- 设计拉宽表:
- a）原始表中重要信息（不好判断的时候，尽量保留所有字段即可）
- b）一些必要类型字段（比如说is_day_clear，is_balance_consume）
- c）维度拉宽（比如对日期/时间进行处理、门店信息拉宽、商品信息拉宽）
- d）还要注意，保持数仓中字段名的统一（类型名、指标、维度字段等）
本次DWD层,共设计为12张事实表,在添加数据的时候,注意具体计算只计算增量，全量计算直接去掉where条件即可。或者在调度时使用补数的方法，将历史分区一天一天跑出来即可！
门店销售明细宽表(8张表降维拉宽): dwd_sale_store_sale_dtl_i 线上余额支付明细表: dwd_sale_shop_sale_balance_pay_i 商城订单表(核销表): dwd_sold_shop_order_i 商城订单明细表(核销表): dwd_sold_shop_order_item_i 商城订单核销明细宽表(10张表降维拉宽): dwd_sold_shop_order_dtl_i 门店库调表(3张表降维拉宽): dwd_stock_store_stock_adj_i 门店收货表(3张表降维拉宽):dwd_order_store_receive_i 门店退货表(3张表降维拉宽): dwd_order_store_return_to_vendor_i 门店退配表(3张表降维拉宽): dwd_order_store_return_to_dc_i 门店调入表(3张表降维拉宽):dwd_order_store_alloc_in_i 门店调出表(3张表降维拉宽):dwd_order_store_alloc_out_i 门店要货表(3张表降维拉宽):dwd_order_store_require_i
1.1.1 门店销售明细宽表
此表为常见的明细打宽的开发任务, 整个业务过程是门店下单, 粒度为订单明细, 维度包含时间、门店、商品、品类等, 事实包括销售数量、金额、成本、余额支付金额等
本次, 我们主要将门店销售的核心三张表进行拉宽操作, 包括门店销售信息表、门店销售明细表、门店销售支付表进行合并拉宽, 同时还需要带上相关的维度表
涉及事实表: 3张
ods_sale_store_sale_info_i(门店销售信息表): 主要描述了门店线下的的销售明细，包括单号、时间、金额、渠道、会员等信息。
ods_sale_store_sale_dtl_i(门店销售明细表): 相对于销售信息表, 多了具体的商品信息, 库存信息等
注意:
1）因为组合商品和组合成分都是会记录到明细表的，这里只需要分析组合商品，所以需要排除掉组合成分。
比如牙膏和牙刷一起组合销售，则牙膏和牙刷一起为组合商品，牙膏、牙刷为组合成分。
combination_flag int comment '1-组合商品，2-组合成分'
2) 有时候库存调整要做一些冲减单，这些单据不是正常的销售单，所以需要排除。过滤条件为offset_flag = 0
ods_sale_store_sale_pay_i(门店销售支付表): 主要描述了每一单的支付类别、支付金额等信息。这个表只是为了取余额销售的信息，所以这里让 pay_type_id=’201’
三张表的关联条件是order_no相同。订单表和明细表进行inner join，支付表要跟订单表做左关联（因为不是每一单都是余额支付）
涉及维度表: 5张
dwd_dim_date_f(时间维度表):记录了一个日期对应的不同的日期，如周一日期、周末日期、月一日期、月末日期等。
dwd_dim_source_type_map_i(交易类型映射表):记录了原始交易渠道对应的归类情况。
dwd_dim_store_i(门店表):记录了门店的基本信息，包括门店销售类型、分店类型、城市、区域、人数、状态等信息。
dwd_dim_goods_i(商品表):记录商品的基本信息，包括ID，编码，名称，分类，售卖信息，订货信息等。
dwd_dim_store_goods_i(门店商品表):跟商品表不同的是，商品表记录的是商品通用的信息，而门店商品表里会记录一些门店独有的信息，比如日清信息，采购信息，柜组信息等。
建表语句
CREATE TABLE IF NOT EXISTS dwd.dwd_sale_store_sale_dtl_i(
-- 门店销售信息表
trade_date_time STRING COMMENT '销售时间',
trade_date STRING COMMENT '交易日期',
-- 时间维度表
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
-- 门店销售信息表
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
parent_store_no STRING COMMENT '母店编码',
store_no STRING COMMENT '店铺编码',
-- 门店表
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
-- 门店销售信息表
trade_type BIGINT COMMENT '结算类型(0.正常交易,1.赠品发放,2.退货,4.培训,5.取消交易)',
-- 交易类型映射表
source_type BIGINT COMMENT '交易来源1:线下POS;2:三方平台;3:传智鲜商城;4:黑马优选团;5:传智大客户;6:传智其他;7:黑马优选;8:优选海淘;9:优选大客户;10:优选POS;11:优选APP;12:优选H5;13:店长工具线下;14:店长工具线上;15:黑马其他',
source_type_name STRING COMMENT '交易来源名称',
-- 门店销售信息表
sale_type BIGINT COMMENT '销售类型 1.实物,2.代客,3.优选小程序,4.离店,5.传智鲜小程序,6.第三方平台,7.其他,8.大客户',
member_type BIGINT COMMENT '会员类型:0非会员,1线上会员,2实体卡会员',
is_balance_consume BIGINT COMMENT '是否有余额支付:0否,1是',
parent_order_no STRING COMMENT '母订单编号',
order_no STRING COMMENT '订单编号',
pos_no STRING COMMENT 'Pos机号',
ser_id STRING COMMENT 'POS机当天序号从1开始递增',
item BIGINT COMMENT '商品在小票的位置',
`sort` BIGINT COMMENT '组合商品分割商品拆出位置',
pay_time STRING COMMENT '支付时间',
last_update_time STRING COMMENT '最后更新时间',
cashier_no STRING COMMENT '收银员编码',
cashier_name STRING COMMENT '收银员名称',
share_user_id STRING COMMENT '分享人用户ID',
commission_amount DECIMAL(27, 2) COMMENT '佣金',
zt_id BIGINT COMMENT '中台ID',
member_id BIGINT COMMENT '会员ID',
card_no STRING COMMENT '卡号',
-- 商品表
first_category_no STRING COMMENT '一级分类编码',
first_category_name STRING COMMENT '一级分类名称',
second_category_no STRING COMMENT '二级分类编码',
second_category_name STRING COMMENT '二级分类名称',
third_category_no STRING COMMENT '三级分类编码',
third_category_name STRING COMMENT '三级分类名称',
-- 门店销售明细表
goods_no STRING COMMENT '商品编码',
-- 商品表
goods_name STRING COMMENT '商品名称',
spec STRING COMMENT '单位',
-- 门店销售明细表
is_component BIGINT COMMENT '是否为组合商品:0否,1是',
-- 门店商品表
supply_team BIGINT COMMENT '供应链团队 1.平台商品,2.优选标品,3.传智鲜标品,4.传智鲜生鲜,5优选POS商品',
dc_no STRING COMMENT '采购仓库编号',
dc_name STRING COMMENT '采购仓库名称',
group_no STRING COMMENT '采购柜组编号',
group_name STRING COMMENT '采购柜组名称',
-- 门店销售明细表
trade_mode_id BIGINT COMMENT '结算方式:1购销,2联营',
vendor_id BIGINT COMMENT '供应商ID',
contract_no STRING COMMENT '合同编号',
-- 门店商品表
is_clean BIGINT COMMENT '商品是否日清:0否,1是',
-- 门店销售明细表
is_daily_clear BIGINT COMMENT '商品是否参加日清活动:0否,1是',
sale_qty DECIMAL(27, 3) COMMENT '商品销售数量',
sale_amount DECIMAL(27, 2) COMMENT '商品销售金额',
dis_amount DECIMAL(27, 2) COMMENT '商品折扣金额',
sale_cost DECIMAL(27, 2) COMMENT '商品销售成本',
-- 门店销售支付表
balance_amount DECIMAL(27, 2) COMMENT '余额支付',
-- 当前写入时间: current_timestamp
write_time TIMESTAMP COMMENT '写入时间'
)
COMMENT '门店销售明细表'
partitioned by(dt STRING COMMENT '核销日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');需求梳理
-
先将三个核心事实表进行合并操作, 形成结果数据表
销售日期:基于门店销售信息表中的库存日期处理 注意:所有时间数据都建议格式化下 母店编码和店铺编码: 母店编码为销售信息表的门店编码, 门店编码使用销售信息表的销售门店编码,如果没有, 再使用门店编码 交易来源:如果为null, 返回1,代表线下POS 会员类型: 如果以OL- 或者 SF- 开头的为1线上会员, 如果是''或者null为0非会员, 否则为实体卡会员 是否余额支付: 基于门店销售支付表,如果订单id为null返回0代表不是,否则返回1代表是 结算类型: 判断trade_id,如果为1 返回0(正常交易), 为2返回2, 如果为3,4返回5 母订单编号: 判断明细表中商品销售金额sale_amount如果小于0采用销售表的source_order_sn(退款单据ID),反之如果大于等于0采用销售表的parent_order_sn. 否则选order_no item,sort,cashier_no,cashier_name,zt_id(member_center_sn),member_id: 如果为null设为0 支付时间: 选择商品信息表的支付日期pay_date 最后更新时间: 选择商品信息表的库存时间deal_date 是否为组合类型: 判断明细表combination_flag,如果为1返回1(是),否则返回0(否) trade_mode_id,share_user_id,commission_amount: 如果为null, 设置为0 余额支付金额: 判断单据总支付金额如果等于0 , 则结果为0; 如果余额支付金额为null,则结果也为0, 否则使用余额支付金额 * (商品销售金额 / 单据总支付金额)
3个表关联字段
trade_date_time, trade_date, hourly, minute, parent_store_no, store_no, source_type, member_type, is_balance_consume, trade_type, parent_order_no, order_no, pos_no, ser_id,item, `sort`, pay_time, last_update_time, cashier_no, cashier_name, zt_id, member_id, card_no, goods_no, is_component, trade_mode_id, vendor_id, contract_no, is_daily_clear, share_user_id, commission_amount, sale_qty, sale_amount, dis_amount, sale_cost, balance_amount
代码实现:
select
date_format(t2.deal_date,'yyyy-MM-dd HH:mm:ss') as trade_date_time,
date_format(t2.trade_date,'yyyy-MM-dd HH:mm:ss') as trade_date ,
hour(t2.deal_date) as hourly,
minute(t2.deal_date) as minute,
t2.store_no as parent_store_no,
coalesce(t2.sale_store_no,t2.store_no) as store_no,
coalesce(t2.source_type,1) as source_type,
if(
t2.member_type like 'OL-%' OR t2.member_type like 'SF-%',
1,
if(
t2.member_type = '' OR t2.member_type is null,
0,
2
)
) as member_type,
if(
t3.order_no is null,
0,
1
) as is_balance_consume,
if(
t2.trade_id = 1,
0,
if(
t2.trade_id = 2,
2,
5
)
) as trade_type,
if(
t1.sale_amount < 0,
t2.source_order_sn,
if(
t1.sale_amount >= 0,
t2.parent_order_sn,
t1.order_no
)
) as parent_order_no,
t2.order_no,
t2.pos_no,
t2.ser_id,
coalesce(t1.item,0) as item,
coalesce(t1.`sort`,0) as sort,
date_format(t2.pay_date,'yyyy-MM-dd HH:mm:ss') as pay_time,
date_format(t2.deal_date,'yyyy-MM-dd HH:mm:ss') as last_update_time,
coalesce(t2.cashier_no,0) as cashier_no,
coalesce(t2.cashier_name,0) as cashier_name,
coalesce(t2.member_center_sn,0) as zt_id,
coalesce(t2.member_id,0) as member_id,
t2.card_no,
t1.goods_no,
if(
t1.combination_flag = 1,
1,
0
) as is_component,
coalesce(t1.trade_mode_id,0) as trade_mode_id,
t1.vendor_id,
t1.contract_no,
t1.is_daily_clear,
coalesce(t1.share_user_id,0) as share_user_id,
coalesce(t1.commission_amount,0) as commission_amount,
t1.sale_qty,
t1.sale_amount,
t1.dis_amount,
t1.sale_cost,
if(
t2.total_pay_amount = 0,
0,
if(
t3.pay_amount is null,
0,
cast(t3.pay_amount * t1.sale_amount / t2.total_pay_amount as decimal(27,2))
)
)as balance_amount
from (select * from ods.ods_sale_store_sale_dtl_i where combination_flag != 2 and offset_flag = 0) t1
join ods.ods_sale_store_sale_info_i t2 on t2.order_no = t1.order_no
left join (select * from ods.ods_sale_store_sale_pay_i where pay_type_id = '201') t3 on t2.order_no = t3.order_no ;-
接着基于结果和其他五个维度表关联, 生成最终维度表
刻钟: 判断0-14为1,15-29为2,30-44为3,45-59位4 刻钟数: 计算格式为hourly*4+刻钟 销售类型: 判断source_type或者original_source_type,当为1返回1, 当为9返回5 , 当为 4,5,6,7,8返回6. 当为11返回8 ,否则返回7 供应链类型: 判断门店商品信息表的 tag 商品标识为null, 返回4 采购仓库编号: 判断门店商品信息表的 dc_no 如果为null 返回-1 采购仓库名称: 判断门店商品信息表的 dc_name 如果为null 返回其他仓 采购柜组编号: 判断门店商品信息表的 group_no 如果为null, 返回-1 采购柜组名称: 判断门店商品信息表的 group_name 如果为null, 返回其他柜组 供应商ID: 判断门店明细表的vendor_id 如果为null, 返回0 是否日清: 判断门店商品信息表的is_clear 如果为null, 返回0 写入时间: 设置为昨天 核销时间: 设置为交易日期
数据插入
-- 友好提醒: 先关联3张事实表,再去关联其他5张维度表
with t4 as(
select
date_format(t2.deal_date,'yyyy-MM-dd HH:mm:ss') as trade_date_time,
date_format(t2.trade_date,'yyyy-MM-dd') as trade_date ,
hour(t2.deal_date) as hourly,
minute(t2.deal_date) as minute,
t2.store_no as parent_store_no,
coalesce(t2.sale_store_no,t2.store_no) as store_no,
coalesce(t2.source_type,1) as source_type,
if(
t2.member_type like 'OL-%' OR t2.member_type like 'SF-%',
1,
if(
t2.member_type = '' OR t2.member_type is null,
0,
2
)
) as member_type,
if(
t3.order_no is null,
0,
1
) as is_balance_consume,
if(
t2.trade_id = 1,
0,
if(
t2.trade_id = 2,
2,
5
)
) as trade_type,
if(
t1.sale_amount < 0,
t2.source_order_sn,
if(
t1.sale_amount >= 0,
t2.parent_order_sn,
t1.order_no
)
) as parent_order_no,
t2.order_no,
t2.pos_no,
t2.ser_id,
coalesce(t1.item,0) as item,
coalesce(t1.`sort`,0) as sort,
date_format(t2.pay_date,'yyyy-MM-dd HH:mm:ss') as pay_time,
date_format(t2.deal_date,'yyyy-MM-dd HH:mm:ss') as last_update_time,
coalesce(t2.cashier_no,0) as cashier_no,
coalesce(t2.cashier_name,0) as cashier_name,
coalesce(t2.member_center_sn,0) as zt_id,
coalesce(t2.member_id,0) as member_id,
t2.card_no,
t1.goods_no,
if(
t1.combination_flag = 1,
1,
0
) as is_component,
coalesce(t1.trade_mode_id,0) as trade_mode_id,
t1.vendor_id,
t1.contract_no,
t1.is_daily_clear,
coalesce(t1.share_user_id,0) as share_user_id,
coalesce(t1.commission_amount,0) as commission_amount,
t1.sale_qty,
t1.sale_amount,
t1.dis_amount,
t1.sale_cost,
if(
t2.total_pay_amount = 0,
0,
if(
t3.pay_amount is null,
0,
cast(t3.pay_amount * t1.sale_amount / t2.total_pay_amount as decimal(27,2))
)
)as balance_amount
from (select * from ods.ods_sale_store_sale_dtl_i where combination_flag != 2 and offset_flag = 0) t1
join ods.ods_sale_store_sale_info_i t2 on t2.order_no = t1.order_no
left join (select * from ods.ods_sale_store_sale_pay_i where pay_type_id = '201') t3 on t2.order_no = t3.order_no
)
-- 查询并插入数据
insert overwrite table dwd.dwd_sale_store_sale_dtl_i partition(dt)
select
t4.trade_date_time,
t4.trade_date,
t5.week_trade_date,
t5.month_trade_date,
t4.hourly,
case
when t4.minute between 0 and 14 then 1
when t4.minute between 15 and 29 then 2
when t4.minute between 30 and 44 then 3
when t4.minute between 45 and 59 then 4
end as quarter,
(
t4.hourly * 4
+
case
when t4.minute between 0 and 14 then 1
when t4.minute between 15 and 29 then 2
when t4.minute between 30 and 44 then 3
when t4.minute between 45 and 59 then 4
end
) as quarters,
t4.parent_store_no,
t4.store_no,
t7.store_name,
t7.store_sale_type,
t7.store_type_code,
t7.worker_num,
t7.store_area,
t7.city_id,
t7.city_name,
t7.region_code,
t7.region_name,
t7.is_day_clear,
t4.trade_type,
t4.source_type,
t6.source_type_name,
case
when t4.source_type = 1 then 1
when t4.source_type = 9 then 5
when t4.source_type in (4,5,6,7,8) then 6
when t4.source_type = 11 then 8
else 7
end as sale_type,
t4.member_type,
t4.is_balance_consume,
t4.parent_order_no,
t4.order_no,
t4.pos_no,
t4.ser_id,
t4.item,
t4.sort,
t4.pay_time,
t4.last_update_time,
t4.cashier_no,
t4.cashier_name,
t4.share_user_id,
t4.commission_amount,
t4.zt_id,
t4.member_id,
t4.card_no,
t8.first_category_no,
t8.first_category_name,
t8.second_category_no,
t8.second_category_name,
t8.third_category_no,
t8.third_category_name,
t4.goods_no,
t8.goods_name,
t8.spec,
t4.is_component,
coalesce(t9.tag,4) as supply_team,
coalesce(t9.dc_no,-1) as dc_no,
coalesce(t9.dc_name,'其他仓') as dc_name,
coalesce(t9.group_no,-1) as group_no,
coalesce(t9.group_name,'其他柜组') as group_name ,
t4.trade_mode_id,
coalesce(t4.vendor_id,0) as vendor_id,
t4.contract_no,
coalesce(t9.is_clear,0) as is_clean,
t4.is_daily_clear,
t4.sale_qty,
t4.sale_amount,
t4.dis_amount,
t4.sale_cost,
t4.balance_amount,
date_sub(current_date(),1) as write_time,
t4.trade_date as dt
from t4
left join dim.dwd_dim_date_f t5 on t4.trade_date = t5.trade_date
left join dim.dwd_dim_source_type_map_i t6 on t6.dt = '2023-11-23' and t4.source_type = t6.original_source_type
left join dim.dwd_dim_store_i t7 on t7.dt = '2023-11-23' and t4.store_no = t7.store_no
left join dim.dwd_dim_goods_i t8 on t8.dt = '2023-11-23' and t4.goods_no = t8.goods_no
left join dim.dwd_dim_store_goods_i t9 on t9.dt = '2023-11-23' and t9.goods_no = t4.goods_no;1.1.2 线上余额支付明细
这个表的目的是为了计算dwd_sold_shop_order_dtl_i(商城核销明细表)，因为是核销维度，有可能有的订单从下单到签收经历了很长时间，在判断是否是余额销售的时候，就需要取到当时下单对应的支付单的情况（因为支付单是按下单时间分区的），不可能把全量数据都拿过来做关联，所以这里就需要一个表来只记录余额支付的明细，这样就极大的降低了数据量。
处理方案: 在门店销售支付表在where条件中设置pay_channel_name = '余额支付'，只保留余额支付的记录。
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_sale_shop_sale_balance_pay_i(
store_no STRING COMMENT '门店编码',
store_name STRING COMMENT '门店名称',
trade_date TIMESTAMP COMMENT '交易日期',
member_id BIGINT COMMENT '会员ID',
zt_id BIGINT COMMENT '中台会员ID',
trade_order_id STRING COMMENT '关联的交易单ID',
pay_order_id STRING COMMENT '支付单id',
order_no STRING COMMENT '订单号',
pay_channel STRING COMMENT '支付渠道',
pay_channel_name STRING COMMENT '支付渠道名称',
trade_order_type BIGINT COMMENT '交易单类型，1消费,2充值,3提现,4退货退款',
trade_order_type_name STRING COMMENT '交易单类型名称',
pay_amount DECIMAL(27, 2) COMMENT '支付对等RMB的金额，比如是积分支付，那这里就是积分所对应的RMB的金额',
trade_merchant STRING COMMENT '交易商家'
)
COMMENT '线上余额支付明细'
partitioned by (dt STRING COMMENT '交易日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
-- 线上余额支付明细
insert overwrite table dwd.dwd_sale_shop_sale_balance_pay_i partition (dt)
select
store_no,
store_name,
trade_date,
member_id,
zt_id,
trade_order_id,
pay_order_id,
order_no,
pay_channel,
pay_channel_name,
trade_order_type,
trade_order_type_name,
pay_amount,
trade_merchant,
dt
from ods.ods_sale_shop_sale_pay_i where pay_channel_name = '余额支付';
-- 注意: 在后续增量的SQL中, 需要添加dt字段获取上一天的数据1.1.3 商城订单表(核销表)
订单下单完成后，会经历各种状态变化，只要更新就会同步过来更新后的数据。因为项目有核销主题、售卖主题，在做具体的分析的时候，希望核销主题的表是按照核销时间（完成时间）进行分区的，售卖主题的表是按照售卖时间（下单时间）进行分区的。所以，这里需要对shop_order表进行处理，根据complete_time和create_time进行分别分区，用于核销主题和售卖主题使用。
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_sold_shop_order_i(
id BIGINT COMMENT '编号',
parent_order_no STRING COMMENT '父单订单号',
order_id STRING COMMENT '订单编号',
is_split BIGINT COMMENT '是否拆单：0-不需要拆单；1-待拆单；2-已拆单',
platform_id BIGINT COMMENT '平台id：1-有赞，2-京东到家，3-美团外卖，4-饿了么',
tid STRING COMMENT '平台订单号',
source_type BIGINT COMMENT '订单来源：10,20,30,40,41,50,60,70',
source_name STRING COMMENT '订单来源名称：10-有赞，20-京东到家，30-美团外卖，40-饿了么，41-百度外卖，50-传智鲜精选，60-黑马优选，70-抖店',
store_no STRING COMMENT '门店编码',
city_id BIGINT COMMENT '城市编号',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
order_status BIGINT,
order_status_desc STRING COMMENT '主订单状态描述：0-新建; 1-待出票；2-待备货；3-待揽件；4-待自提; 5-配送中；6-已完成；7-已取消',
pay_type BIGINT COMMENT '支付类型：1-线下支付；2-线上支付',
trade_type STRING COMMENT '交易类型。取值范围：fixed(一口价) gift(送礼）bulk_purchase（来自分销商的采购）present （赠品领取）group （拼团订单） pifa （批发订单） cod （货到付款） peer （代付） qrcode（扫码商家二维码直接支付的交易）qrcode_3rd（线下收银台二维码交易)',
is_deleted BIGINT COMMENT '是否有效，1：已删除，0：正常',
order_create_time TIMESTAMP COMMENT '平台订单创建时间',
order_pay_time TIMESTAMP COMMENT '订单支付时间',
create_time TIMESTAMP COMMENT '创建时间',
print_status BIGINT COMMENT '打印状态：0-未打票;1-已打票',
print_time TIMESTAMP COMMENT '出票时间',
stock_up_status BIGINT COMMENT '门店处理状态：0-待备货/1-已备货',
stock_up_time TIMESTAMP COMMENT '备货完成时间',
order_type BIGINT COMMENT '配送类型（真正的订单类型由业务类型来决定）：1-及时送；2-隔日送；3-自提单',
express_type BIGINT COMMENT '配送方式：0-三方平台配送；1-自配送；2-快递；3-自提',
receive_time TIMESTAMP COMMENT '要求送达/自提时间',
express_code STRING COMMENT '配送单号',
delivery_status BIGINT COMMENT '配送状态：0-待配送；1-配送中；2-已送达',
delivery_time TIMESTAMP COMMENT '配送时间',
pick_up_status BIGINT COMMENT '自提状态：0-待自提；1-已自提',
qr_code STRING COMMENT 'qr提货码',
pick_up_time TIMESTAMP COMMENT '自提时间',
complete_time TIMESTAMP COMMENT '订单完结时间',
is_cancel BIGINT COMMENT '是否取消',
cancel_time TIMESTAMP COMMENT '取消时间',
cancel_reason STRING COMMENT '取消原因',
refund_status BIGINT COMMENT '退款状态：0未退款，1部分退款，2已全额退款',
refund_time TIMESTAMP COMMENT '已退款时间',
last_update_time TIMESTAMP COMMENT '最新更新时间',
order_total_amount DECIMAL(27, 2) COMMENT '订单总金额',
product_total_amount DECIMAL(27, 2) COMMENT '商品总金额（原价）',
pack_amount DECIMAL(27, 2) COMMENT '餐盒费/打包费',
delivery_amount DECIMAL(27, 2) COMMENT '配送费',
discount_amount DECIMAL(27, 2) COMMENT '订单优惠金额=商家承担优惠金额+平台补贴金额',
seller_discount_amount DECIMAL(27, 2) COMMENT '商家承担优惠金额',
platform_allowance_amount DECIMAL(27, 2) COMMENT '平台补贴金额',
real_paid_amount DECIMAL(27, 2) COMMENT '实付金额',
product_discount DECIMAL(27, 2) COMMENT '商品优惠金额',
real_product_amount DECIMAL(27, 2) COMMENT '商品实际金额',
buyer_id BIGINT COMMENT '买家id',
buyer_phone STRING COMMENT '买家电话',
buyer_remark STRING COMMENT '买家备注',
r_name STRING COMMENT '收货人姓名',
r_tel STRING COMMENT '收货人电话',
r_province STRING COMMENT '收货人省份',
r_city STRING COMMENT '收货人城市',
r_district STRING COMMENT '收货人区域',
r_address STRING COMMENT '收货人地址',
r_zipcode STRING COMMENT '收货人邮编',
is_tuan_head BIGINT COMMENT '是否为团长订单',
store_leader_id BIGINT COMMENT '团长id',
order_group_no STRING COMMENT '团单号',
commision_amount DECIMAL(27, 2) COMMENT '抽佣金额',
settle_amount DECIMAL(27, 2) COMMENT '结算金额',
points_amount DECIMAL(27, 2) COMMENT '积分抵扣金额',
pay_point BIGINT COMMENT '消费积分数',
balance_amount DECIMAL(27, 2) COMMENT '余额扣除金额',
pay_channel_amount DECIMAL(27, 2) COMMENT '通过支付渠道支付的金额',
point_amount DECIMAL(27, 2) COMMENT '消费赠送积分',
sync_erp_status BIGINT COMMENT '同步erp状态',
sync_erp_msg STRING COMMENT '同步erp失败消息'
)
COMMENT '商城订单表(核销表)'
partitioned by (dt STRING COMMENT '完成日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');ods_sale_shop_order_i 是按照最后更新时间进行分区的，所以，取出当天更新的数据，并且只取完成时间是当天的数据，插入dwd表时按照complete_time分区即可。
数据插入:
-- 商城订单表(核销表)
insert overwrite table dwd.dwd_sold_shop_order_i partition (dt)
select
id,
parent_order_no,
order_id,
is_split,
platform_id,
tid,
source_type,
source_name,
store_no,
city_id,
city_name,
region_code,
order_status,
order_status_desc,
pay_type,
trade_type,
is_deleted,
order_create_time,
order_pay_time,
create_time,
print_status,
print_time,
stock_up_status,
stock_up_time,
order_type,
express_type,
receive_time,
express_code,
delivery_status,
delivery_time,
pick_up_status,
qr_code,
pick_up_time,
complete_time,
is_cancel,
cancel_time,
cancel_reason,
refund_status,
refund_time,
last_update_time,
order_total_amount,
product_total_amount,
pack_amount,
delivery_amount,
discount_amount,
seller_discount_amount,
platform_allowance_amount,
real_paid_amount,
product_discount,
real_product_amount,
buyer_id,
buyer_phone,
buyer_remark,
r_name,
r_tel,
r_province,
r_city,
r_district,
r_address,
r_zipcode,
is_tuan_head,
store_leader_id,
order_group_no,
commision_amount,
settle_amount,
points_amount,
pay_point,
balance_amount,
pay_channel_amount,
point_amount,
sync_erp_status,
sync_erp_msg,
date_format(complete_time,'yyyy-MM-dd') as dt
from ods.ods_sale_shop_order_i where complete_time is not null;
-- 后续增量只需加上where dt = '2023-11-24' and date_format(complete_time,'yyyy-MM-dd') = '2023-11-24';1.1.4 订单明细表(核销表)
与商城订单表一样, 只获取核销后的订单数据, 以及是最终完结的数据, 在售卖主题中, 获取下单时的信息数据即可
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_sold_shop_order_item_i(
id BIGINT COMMENT '自增id',
order_id STRING COMMENT '订单编号',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
weight DECIMAL(27, 3) COMMENT '重量，单位kg',
quantity BIGINT COMMENT '数量',
unit STRING COMMENT '单位',
sale_qty DECIMAL(27, 3) COMMENT '销售数量',
disp_price DECIMAL(27, 2) COMMENT 'sku展示价格',
pay_price DECIMAL(27, 2) COMMENT '价格',
sale_amount DECIMAL(27, 2) COMMENT '单品销售金额',
dis_amount DECIMAL(27, 2) COMMENT '单品总折扣金额',
sale_cost DECIMAL(27, 2) COMMENT '销售成本',
sale_type BIGINT COMMENT '类型：1-常规；2-赠品',
create_time TIMESTAMP COMMENT '创建时间',
complete_time TIMESTAMP COMMENT '完成时间',
last_update_time TIMESTAMP COMMENT '更新时间',
activity_plat_city_goods_id BIGINT COMMENT '活动商品区域id',
activity_type BIGINT COMMENT '活动类型(11:拼团 21:秒杀)',
item_goods_key STRING COMMENT '虚拟字段，itemgoodskey',
is_deleted BIGINT COMMENT '是否删除：0-否；1-删除',
transfer_paper_no STRING COMMENT '要货单号',
serial_no BIGINT COMMENT '商品序号，每个订单下起始都为1',
is_delivery BIGINT COMMENT '仓发是否配送；1:配送',
goods_source_type BIGINT COMMENT '商品来源类型：1-生鲜品；2-标品',
trade_mode_id BIGINT COMMENT '结算方式:1购销,2联营',
vendor_id BIGINT COMMENT '供应商ID',
contract_no STRING COMMENT '合同编号'
)
COMMENT '订单明细表（核销表）'
partitioned by (dt STRING COMMENT '完成日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');处理方式与商城订单表也是类似的
数据插入:
-- 商城订单明细表
insert overwrite table dwd.dwd_sold_shop_order_item_i partition (dt)
select
id,
order_id,
goods_no,
goods_name,
weight,
quantity,
unit,
sale_qty,
disp_price,
pay_price,
sale_amount,
dis_amount,
sale_cost,
sale_type,
create_time,
complete_time,
last_update_time,
activity_plat_city_goods_id,
activity_type,
item_goods_key,
is_deleted,
transfer_paper_no,
serial_no,
is_delivery,
goods_source_type,
trade_mode_id,
vendor_id,
contract_no,
date_format(complete_time,'yyyy-MM-dd') as dt
from ods.ods_sale_shop_order_item_i where complete_time is not null;
-- 后续增量只需加上where dt = '2023-11-24' and date_format(complete_time,'yyyy-MM-dd') = '2023-11-24';1.1.5 商城核销明细宽表
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_sold_shop_order_dtl_i(
complete_time STRING COMMENT '订单完成时间',
trade_date STRING COMMENT '交易日期',
-- 日期表
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
parent_order_no STRING COMMENT '父单订单号/源单号',
order_id STRING COMMENT '订单编号',
trade_type BIGINT COMMENT '结算类型(0.正常交易,1.赠品发放,2.退货,4.培训,5.取消交易)',
is_split BIGINT COMMENT '是否拆单：0-不需要拆单；1-待拆单；2-已拆单',
platform_id BIGINT COMMENT '平台id：1-有赞，2-京东到家，3-美团外卖，4-饿了么',
tid STRING COMMENT '平台订单号',
source_type BIGINT COMMENT '订单来源：10,20,30,40,41,50,60,70',
source_name STRING COMMENT '订单来源名称：10-有赞，20-京东到家，30-美团外卖，40-饿了么，41-百度外卖，50-传智鲜精选，60-黑马优选，70-抖店',
order_type BIGINT COMMENT '配送类型（真正的订单类型由业务类型来决定）：1-及时送；2-隔日送；3-自提单',
express_type BIGINT COMMENT '配送方式：0-三方平台配送；1-自配送；2-快递；3-自提',
order_status BIGINT,
order_status_desc STRING COMMENT '主订单状态描述：0-新建; 1-待出票；2-待备货；3-待揽件；4-待自提; 5-配送中；6-已完成；7-已取消',
pay_type BIGINT COMMENT '支付类型：1-线下支付；2-线上支付',
is_balance_consume BIGINT COMMENT '是否余额支付：1是，0否',
store_no STRING COMMENT '店铺编码',
-- 分店信息表
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
order_create_time TIMESTAMP COMMENT '平台订单创建时间',
order_pay_time TIMESTAMP COMMENT '订单支付时间',
create_time TIMESTAMP COMMENT '创建时间',
is_cancel BIGINT COMMENT '是否取消',
cancel_time TIMESTAMP COMMENT '取消时间',
cancel_reason STRING COMMENT '取消原因',
last_update_time TIMESTAMP COMMENT '最新更新时间',
-- 会员基础信息表
zt_id BIGINT COMMENT '中台ID',
buyer_id BIGINT COMMENT '买家id',
buyer_phone STRING COMMENT '买家电话',
buyer_remark STRING COMMENT '买家备注',
r_name STRING COMMENT '收货人姓名',
r_tel STRING COMMENT '收货人电话',
r_province STRING COMMENT '收货人省份',
r_city STRING COMMENT '收货人城市',
r_district STRING COMMENT '收货人区域',
r_address STRING COMMENT '收货人地址',
r_zipcode STRING COMMENT '收货人邮编',
is_tuan_head BIGINT COMMENT '是否为团长订单',
store_leader_id BIGINT COMMENT '团长id',
order_group_no STRING COMMENT '团单号',
commission_amount DECIMAL(27, 2) COMMENT '抽佣金额',
settle_amount DECIMAL(27, 2) COMMENT '结算金额',
-- 商品表
first_category_no STRING COMMENT '一级分类编码',
first_category_name STRING COMMENT '一级分类名称',
second_category_no STRING COMMENT '二级分类编码',
second_category_name STRING COMMENT '二级分类名称',
third_category_no STRING COMMENT '三级分类编码',
third_category_name STRING COMMENT '三级分类名称',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
weight DECIMAL(27, 3) COMMENT '重量，单位kg',
quantity DECIMAL(27, 3) COMMENT '数量',
unit STRING COMMENT '单位',
sale_qty DECIMAL(27, 3) COMMENT '销售数量',
disp_price DECIMAL(27, 2) COMMENT 'sku展示价格',
pay_price DECIMAL(27, 2) COMMENT '价格',
sale_amount DECIMAL(27, 2) COMMENT '单品销售金额',
dis_amount DECIMAL(27, 2) COMMENT '单品总折扣金额',
sale_cost DECIMAL(27, 2) COMMENT '销售成本',
sale_type BIGINT COMMENT '类型：1-常规；2-赠品',
activity_plat_city_goods_id BIGINT COMMENT '活动商品区域id',
activity_type BIGINT COMMENT '活动类型(11:拼团 21:秒杀)',
order_total_amount DECIMAL(27, 2) COMMENT '订单总金额(平摊)',
order_discount_amount DECIMAL(27, 2) COMMENT '订单优惠金额=商家承担优惠金额+平台补贴金额(平摊)',
order_paid_amount DECIMAL(27, 2) COMMENT '实付金额(平摊)',
balance_amount DECIMAL(27, 2) COMMENT '余额支付',
-- 门店商品信息表
supply_team BIGINT COMMENT '供应链团队 1.平台商品,2.优选标品,3.传智鲜标品,4.传智鲜生鲜,5优选POS商品',
dc_no STRING COMMENT '采购仓库编号',
dc_name STRING COMMENT '采购仓库名称',
group_no STRING COMMENT '采购柜组编号',
group_name STRING COMMENT '采购柜组名称',
trade_mode_id BIGINT COMMENT '结算方式:1购销,2联营',
vendor_id BIGINT COMMENT '供应商ID',
contract_no STRING COMMENT '合同编号'
)
COMMENT '商城核销明细表'
partitioned by (dt STRING COMMENT '核销日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');需求梳理:
一共需要5张事实表: dwd_sold_shop_order_i、 dwd_sold_shop_order_item_i、 dwd_sale_shop_sale_balance_pay_i: dwd_sale_shop_sale_balance_pay_i用order_no与订单表和订单明细表的order_id关联。 ods_sale_shop_refund_i 退款表，主要记录退款单号、时间、原因、金额等信息。 ods_sale_shop_refund_item_i 退款明细表，主要记录退款的商品信息以及活动信息等。
关联事实表思路:
先 完成正常订单的 关联 , 再完成 退款订单的关联 ,最后两个结果合并到一起是5张事实表的最终关联结果
1- 先将 商城订单表、订单明细表、线上余额支付明细 3个表进行关联: 其处理操作与门店销售明细宽表类似 1- 当is_cancel 为1是, 返回5 否则返回0 2- 是否为余额支付, 判断支付表即可 3- commission_amount、settle_amount order_total_amount order_discount_amount order_paid_amount balance_amount 需要计算
2- 接着将 退款表、退款明细表、商城订单表以及线上余额支付明细表 4个表 进行进行关联: 其处理操作与门店销售明细宽表类似 1- 日期使用退款的创建时间即可 2- trade_type(结算类型), 如果cancel_time不为null, 返回5, 否则返回2 3- 需要将计算后的commission_amount(抽佣金额), settle_amount(结算金额)调整为负数 5- quantity,qty,amount,cost调整为负数(做对冲(冲减单)) 6- weight,unit,dis_amount,order_discount_amount调整为0 7- sale_type类型只有1 8- unit设置为空即可
3- 将1和2的结果 基于union all 关联在一起, 然后和五张维度表进行关联: 其处理操作与门店销售明细宽表类似
1.先关联 正常订单 涉及的表:
商城订单表: dwd_sold_shop_order_i、 商城订单明细表: dwd_sold_shop_order_item_i、 线上余额支付明细表: dwd_sale_shop_sale_balance_pay_i 3表关联条件: dwd_sale_shop_sale_balance_pay_i用order_no与订单表和订单明细表的order_id关
2.再关联 退款订单 涉及的表:
因为退款单要获取原单的记录，所以退款表还要与订单表进行关联来取一些字段。比较庆幸的一点是，在退款的时候，订单表也会进行更新，这样在关联订单的时候，只要在对应的那个分区里就可以找到对应的记录，不用再去遍历多个分区来找。
商城退款表:ods_sale_shop_refund_i 商城退款明细表:ods_sale_shop_refund_item_i 商城订单表: dwd_sold_shop_order_i 线上余额支付明细表: dwd_sale_shop_sale_balance_pay_i
-
将1和2的结果 基于union all 关联在一起, 然后和五张维度表进行关联: 其处理操作与门店销售明细宽表类似
需要五张维表： dwd_dim_date_f dwd_dim_store_i dwd_dim_goods_i dwd_dim_store_goods_i ods_mem_member_union_i 注意:会员基础信息表，记录了会员的ID，联系方式，归属门店，来源，状态等信息。
数据插入:
-- 商城核销明细宽表:
with t5 as (
select
t1.complete_time,
date_format(t1.complete_time,'yyyy-MM-dd') as trade_date,
hour(t1.complete_time) as hourly,
minute(t1.complete_time) as minute,
t1.parent_order_no,
t1.order_id,
t1.trade_type,
t1.is_split,
t1.platform_id,
t1.tid,
t1.source_type,
t1.source_name,
t1.order_type,
t1.express_type,
t1.order_status,
t1.order_status_desc,
t1.pay_type,
if(
t3.order_no is not null,
1,
0
) as is_balance_consume,
t1.store_no,
t1.order_create_time,
t1.order_pay_time,
t1.create_time,
if(
t1.is_cancel = 1,
5,
0
) as is_cancel,
t1.cancel_time,
t1.cancel_reason,
t1.last_update_time,
t1.buyer_id,
t1.buyer_phone,
t1.buyer_remark,
t1.r_name,
t1.r_tel,
t1.r_province,
t1.r_city,
t1.r_district,
t1.r_address,
t1.r_zipcode,
t1.is_tuan_head,
t1.store_leader_id,
t1.order_group_no,
t1.commision_amount * t2.sale_amount / t1.real_product_amount as commission_amount, -- 抽佣金额
t1.settle_amount * t2.sale_amount / t1.real_product_amount as settle_amount, -- 结算金额
t2.goods_no,
t2.weight,
t2.quantity,
t2.unit,
t2.sale_qty,
t2.disp_price,
t2.pay_price,
t2.sale_amount,
t2.dis_amount,
t2.sale_cost,
t2.sale_type,
t2.activity_plat_city_goods_id,
t2.activity_type,
t1.order_total_amount * t2.sale_amount / t1.real_product_amount as order_total_amount, -- 订单总金额(平摊)
t1.discount_amount * t2.sale_amount / t1.real_product_amount as order_discount_amount, -- 订单优惠金额 = 商家承担优惠金额 + 平台补贴金额 (平摊)
if(
t1.real_paid_amount = 0,
0,
t1.real_paid_amount * t2.sale_amount / t1.real_product_amount
) as order_paid_amount, -- 实付金额(平摊)
if(
t1.real_paid_amount = 0,
0,
if(
t3.pay_amount is null,
0,
t3.pay_amount * t2.sale_amount / t1.real_product_amount
)
) as balance_amount, -- 余额支付金额 (平摊) 首先判断单据金额是否等于0 , 则结果为0, 如果不是, 判断是否有余额支付金额, 如果有, 使用余额金额 * 商品销售金额 / 单据金额
t2.trade_mode_id,
t2.vendor_id,
t2.contract_no,
t1.dt
from dwd.dwd_sold_shop_order_i t1
join dwd.dwd_sold_shop_order_item_i t2 on t1.order_id = t2.order_id
left join dwd.dwd_sale_shop_sale_balance_pay_i t3 on t1.order_id = t3.order_no
-- 合并
union all
-- 步骤二:
select
t1.create_time as complete_time,
date_format(t1.create_time,'yyyy-MM-dd') as trade_date,
hour(t1.create_time) as hourly,
minute(t1.create_time) as minute,
t3.parent_order_no,
t3.order_id,
if(
t1.cancel_time is not null,
'5',
'2'
) as trade_type,
t3.is_split,
t3.platform_id,
t3.tid,
t3.source_type,
t3.source_name,
t3.order_type,
t3.express_type,
t3.order_status,
t3.order_status_desc,
t3.pay_type,
if(
t1.order_no is not null,
1,
0
) as is_balance_consume,
t1.store_no,
t3.order_create_time,
t3.order_pay_time,
t3.create_time,
if(
t3.is_cancel = 1,
5,
0
) as is_cancel,
t1.cancel_time,
t3.cancel_reason,
t3.last_update_time,
t3.buyer_id,
t3.buyer_phone,
t3.buyer_remark,
t3.r_name,
t3.r_tel,
t3.r_province,
t3.r_city,
t3.r_district,
t3.r_address,
t3.r_zipcode,
t3.is_tuan_head,
t1.store_leader_id,
t3.order_group_no,
-(t3.commision_amount * t2.amount / t3.real_product_amount) as commission_amount, -- 抽佣金额
-(t3.settle_amount * t2.amount / t3.real_product_amount) as settle_amount, -- 结算金额
t2.goods_no,
0 as weight,
-t2.quantity,
'' as unit,
-t2.qty,
t2.amount / t2.quantity as disp_price,
t2.amount / t2.quantity as pay_price,
-t2.amount ,
0 as dis_amount,
-t2.cost as sale_cost,
1 as sale_type,
t2.activity_plat_city_goods_id,
t2.activity_type,
-t2.amount as order_total_amount, -- 订单总金额(平摊)
0 as order_discount_amount, -- 订单优惠金额 = 商家承担优惠金额 + 平台补贴金额 (平摊)
-t2.amount as order_paid_amount, -- 实付金额(平摊)
- if(
t3.real_paid_amount = 0,
0,
if(
t4.pay_amount is null,
0,
t4.pay_amount * t2.amount / t3.real_product_amount
)
) as balance_amount, -- 余额支付金额 (平摊) 首先判断单据金额是否等于0 , 则结果为0, 如果不是, 判断是否有余额支付金额, 如果有, 使用余额金额 * 商品销售金额 / 单据金额
t2.trade_mode_id,
t2.vendor_id,
t2.contract_no,
t1.dt
from ods.ods_sale_shop_refund_i t1
join ods.ods_sale_shop_refund_item_i t2 on t1.refund_no = t2.refund_no
left join dwd.dwd_sold_shop_order_i t3 on t1.order_no = t3.order_id
left join dwd.dwd_sale_shop_sale_balance_pay_i t4 on t4.order_no = t1.order_no
)
insert overwrite table dwd.dwd_sold_shop_order_dtl_i partition (dt)
select
t5.complete_time,
t5.trade_date,
t6.week_trade_date,
t6.month_trade_date,
t5.hourly,
case
when t5.minute between 0 and 14 then 1
when t5.minute between 15 and 29 then 2
when t5.minute between 30 and 44 then 3
when t5.minute between 45 and 59 then 4
end as quarter,
(
t5.hourly * 4
+
case
when t5.minute between 0 and 14 then 1
when t5.minute between 15 and 29 then 2
when t5.minute between 30 and 44 then 3
when t5.minute between 45 and 59 then 4
end
) as quarters,
t5.parent_order_no,
t5.order_id,
t5.trade_type,
t5.is_split,
t5.platform_id,
t5.tid,
t5.source_type,
t5.source_name,
t5.order_type,
t5.express_type,
t5.order_status,
t5.order_status_desc,
t5.pay_type,
t5.is_balance_consume,
t5.store_no,
t7.store_name,
t7.store_sale_type,
t7.store_type_code,
t7.worker_num,
t7.store_area,
t7.city_id,
t7.city_name,
t7.region_code,
t7.region_name,
t7.is_day_clear,
t5.order_create_time,
t5.order_pay_time,
t5.create_time,
t5.is_cancel,
t5.cancel_time,
t5.cancel_reason,
t5.last_update_time,
t10.zt_id,
t5.buyer_id,
t5.buyer_phone,
t5.buyer_remark,
t5.r_name,
t5.r_tel,
t5.r_province,
t5.r_city,
t5.r_district,
t5.r_address,
t5.r_zipcode,
t5.is_tuan_head,
t5.store_leader_id,
t5.order_group_no,
t5.commission_amount,
t5.settle_amount,
t8.first_category_no,
t8.first_category_name,
t8.second_category_no,
t8.second_category_name,
t8.third_category_no,
t8.third_category_name,
t5.goods_no,
t8.goods_name,
t5.weight,
t5.quantity,
t5.unit,
t5.sale_qty,
t5.disp_price,
t5.pay_price,
t5.sale_amount,
t5.dis_amount,
t5.sale_cost,
t5.sale_type,
t5.activity_plat_city_goods_id,
t5.activity_type,
t5.order_total_amount,
t5.order_discount_amount,
t5.order_paid_amount,
t5.balance_amount,
coalesce(t9.tag,4) as supply_team,
coalesce(t9.dc_no,-1) as dc_no,
coalesce(t9.dc_name,'其他仓') as dc_name,
coalesce(t9.group_no,-1) as group_no,
coalesce(t9.group_name,'其他柜组') as group_name ,
t5.trade_mode_id,
coalesce(t5.vendor_id,0) as vendor_id,
t5.contract_no,
t5.dt
from t5
left join dim.dwd_dim_date_f t6 on t5.trade_date = t6.trade_date
left join dim.dwd_dim_store_i t7 on t5.store_no = t7.store_no and t7.dt = '2023-11-23'
left join dim.dwd_dim_goods_i t8 on t5.goods_no = t8.goods_no and t8.dt = '2023-11-23'
left join dim.dwd_dim_store_goods_i t9 on t8.goods_no = t9.goods_no and t7.store_no = t9.store_no and t9.dt = '2023-11-23'
left join ods.ods_mem_member_union_i t10 on t5.buyer_id = t10.member_id;1.1.6 门店库调表
库调即库存调整，类型有日清活动、盘点更正、报损/溢等。对于这些情况，都是广义上的损耗。为了计算损耗，需要对库调表进行加宽处理。
需求说明: 对源表进行加宽处理，处理时间、添加门店信息等。
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_stock_store_stock_adj_i(
trade_date STRING COMMENT '库存处理日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '库调单号',
order_source BIGINT COMMENT '下单来源，小程序、pc等',
store_no STRING COMMENT '店铺编码',
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
adj_type_big STRING COMMENT '库调类型（大类）',
adj_type_small STRING COMMENT '库调类型（小类）',
adj_reason_big STRING COMMENT '库调原因（大类）',
adj_reason_small STRING COMMENT '库调原因（小类）',
adj_qty DECIMAL(27, 3) COMMENT '库调数量',
adj_price DECIMAL(27, 2) COMMENT '库调单价',
adj_amount DECIMAL(27, 2) COMMENT '库调金额',
create_time TIMESTAMP COMMENT '创建时间',
stock_deal_time TIMESTAMP COMMENT '库存处理时间',
sync_time TIMESTAMP COMMENT '数据同步时间',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称'
)
COMMENT '门店库调单'
partitioned by(dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');涉及表: 门店库调表 和 日期表以及分店信息表 进行关联
分析处理:
时间采用库存处理时间
数据插入:
insert overwrite table dwd.dwd_stock_store_stock_adj_i partition (dt)
select
t2.trade_date,
t2.week_trade_date,
t2.month_trade_date,
hour(t1.stock_deal_time) as hourly,
case
when minute(t1.stock_deal_time) between 0 and 14 then 1
when minute(t1.stock_deal_time) between 15 and 29 then 2
when minute(t1.stock_deal_time) between 30 and 44 then 3
when minute(t1.stock_deal_time) between 45 and 59 then 4
end as quarter,
(
hour(t1.stock_deal_time) * 4
+
case
when minute(t1.stock_deal_time) between 0 and 14 then 1
when minute(t1.stock_deal_time) between 15 and 29 then 2
when minute(t1.stock_deal_time) between 30 and 44 then 3
when minute(t1.stock_deal_time) between 45 and 59 then 4
end
) as quarters,
t1.id,
t1.uid,
t1.order_id,
t1.order_source,
t1.store_no,
t1.store_name,
t3.store_sale_type,
t3.store_type_code,
t3.worker_num,
t3.store_area,
t3.city_id,
t3.city_name,
t3.region_code,
t3.region_name,
t3.is_day_clear,
t1.goods_no,
t1.goods_name,
t1.adj_type_big,
t1.adj_type_small,
t1.adj_reason_big,
t1.adj_reason_small,
t1.adj_qty,
t1.adj_price,
t1.adj_amount,
t1.create_time,
t1.stock_deal_time,
t1.sync_time,
t1.vendor_no,
t1.vendor_name,
t2.trade_date as dt
from ods.ods_stock_store_stock_adj_i t1
left join dim.dwd_dim_date_f t2 on date_format(t1.stock_deal_time,'yyyy-MM-dd') = t2.trade_date
left join dim.dwd_dim_store_i t3 on t3.dt = '2023-11-23' and t1.store_no = t3.store_no;1.1.7 门店收货单
门店的收货情况，包括单号，时间，商品信息，数量和金额等。
涉及表: ods_order_store_receive_i(门店收货单)、时间表、分店信息表关联
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_order_store_receive_i(
trade_date STRING COMMENT '库存处理日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '收货单号',
order_source BIGINT COMMENT '下单来源，小程序、pc等',
store_no STRING COMMENT '店铺编码',
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
dc_no STRING COMMENT '配送中心编码',
dc_name STRING COMMENT '配送中心名称',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称',
order_type BIGINT COMMENT '订单类型，1-直送，2-配送，3-代发',
receive_price DECIMAL(27, 2) COMMENT '收货价',
receive_qty DECIMAL(27, 3) COMMENT '收货数量',
git_qty DECIMAL(27, 3) COMMENT '赠品数量',
create_time TIMESTAMP COMMENT '创建时间',
stock_deal_time TIMESTAMP COMMENT '库存处理时间',
dc_send_order_id STRING COMMENT '仓库发货单号',
red_order_id STRING COMMENT '被红冲单号',
contract_no STRING COMMENT '合同编号',
contract_name STRING COMMENT '合同名称',
trade_mode BIGINT COMMENT '1-直营，2-联营',
order_source_type BIGINT COMMENT '订货标识，0-门店订货，1-采购配货',
sync_time TIMESTAMP COMMENT '数据同步时间'
)
COMMENT '门店收货单'
partitioned by (dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
计算处理: 日期字段 stock_deal_time
insert overwrite table dwd.dwd_order_store_receive_i partition(dt)
select
date_format(t.stock_deal_time,'yyyy-MM-dd') as trade_date
,dd.week_trade_date
,dd.month_trade_date
,hour(t.stock_deal_time) as hourly
,case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as `quarter`
,hour(t.stock_deal_time)*4 + case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as quarters
,t.id
,t.uid
,t.order_id
,t.order_source
,t.store_no
,bs.store_name
,bs.store_sale_type
,bs.store_type_code
,bs.worker_num
,bs.store_area
,bs.city_id
,bs.city_name
,bs.region_code
,bs.region_name
,bs.is_day_clear
,t.goods_no
,t.goods_name
,t.dc_no
,t.dc_name
,t.vendor_no
,t.vendor_name
,t.order_type
,t.receive_price
,t.receive_qty
,t.git_qty
,t.create_time
,t.stock_deal_time
,t.dc_send_order_id
,t.red_order_id
,t.contract_no
,t.contract_name
,t.trade_mode
,t.order_source_type
,t.sync_time
,date_format(t.stock_deal_time,'yyyy-MM-dd') as dt
from ods.ods_order_store_receive_i t
inner join dim.dwd_dim_date_f as dd
on date_format(t.stock_deal_time,'yyyy-MM-dd')=dd.trade_date
inner join dim.dwd_dim_store_i as bs
on t.store_no=bs.store_no and bs.dt= '2023-11-23';1.1.8 门店退货单
门店退给供应商的商品情况，包括单号、门店、时间、商品信息、数量、金额、原因等。
涉及表: ods_order_store_return_to_vendor_i(门店退货单)、时间表、门店信息表
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_order_store_return_to_vendor_i(
trade_date STRING COMMENT '库存处理日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '退配单号',
order_source BIGINT COMMENT '下单来源，小程序、pc、智能补货、系统等',
store_no STRING COMMENT '门店编码',
store_name STRING COMMENT '门店名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
dc_no STRING COMMENT '配送中心编码',
dc_name STRING COMMENT '配送中心名称',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称',
return_price DECIMAL(27, 2) COMMENT '退配价',
return_qty DECIMAL(27, 3) COMMENT '退配数量',
create_time TIMESTAMP COMMENT '创建时间',
stock_deal_time TIMESTAMP COMMENT '库存处理时间',
original_order_id STRING COMMENT '退配原单号',
is_fresh BIGINT COMMENT '是否为生鲜店，0-否，1-是',
is_entity BIGINT COMMENT '是否实物退回，0-否，1-是',
responsible_person STRING COMMENT '责任归属方',
return_reason_big STRING COMMENT '退配原因（大类）',
return_desc_big STRING COMMENT '退配说明（大类）',
return_reason_small STRING COMMENT '退配原因（小类）',
return_desc_small STRING COMMENT '退配说明（小类）',
sync_time TIMESTAMP COMMENT '数据同步时间'
)
COMMENT '门店退货单'
partitioned by (dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
计算:时间字段 stock_deal_time
insert overwrite table dwd.dwd_order_store_return_to_vendor_i partition (dt)
select
date_format(t.stock_deal_time,'yyyy-MM-dd') as trade_date
,dd.week_trade_date
,dd.month_trade_date
,hour(t.stock_deal_time) as hourly
,case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as `quarter`
,hour(t.stock_deal_time)*4 + case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as quarters
,t.id
,t.uid
,t.order_id
,t.order_source
,t.store_no
,bs.store_name
,bs.store_sale_type
,bs.store_type_code
,bs.worker_num
,bs.store_area
,bs.city_id
,bs.city_name
,bs.region_code
,bs.region_name
,bs.is_day_clear
,t.goods_no
,t.goods_name
,t.dc_no
,t.dc_name
,t.vendor_no
,t.vendor_name
,t.return_price
,t.return_qty
,t.create_time
,t.stock_deal_time
,t.original_order_id
,t.is_fresh
,t.is_entity
,t.responsible_person
,t.return_reason_big
,t.return_desc_big
,t.return_reason_small
,t.return_desc_small
,t.sync_time
,date_format(t.stock_deal_time,'yyyy-MM-dd') as dt
from ods.ods_order_store_return_to_vendor_i t
inner join dim.dwd_dim_date_f as dd
on date_format(t.stock_deal_time,'yyyy-MM-dd') = dd.trade_date
inner join dim.dwd_dim_store_i as bs
on t.store_no=bs.store_no and bs.dt = '2023-11-23'
;1.1.9 门店退配单
门店退给大仓的商品情况，包括单号、门店、时间、商品信息、数量、金额、原因等。
涉及表: ods_order_store_return_to_dc_i(门店退配单)、时间表、门店信息表
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_order_store_return_to_dc_i(
trade_date STRING COMMENT '库存处理日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '退配单号',
order_source BIGINT COMMENT '下单来源，小程序、pc、智能补货、系统等',
store_no STRING COMMENT '门店编码',
store_name STRING COMMENT '门店名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
dc_no STRING COMMENT '配送中心编码',
dc_name STRING COMMENT '配送中心名称',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称',
return_price DECIMAL(27, 2) COMMENT '退配价',
return_qty DECIMAL(27, 3) COMMENT '退配数量',
create_time TIMESTAMP COMMENT '创建时间',
stock_deal_time TIMESTAMP COMMENT '库存处理时间',
original_order_id STRING COMMENT '退配原单号',
is_fresh BIGINT COMMENT '是否为生鲜店，0-否，1-是',
is_entity BIGINT COMMENT '是否实物退回，0-否，1-是',
responsible_person STRING COMMENT '责任归属方',
return_reason_big STRING COMMENT '退配原因（大类）',
return_desc_big STRING COMMENT '退配说明（大类）',
return_reason_small STRING COMMENT '退配原因（小类）',
return_desc_small STRING COMMENT '退配说明（小类）',
sync_time TIMESTAMP COMMENT '数据同步时间',
batch_type_id STRING COMMENT '批次类型id'
)
COMMENT '门店退配单'
partitioned by (dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
计算:时间字段 stock_deal_time
insert overwrite table dwd.dwd_order_store_return_to_dc_i partition (dt)
select
date_format(t.stock_deal_time,'yyyy-MM-dd') as trade_date
,dd.week_trade_date
,dd.month_trade_date
,hour(t.stock_deal_time) as hourly
,case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as `quarter`
,hour(t.stock_deal_time)*4 + case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as quarters
,t.id
,t.uid
,t.order_id
,t.order_source
,t.store_no
,bs.store_name
,bs.store_sale_type
,bs.store_type_code
,bs.worker_num
,bs.store_area
,bs.city_id
,bs.city_name
,bs.region_code
,bs.region_name
,bs.is_day_clear
,t.goods_no
,t.goods_name
,t.dc_no
,t.dc_name
,t.vendor_no
,t.vendor_name
,t.return_price
,t.return_qty
,t.create_time
,t.stock_deal_time
,t.original_order_id
,t.is_fresh
,t.is_entity
,t.responsible_person
,t.return_reason_big
,t.return_desc_big
,t.return_reason_small
,t.return_desc_small
,t.sync_time
,t.batch_type_id
,date_format(t.stock_deal_time,'yyyy-MM-dd') as dt
from ods.ods_order_store_return_to_dc_i t
inner join dim.dwd_dim_date_f as dd
on date_format(t.stock_deal_time,'yyyy-MM-dd') = dd.trade_date
inner join dim.dwd_dim_store_i as bs
on t.store_no = bs.store_no and bs.dt = '2023-11-23'
;1.1.10 门店调入单
从其他门店调入本店的单据信息，包括时间、单号、商品、门店、商品信息、数量和金额等。
涉及表: ods_order_store_alloc_in_i(门店调入单) 、时间表、门店表
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_order_store_alloc_in_i(
trade_date STRING COMMENT '库存处理日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '调拨单号',
order_source BIGINT COMMENT '下单来源，小程序、PC等',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
alloc_in_store_no STRING COMMENT '调入门店编码',
alloc_in_store_name STRING COMMENT '调入门店名称',
alloc_in_store_sale_type BIGINT COMMENT '调入店铺销售类型',
alloc_in_store_type_code BIGINT COMMENT '调入分店类型',
alloc_in_worker_num BIGINT COMMENT '调入门店员工人数',
alloc_in_store_area DECIMAL(27, 2) COMMENT '调入门店面积',
alloc_in_city_id BIGINT COMMENT '调入门店城市ID',
alloc_in_city_name STRING COMMENT '调入门店城市名称',
alloc_in_region_code STRING COMMENT '调入门店区域编码',
alloc_in_region_name STRING COMMENT '调入门店区域名称',
alloc_in_is_day_clear BIGINT COMMENT '调入门店是否日清:0否,1是',
alloc_out_store_no STRING COMMENT '调出门店编码',
alloc_out_store_name STRING COMMENT '调出门店名称',
alloc_price DECIMAL(27, 2) COMMENT '调拨单价',
alloc_qty DECIMAL(27, 3) COMMENT '调拨数量',
alloc_reason STRING COMMENT '调拨原因',
alloc_amount DECIMAL(27, 2) COMMENT '调拨金额',
create_time TIMESTAMP COMMENT '创建时间',
stock_deal_time TIMESTAMP COMMENT '库存处理时间',
sync_time TIMESTAMP COMMENT '数据同步时间',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称'
)
COMMENT '门店调入单'
partitioned by (dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
计算:时间字段为stock_deal_time
insert overwrite table dwd.dwd_order_store_alloc_in_i partition (dt)
select
date_format(t.stock_deal_time,'yyyy-MM-dd') as trade_date
,dd.week_trade_date
,dd.month_trade_date
,hour(t.stock_deal_time) as hourly
,case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as `quarter`
,hour(t.stock_deal_time)*4 + case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as quarters
,t.id
,t.uid
,t.order_id
,t.order_source
,t.goods_no
,t.goods_name
,t.alloc_in_store_no
,bs.store_name as alloc_in_store_name
,bs.store_sale_type as alloc_in_store_sale_type
,bs.store_type_code as alloc_in_store_type_code
,bs.worker_num as alloc_in_worker_num
,bs.store_area as alloc_in_store_area
,bs.city_id as alloc_in_city_id
,bs.city_name as alloc_in_city_name
,bs.region_code as alloc_in_region_code
,bs.region_name as alloc_in_region_name
,bs.is_day_clear as alloc_in_is_clear
,t.alloc_out_store_no
,t.alloc_out_store_name
,t.alloc_price
,t.alloc_qty
,t.alloc_reason
,t.alloc_amount
,t.create_time
,t.stock_deal_time
,t.sync_time
,t.vendor_no
,t.vendor_name
,date_format(t.stock_deal_time,'yyyy-MM-dd') as dt
from ods.ods_order_store_alloc_in_i t
inner join dim.dwd_dim_date_f as dd
on date_format(t.stock_deal_time,'yyyy-MM-dd') = dd.trade_date
inner join dim.dwd_dim_store_i as bs
on t.alloc_in_store_no=bs.store_no and bs.dt = '2023-11-23'
;1.1.11 门店调出单
从本店调入其他门店的单据信息，包括时间、单号、商品、门店、商品信息、数量和金额等。
涉及表: ods_order_store_alloc_out_i(门店调出单)、时间表、门店信息表
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_order_store_alloc_out_i(
trade_date STRING COMMENT '库存处理日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '调拨单号',
order_source BIGINT COMMENT '下单来源，小程序、PC等',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
alloc_in_store_no STRING COMMENT '调入门店编码',
alloc_in_store_name STRING COMMENT '调入门店名称',
alloc_out_store_no STRING COMMENT '调出门店编码',
alloc_out_store_name STRING COMMENT '调出门店名称',
alloc_out_store_sale_type BIGINT COMMENT '调出店铺销售类型',
alloc_out_store_type_code BIGINT COMMENT '调出分店类型',
alloc_out_worker_num BIGINT COMMENT '调出门店员工人数',
alloc_out_store_area DECIMAL(27, 2) COMMENT '调出门店面积',
alloc_out_city_id BIGINT COMMENT '调出门店城市ID',
alloc_out_city_name STRING COMMENT '调出门店城市名称',
alloc_out_region_code STRING COMMENT '调出门店区域编码',
alloc_out_region_name STRING COMMENT '调出门店区域名称',
alloc_out_is_day_clear BIGINT COMMENT '调出门店是否日清:0否,1是',
alloc_price DECIMAL(27, 2) COMMENT '调拨单价',
alloc_qty DECIMAL(27, 3) COMMENT '调拨数量',
alloc_reason STRING COMMENT '调拨原因',
alloc_amount DECIMAL(27, 2) COMMENT '调拨金额',
create_time TIMESTAMP COMMENT '创建时间',
stock_deal_time TIMESTAMP COMMENT '库存处理时间',
sync_time TIMESTAMP COMMENT '数据同步时间',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称'
)
COMMENT '门店调出单'
partitioned by (dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
计算SQL: 时间字段为stock_deal_time
insert overwrite table dwd.dwd_order_store_alloc_out_i partition (dt)
select
date_format(t.stock_deal_time,'yyyy-MM-dd') as trade_date
,dd.week_trade_date
,dd.month_trade_date
,hour(t.stock_deal_time) as hourly
,case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as `quarter`
,hour(t.stock_deal_time)*4 + case when minute(t.stock_deal_time)>=0 and minute(t.stock_deal_time)<15 then 1
when minute(t.stock_deal_time)>=15 and minute(t.stock_deal_time)<30 then 2
when minute(t.stock_deal_time)>=30 and minute(t.stock_deal_time)<45 then 3
when minute(t.stock_deal_time)>=45 and minute(t.stock_deal_time)<60 then 4
end as quarters
,t.id
,t.uid
,t.order_id
,t.order_source
,t.goods_no
,t.goods_name
,t.alloc_in_store_no
,t.alloc_in_store_name
,t.alloc_out_store_no
,bs.store_name as alloc_out_store_name
,bs.store_sale_type as alloc_out_store_sale_type
,bs.store_type_code as alloc_out_store_type_code
,bs.worker_num as alloc_out_worker_num
,bs.store_area as alloc_out_store_area
,bs.city_id as alloc_out_city_id
,bs.city_name as alloc_out_city_name
,bs.region_code as alloc_out_region_code
,bs.region_name as alloc_out_region_name
,bs.is_day_clear as alloc_out_is_clear
,t.alloc_price
,t.alloc_qty
,t.alloc_reason
,t.alloc_amount
,t.create_time
,t.stock_deal_time
,t.sync_time
,t.vendor_no
,t.vendor_name
,date_format(t.stock_deal_time,'yyyy-MM-dd') as dt
from ods.ods_order_store_alloc_out_i t
inner join dim.dwd_dim_date_f as dd
on date_format(t.stock_deal_time,'yyyy-MM-dd') = dd.trade_date
inner join dim.dwd_dim_store_i as bs
on t.alloc_out_store_no=bs.store_no and bs.dt = '2023-11-23'
;1.1.12 门店要货单
门店的要货信息，包括时间、单号、门店、商品信息、数量和金额、要货方式、状态等。
涉及表: ods_order_store_require_i(门店要货单)和日期表和门店表
建表语句:
CREATE TABLE IF NOT EXISTS dwd.dwd_order_store_require_i(
trade_date STRING COMMENT '确认日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
id BIGINT COMMENT '主键',
uid STRING COMMENT '唯一标识',
order_id STRING COMMENT '要货单号',
order_source BIGINT COMMENT '下单来源，小程序、pc、智能补货、系统等',
store_no STRING COMMENT '门店编码',
store_name STRING COMMENT '门店名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
dc_no STRING COMMENT '配送中心编码',
dc_name STRING COMMENT '配送中心名称',
vendor_no STRING COMMENT '供应商编码',
vendor_name STRING COMMENT '供应商名称',
group_no STRING COMMENT '采购柜组编号',
require_price DECIMAL(27, 2) COMMENT '要货价格',
require_qty DECIMAL(27, 3) COMMENT '要货数量',
create_time TIMESTAMP COMMENT '创建时间',
send_time TIMESTAMP COMMENT '预计送货时间',
collect_require_order_id STRING COMMENT '要货汇总单号',
require_type_code BIGINT COMMENT '要货类型：1-直送，2-配送，3-代发',
is_online BIGINT COMMENT '1-线上，0-线下',
confirm_time TIMESTAMP COMMENT '审核时间',
is_canceled BIGINT COMMENT '1-取消，0-正常',
sync_time TIMESTAMP COMMENT '数据同步时间',
is_urgent BIGINT COMMENT '是否加急 0,否 1，是',
original_order_price DECIMAL(27, 2) COMMENT '原单价'
)
COMMENT '门店要货单'
partitioned by (dt STRING COMMENT 'confirm_time时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');数据插入:
计算: 时间字段为confirm_time
insert overwrite table dwd.dwd_order_store_require_i partition (dt)
select
date_format(t.confirm_time,'yyyy-MM-dd') as trade_date
,dd.week_trade_date
,dd.month_trade_date
,hour(t.confirm_time) as hourly
,case when minute(t.confirm_time)>=0 and minute(t.confirm_time)<15 then 1
when minute(t.confirm_time)>=15 and minute(t.confirm_time)<30 then 2
when minute(t.confirm_time)>=30 and minute(t.confirm_time)<45 then 3
when minute(t.confirm_time)>=45 and minute(t.confirm_time)<60 then 4
end as `quarter`
,hour(t.confirm_time)*4 + case when minute(t.confirm_time)>=0 and minute(t.confirm_time)<15 then 1
when minute(t.confirm_time)>=15 and minute(t.confirm_time)<30 then 2
when minute(t.confirm_time)>=30 and minute(t.confirm_time)<45 then 3
when minute(t.confirm_time)>=45 and minute(t.confirm_time)<60 then 4
end as quarters
,t.id
,t.uid
,t.order_id
,t.order_source
,t.store_no
,bs.store_name
,bs.store_sale_type
,bs.store_type_code
,bs.worker_num
,bs.store_area
,bs.city_id
,bs.city_name
,bs.region_code
,bs.region_name
,bs.is_day_clear
,t.goods_no
,t.goods_name
,t.dc_no
,t.dc_name
,t.vendor_no
,t.vendor_name
,t.group_no
,t.require_price
,t.require_qty
,t.create_time
,t.send_time
,t.collect_require_order_id
,t.require_type_code
,t.is_online
,t.confirm_time
,t.is_canceled
,t.sync_time
,t.is_urgent
,t.original_order_price
,date_format(t.confirm_time,'yyyy-MM-dd') as dt
from ods.ods_order_store_require_i t
inner join dim.dwd_dim_date_f as dd
on date_format(t.confirm_time,'yyyy-MM-dd') = dd.trade_date
inner join dim.dwd_dim_store_i as bs
on t.store_no=bs.store_no and bs.dt = '2023-11-23'
;1.2 DWM层
数据仓库中间层: ==Data Warehouse Middle==
为DWS层准备数据,提前合并部分数据
注意：具体计算只计算增量，全量计算直接去掉where条件即可。或者在调度时使用补数的方法，将历史分区一天一天跑出来即可！
1.2.1 商品销售明细(核销)
dwm_sold_goods_sold_dtl_i作为核销主题的小宽表，要能涵盖线上线下各渠道的销售数据。这个逻辑比较简单，将dwd_sale_store_sale_dtl_i与dwd_sold_shop_order_dtl_i两表进行合并即可。需要注意的就是字段要对齐，对于补充的字段要按照定义进行补充。比如 0 as is_online_order,4 as order_type ,4 as express_type 等。
建表语句:
CREATE TABLE IF NOT EXISTS dwm.dwm_sold_goods_sold_dtl_i(
trade_date_time STRING COMMENT '核销时间',
trade_date STRING COMMENT '交易日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
parent_store_no STRING COMMENT '母店编码',
store_no STRING COMMENT '店铺编码',
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
trade_type BIGINT COMMENT '结算类型(0.正常交易,1.赠品发放,2.退货,4.培训,5.取消交易)',
source_type BIGINT COMMENT '交易来源1:线下POS;2:三方平台;3:传智鲜商城;4:黑马优选团;5:传智大客户;6:传智其他;7:黑马优选;8:优选海淘;9:优选大客户;10:优选POS;11:优选APP;12:优选H5;13:店长工具线下;14:店长工具线上;15:黑马其他',
source_type_name STRING COMMENT '交易来源名称',
sale_type BIGINT COMMENT '销售类型 1.实物,2.代客,3.优选小程序,4.离店,5.传智鲜小程序,6.第三方平台,7.其他,8.大客户',
is_online_order BIGINT COMMENT '是否为线上单:0否,1是',
member_type BIGINT COMMENT '会员类型:0非会员,1线上会员,2实体卡会员',
is_balance_consume BIGINT COMMENT '是否有余额支付:0否,1是',
order_type BIGINT COMMENT '配送类型（真正的订单类型由业务类型来决定）：1-及时送；2-隔日送；3-自提单；4-线下单',
express_type BIGINT COMMENT '配送方式：0-三方平台配送；1-自配送；2-快递；3-自提；4-线下',
parent_order_no STRING COMMENT '母订单编号',
order_no STRING COMMENT '订单编号',
create_time STRING COMMENT '创建时间',
is_cancel BIGINT COMMENT '是否取消',
cancel_time STRING COMMENT '取消时间',
last_update_time STRING COMMENT 'pos_sale表最后一次更新时间',
zt_id BIGINT COMMENT '中台ID',
member_id BIGINT COMMENT '会员ID',
card_no STRING COMMENT '卡号',
share_user_id STRING COMMENT '分享人用户ID',
commission_amount DECIMAL(27, 2) COMMENT '佣金',
is_tuan_head BIGINT COMMENT '是否为团长订单',
store_leader_id BIGINT COMMENT '团长id',
order_group_no STRING COMMENT '团单号',
first_category_no STRING COMMENT '一级分类编码',
first_category_name STRING COMMENT '一级分类名称',
second_category_no STRING COMMENT '二级分类编码',
second_category_name STRING COMMENT '二级分类名称',
third_category_no STRING COMMENT '三级分类编码',
third_category_name STRING COMMENT '三级分类名称',
goods_no STRING COMMENT '商品编码',
goods_name STRING COMMENT '商品名称',
supply_team BIGINT COMMENT '供应链团队 1.平台商品,2.优选标品,3.传智鲜标品,4.传智鲜生鲜,5优选POS商品',
dc_no STRING COMMENT '采购仓库编号',
dc_name STRING COMMENT '采购仓库名称',
group_no STRING COMMENT '采购柜组编号',
group_name STRING COMMENT '采购柜组名称',
trade_mode_id BIGINT COMMENT '结算方式:1购销,2联营',
vendor_id BIGINT COMMENT '供应商ID',
contract_no STRING COMMENT '合同编号',
is_clean BIGINT COMMENT '商品是否日清:0否,1是',
is_daily_clear BIGINT COMMENT '商品是否参加日清活动:0否,1是',
sale_qty DECIMAL(27, 3) COMMENT '商品销售数量',
sale_amount DECIMAL(27, 2) COMMENT '商品销售金额',
dis_amount DECIMAL(27, 2) COMMENT '商品折扣金额',
sale_cost DECIMAL(27, 2) COMMENT '商品销售成本',
balance_amount DECIMAL(27, 2) COMMENT '余额支付',
order_total_amount DECIMAL(27, 2) COMMENT '订单总金额(平摊)',
order_discount_amount DECIMAL(27, 2) COMMENT '订单优惠金额=商家承担优惠金额+平台补贴金额(平摊)',
order_paid_amount DECIMAL(27, 2) COMMENT '实付金额(平摊)'
)
COMMENT '商品销售明细(核销)'
partitioned by(dt STRING COMMENT '核销日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');处理说明:
1.dwd.dwd_sale_store_sale_dtl_i(门店销售明细宽表): 1- 新增is_online_order(是否为线上单), 设置值为0 2- 新增order_type(配送类型), 设置值为4 3- 新增express_type(配送方式), 设置值为4 4- 新增is_cancel(是否取消), 判断: 根据trade_type(结算类型)当为5返回1, 否则返回0 5- 新增cancel_time(取消时间),判断: 当 trade_type(结算类型)为5返回last_update_time,否则为空 6- 新增is_tuan_head(是否为团长订单), 设置值为0 7- 新增store_leader_id(团长ID) , 设置为0 8- 新增order_group_no(团单号), 设置为0 9- 新增order_total_amount(订单总金额(平摊))、order_discount_amount(订单优惠金额)、order_paid_amount(实付金额(平摊)) 分别对应的值为 sale_amount、dis_amount、sale_amount 2.dwd.dwd_sold_shop_order_dtl_i(商城核销明细表): 1- 修改 source_type, 由原来的订单来源, 变更为交易来源: 需要对应值重新映射 2- 新增 source_type_name(交易来源名称), 设置对应编号的对应值即可 3- 修改 sale_type为销售类型, 基于 原表中source_type 4- order_no(订单编号)为order_id 5- member_id(会员ID)为buyer_id 6- 新增 card_no(卡号), 值为空 7- 新增 share_user_id(分享人用户ID), 值为0 8- 新增is_clean(商品是否日清:0否,1是) 和 is_daily_clear(商品是否参加日清活动:0否,1是) 值为0 9- create_time、cancel_time、last_update_time 将三个时间戳类型 转换为字符串 10- 类型调整: order_group_no 调整为int类型 3.上述两个表union all即可
数据插入:
-- DWM层: 销售明细宽表(将线上和线下的两部分数据进行union all 合并)
insert overwrite table dwm.dwm_sold_goods_sold_dtl_i partition(dt)
select
trade_date_time,
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
parent_store_no,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
trade_type,
source_type,
source_type_name,
sale_type,
0 as is_online_order,
member_type,
is_balance_consume,
4 as order_type,
4 as express_type,
parent_order_no,
order_no,
trade_date_time as create_time,
if(
trade_type = 5,
1,
0
) as is_cancel,
if(
trade_type = 5,
last_update_time,
''
) as cancel_time,
last_update_time,
zt_id,
member_id,
card_no,
cast(share_user_id as int) as share_user_id,
commission_amount,
0 as is_tuan_head,
0 as store_leader_id,
'0' as order_group_no,
first_category_no,
first_category_name,
second_category_no,
second_category_name,
third_category_no,
third_category_name,
goods_no,
goods_name,
supply_team,
dc_no,
dc_name,
group_no,
group_name,
trade_mode_id,
vendor_id,
contract_no,
is_clean,
is_daily_clear,
sale_qty,
sale_amount,
dis_amount,
sale_cost,
balance_amount,
sale_amount as order_total_amount,
dis_amount as order_discount_amount,
sale_amount as order_paid_amount,
dt
from dwd.dwd_sale_store_sale_dtl_i -- 后续增量加上 where dt = '昨天'
union all
select
cast(complete_time as timestamp) as trade_date_time,
cast(trade_date as timestamp) as trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no as parent_store_no,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
trade_type,
if(
source_type in(10,20,30,40,41,70),
2,
if(
source_type = 50,
3,
7
)
) as source_type,
if(
source_type in(10,20,30,40,41,70),
'三方平台',
if(
source_type = 50,
'传智鲜商城',
'黑马优选'
)
) as source_type_name,
if(
source_type in(10,20,30,40,41,70),
6,
if(
source_type = 50,
5,
3
)
) as sale_type,
1 as is_online_order,
1 as member_type,
is_balance_consume,
order_type,
express_type,
parent_order_no,
order_id as order_no,
create_time,
is_cancel,
date_format(cancel_time,'yyyy-MM-dd HH:mm:ss') as cancel_time,
date_format(last_update_time,'yyyy-MM-dd HH:mm:ss') as last_update_time,
zt_id,
buyer_id as member_id,
'' as card_no,
0 as share_user_id,
commission_amount,
is_tuan_head,
store_leader_id,
order_group_no,
first_category_no,
first_category_name,
second_category_no,
second_category_name,
third_category_no,
third_category_name,
goods_no,
goods_name,
supply_team,
dc_no,
dc_name,
group_no,
group_name,
trade_mode_id,
vendor_id,
contract_no,
0 as is_clean,
0 as is_daily_clear,
sale_qty,
sale_amount,
dis_amount,
sale_cost,
balance_amount,
order_total_amount,
order_discount_amount,
order_paid_amount,
dt
from dwd.dwd_sold_shop_order_dtl_i;1.2.2 门店商品损耗刻表
门店商品损耗刻表，粒度为门店商品刻，包含时间、门店、商品、品类等信息，便于dws/ads层使用。
建表语句:
CREATE TABLE IF NOT EXISTS dwm.dwm_stock_store_goods_loss_quarter_i(
trade_date STRING COMMENT '交易日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
store_no STRING COMMENT '店铺编码',
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
-- 商品表
first_category_no STRING COMMENT '一级分类编码',
first_category_name STRING COMMENT '一级分类名称',
second_category_no STRING COMMENT '二级分类编码',
second_category_name STRING COMMENT '二级分类名称',
third_category_no STRING COMMENT '三级分类编码',
third_category_name STRING COMMENT '三级分类名称',
goods_no STRING COMMENT '商品编码',
-- 商品表
goods_name STRING COMMENT '商品名称',
-- 门店日清商品表
is_clean BIGINT COMMENT '商品是否日清',
loss_qty DECIMAL(27, 3) COMMENT '损耗数量',
loss_amount DECIMAL(27, 2) COMMENT '损耗金额'
)
COMMENT '门店商品损耗刻表'
partitioned by(dt STRING COMMENT '库存处理时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');处理说明:
基于dwd_stock_store_stock_adj_i表进行计算，因为dwd_stock_store_stock_adj_i是明细表，需要进行聚合操作。聚合之后再关联维表进行拉宽，可以提高效率。
需求说明:
根据维表表以外的其他的字段进行分组, 聚合损耗数量和损耗金额
损耗类别: adj_type_big in ('日清', '报损/溢', '人工盘点', '盘点更正', '周清')
数据插入:
-- 计算损耗
with t1 as (
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no,
sum(adj_qty) as loss_qty,
sum(adj_amount) as loss_amount
from dwd.dwd_stock_store_stock_adj_i where adj_type_big in ('日清','报损/溢','人工盘点','盘点更正','周清')
group by
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no
)
insert overwrite table dwm.dwm_stock_store_goods_loss_quarter_i partition (dt)
select
t1.trade_date,
t1.week_trade_date,
t1.month_trade_date,
t1.hourly,
t1.quarter,
t1.quarters,
t1.store_no,
t1.store_name,
t1.store_sale_type,
t1.store_type_code,
t1.worker_num,
t1.store_area,
t1.city_id,
t1.city_name,
t1.region_code,
t1.region_name,
t1.is_day_clear,
t2.first_category_no,
t2.first_category_name,
t2.second_category_no,
t2.second_category_name,
t2.third_category_no,
t2.third_category_name,
t1.goods_no,
t2.goods_name,
t2.is_clear as is_clean,
t1.loss_qty,
t1.loss_amount,
trade_date as dt
from t1 left join dim.dwd_dim_store_goods_i t2
on t1.store_no = t2.store_no and t1.goods_no = t2.goods_no and t2.dt = '2023-11-23';1.2.3 门店商品收货刻表
净收货=收货-退货-退配+调入-调出，所以需要综合五张dwd的order主题的表。这里为了便于其他分析，不仅存净收货指标，也把收货、退货、退配、调入、调出这几个指标分别存上。
建表语句:
CREATE TABLE IF NOT EXISTS dwm.dwm_order_store_goods_receipt_quarter_i(
trade_date STRING COMMENT '交易日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
store_no STRING COMMENT '店铺编码',
-- 门店信息表
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
-- 商品表
first_category_no STRING COMMENT '一级分类编码',
first_category_name STRING COMMENT '一级分类名称',
second_category_no STRING COMMENT '二级分类编码',
second_category_name STRING COMMENT '二级分类名称',
third_category_no STRING COMMENT '三级分类编码',
third_category_name STRING COMMENT '三级分类名称',
goods_no STRING COMMENT '商品编码',
-- 商品表
goods_name STRING COMMENT '商品名称',
-- 门店日清商品表
is_clean BIGINT COMMENT '商品是否日清',
receive_qty DECIMAL(27, 3) COMMENT '收货数量(从供应商或者仓库)',
receive_amount DECIMAL(27, 2) COMMENT '收货金额',
return_vendor_qty DECIMAL(27, 3) COMMENT '退货数量(退给供应商)',
return_vendor_amount DECIMAL(27, 2) COMMENT '退货金额',
return_dc_qty DECIMAL(27, 3) COMMENT '退配数量(退给仓库)',
return_dc_amount DECIMAL(27, 2) COMMENT '退配金额',
allocation_in_qty DECIMAL(27, 3) COMMENT '调入数量',
allocation_in_amount DECIMAL(27, 2) COMMENT '调入金额',
allocation_out_qty DECIMAL(27, 3) COMMENT '调出数量',
allocation_out_amount DECIMAL(27, 2) COMMENT '调出金额',
receipt_qty DECIMAL(27, 3) COMMENT '净收货数量',
receipt_amount DECIMAL(27, 2) COMMENT '净收货金额(收货-退货-退配+调入-调出)',
receipt_cost DECIMAL(27, 2) COMMENT '净收货成本(总额95%)'
)
COMMENT '门店商品收货刻表'
partitioned by(dt STRING COMMENT '收货日期')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');处理说明:
五张dwd表进行union，然后进行group by即可。需要注意的是，要看清源表的数据的正负，比如退货和退配，本身就是负值，此时直接加减即可。但是调入调出就需要变一下符号。
第一步: 将收货 退货 退配 调入 调出 分别获取出来 第二步: 将五个表union all,生成临时结果集t1 第三步: 执行group by 操作计算最终的收货 退货 退配 调入 调出 相关值,生成临时结果集t2 第四步: 计算最终的净收货,插入到dwm_order_store_goods_receipt_quarter_i表中
数据插入:
with t1 as (
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no,
receive_qty,
receive_price as receive_amount,
0 as return_vendor_qty,
0 as return_vendor_amount,
0 as return_dc_qty,
0 as return_dc_amount,
0 as allocation_in_qty,
0 as allocation_in_amount,
0 as allocation_out_qty,
0 as allocation_out_amount
from dwd.dwd_order_store_receive_i
union all
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no,
0 as receive_qty,
0 as receive_amount,
return_qty as return_vendor_qty,
return_price as return_vendor_amount,
0 as return_dc_qty,
0 as return_dc_amount,
0 as allocation_in_qty,
0 as allocation_in_amount,
0 as allocation_out_qty,
0 as allocation_out_amount
from dwd.dwd_order_store_return_to_vendor_i
union all
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no,
0 as receive_qty,
0 as receive_amount,
0 as return_vendor_qty,
0 as return_vendor_amount,
return_qty as return_dc_qty,
return_price as return_dc_amount,
0 as allocation_in_qty,
0 as allocation_in_amount,
0 as allocation_out_qty,
0 as allocation_out_amount
from dwd.dwd_order_store_return_to_dc_i
union all
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
alloc_in_store_no as store_no,
alloc_in_store_name as store_name,
alloc_in_store_sale_type as store_sale_type,
alloc_in_store_type_code as store_type_code,
alloc_in_worker_num as worker_num,
alloc_in_store_area as store_area,
alloc_in_city_id as city_id,
alloc_in_city_name as city_name,
alloc_in_region_code as region_code,
alloc_in_region_name as region_name,
alloc_in_is_day_clear as is_day_clear,
goods_no,
0 as receive_qty,
0 as receive_amount,
0 as return_vendor_qty,
0 as return_vendor_amount,
0 as return_dc_qty,
0 as return_dc_amount,
alloc_qty as allocation_in_qty,
alloc_amount as allocation_in_amount,
0 as allocation_out_qty,
0 as allocation_out_amount
from dwd.dwd_order_store_alloc_in_i
union all
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
alloc_out_store_no as store_no,
alloc_out_store_name as store_name,
alloc_out_store_sale_type as store_sale_type,
alloc_out_store_type_code as store_type_code,
alloc_out_worker_num as worker_num,
alloc_out_store_area as store_area,
alloc_out_city_id as city_id,
alloc_out_city_name as city_name,
alloc_out_region_code as region_code,
alloc_out_region_name as region_name,
alloc_out_is_day_clear as is_day_clear,
goods_no,
0 as receive_qty,
0 as receive_amount,
0 as return_vendor_qty,
0 as return_vendor_amount,
0 as return_dc_qty,
0 as return_dc_amount,
0 as allocation_in_qty,
0 as allocation_in_amount,
alloc_qty as allocation_out_qty,
alloc_amount as allocation_out_amount
from dwd.dwd_order_store_alloc_out_i
),
t2 as (
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no,
sum(receive_qty) as receive_qty,
sum(receive_qty * receive_amount) as receive_amount,
sum(return_vendor_qty) as return_vendor_qty,
sum(return_vendor_qty*return_vendor_amount) as return_vendor_amount,
sum(return_dc_qty) as return_dc_qty,
sum(return_dc_qty * return_dc_amount) as return_dc_amount,
sum(allocation_in_qty) as allocation_in_qty,
sum(allocation_in_amount) as allocation_in_amount,
sum(allocation_out_qty) as allocation_out_qty,
sum(allocation_out_amount) as allocation_out_amount
from t1
group by
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no
)
insert overwrite table dwm.dwm_order_store_goods_receipt_quarter_i partition (dt)
select
t2.trade_date,
t2.week_trade_date,
t2.month_trade_date,
t2.hourly,
t2.quarter,
t2.quarters,
t2.store_no,
t2.store_name,
t2.store_sale_type,
t2.store_type_code,
t2.worker_num,
t2.store_area,
t2.city_id,
t2.city_name,
t2.region_code,
t2.region_name,
t2.is_day_clear,
t3.first_category_no,
t3.first_category_name,
t3.second_category_no,
t3.second_category_name,
t3.third_category_no,
t3.third_category_name,
t2.goods_no,
t3.goods_name,
t3.is_clear as is_clean,
t2.receive_qty,
t2.receive_amount,
t2.return_vendor_qty,
t2.return_vendor_amount,
t2.return_dc_qty,
t2.return_dc_amount,
t2.allocation_in_qty,
t2.allocation_in_amount,
t2.allocation_out_qty,
t2.allocation_out_amount,
t2.receive_qty + return_vendor_qty + return_dc_qty + allocation_in_qty + allocation_out_qty as receipt_qty,
t2.receive_amount + t2.return_vendor_amount + t2.return_dc_amount + t2.allocation_in_amount + t2.allocation_out_amount as receipt_amount,
(t2.receive_amount + t2.return_vendor_amount + t2.return_dc_amount + t2.allocation_in_amount + t2.allocation_out_amount) * 0.95 as receipt_cost,
t2.trade_date as dt
from t2 left join dim.dwd_dim_store_goods_i t3 on t3.dt = '2023-11-23' and t2.store_no = t3.store_no and t2.goods_no = t3.goods_no;1.2.4 门店商品要货刻表
基于dwd_order_store_require_i进行聚合操作。聚合之后再关联维表进行拉宽，可以提高效率。
建表语句:
CREATE TABLE IF NOT EXISTS dwm.dwm_order_store_goods_require_quarter_i(
trade_date STRING COMMENT '交易日期',
week_trade_date STRING COMMENT '周一日期',
month_trade_date STRING COMMENT '月一日期',
hourly BIGINT COMMENT '交易小时(0-23)',
quarter BIGINT COMMENT '刻钟:1.0-15,2.15-30,3.30-45,4.45-60',
quarters BIGINT COMMENT '刻钟数:hourly*4+quarters',
store_no STRING COMMENT '店铺编码',
store_name STRING COMMENT '店铺名称',
store_sale_type BIGINT COMMENT '店铺销售类型',
store_type_code BIGINT COMMENT '分店类型',
worker_num BIGINT COMMENT '员工人数',
store_area DECIMAL(27, 2) COMMENT '门店面积',
city_id BIGINT COMMENT '城市ID',
city_name STRING COMMENT '城市名称',
region_code STRING COMMENT '区域编码',
region_name STRING COMMENT '区域名称',
is_day_clear BIGINT COMMENT '是否日清:0否,1是',
-- 商品表
first_category_no STRING COMMENT '一级分类编码',
first_category_name STRING COMMENT '一级分类名称',
second_category_no STRING COMMENT '二级分类编码',
second_category_name STRING COMMENT '二级分类名称',
third_category_no STRING COMMENT '三级分类编码',
third_category_name STRING COMMENT '三级分类名称',
goods_no STRING COMMENT '商品编码',
-- 商品表
goods_name STRING COMMENT '商品名称',
-- 门店日清商品表
is_clean BIGINT COMMENT '商品是否日清',
require_qty DECIMAL(27, 3) COMMENT '要货数量',
require_amount DECIMAL(27, 2) COMMENT '要货金额'
)
COMMENT '门店商品要货刻表'
partitioned by(dt STRING COMMENT 'confirm_time确认时间')
row format delimited fields terminated by ','
stored as orc
tblproperties ('orc.compress'='SNAPPY');处理说明:
需要对dwd_order_store_require_i表的数据进行过滤： require_type_code=2 -- 选择要货类型为配送的 and is_canceled=0 -- 选择没有取消的 and collect_require_order_id is not null and collect_require_order_id <>'' -- 选择要货汇总单不为空的，即已经生效的要货单
数据插入:
-- 门店商品要货刻表
with t1 as (
select
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no,
sum(require_qty) as require_qty,
sum(require_price * require_qty ) as require_amount
from dwd.dwd_order_store_require_i
where require_type_code = 2 and is_canceled = 0 and collect_require_order_id is not null and collect_require_order_id != ''
group by
trade_date,
week_trade_date,
month_trade_date,
hourly,
quarter,
quarters,
store_no,
store_name,
store_sale_type,
store_type_code,
worker_num,
store_area,
city_id,
city_name,
region_code,
region_name,
is_day_clear,
goods_no
)
insert overwrite table dwm.dwm_order_store_goods_require_quarter_i partition (dt)
select
t1.trade_date,
t1.week_trade_date,
t1.month_trade_date,
t1.hourly,
t1.quarter,
t1.quarters,
t1.store_no,
t1.store_name,
t1.store_sale_type,
t1.store_type_code,
t1.worker_num,
t1.store_area,
t1.city_id,
t1.city_name,
t1.region_code,
t1.region_name,
t1.is_day_clear,
t2.first_category_no,
t2.first_category_name,
t2.second_category_no,
t2.second_category_name,
t2.third_category_no,
t2.third_category_name,
t1.goods_no,
t2.goods_name,
t2.is_clear as is_clean,
t1.require_qty,
t1.require_amount,
t1.trade_date as dt
from t1 left join dim.dwd_dim_store_goods_i t2
on t1.store_no = t2.store_no and t1.goods_no = t2.goods_no and t2.dt = '2023-11-23'