hive 之select 中文乱码
此处的中文乱码和mysql的库表 编码 latin utf 无关。
直接上案例。
有时候我们需要自定义一列，有时是汉字有时是字母，结果遇到这种情况了。
说实话看到这真是糟心。这谁受得了。
单独select 没有任何问题。
这是怎么回事呢？ 经过一番检查，发现有个地方类似与 "境内" as col但是没乱码，
此时怀疑就是if 函数起了作用，但是一时间不知道是为啥。。
经过多方面测试 concat("境内") concat_ws("","境内")没用，
concat_ws("",arrary("境内")) 有用，此时也不知道如何下手，只有掏出大杀器 explain.
起作用的
Plan optimized by CBO.
""
Vertex dependency in root stage
Map 1 <- Map 3 (BROADCAST_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
""
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 2
File Output Operator [FS_14]
Select Operator [SEL_13] (rows=105 width=273)
" Output:[""_col0"",""_col1"",""_col2"",""_col3"",""_col4"",""_col5"",""_col6"",""_col7""]"
Group By Operator [GBY_12] (rows=105 width=273)
" Output:[""_col0"",""_col1"",""_col2"",""_col3"",""_col4""],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4"
<-Map 1 [SIMPLE_EDGE] vectorized
SHUFFLE [RS_28]
" PartitionCols:_col0, _col1, _col2, _col3, _col4"
Group By Operator [GBY_27] (rows=211 width=273)
" Output:[""_col0"",""_col1"",""_col2"",""_col3"",""_col4""],keys:_col1, _col2, _col3, _col4, _col5"
Map Join Operator [MAPJOIN_26] (rows=211 width=273)
" Conds:SEL_25._col0=RS_23._col0(Inner),Output:[""_col1"",""_col2"",""_col3"",""_col4"",""_col5""]"
<-Map 3 [BROADCAST_EDGE] vectorized
BROADCAST [RS_23]
PartitionCols:_col0
Select Operator [SEL_22] (rows=1 width=736)
" Output:[""_col0"",""_col1"",""_col2"",""_col3""]"
Filter Operator [FIL_21] (rows=1 width=736)
predicate:bank_code is not null
TableScan [TS_3] (rows=1 width=736)
" dwapsdata@dw_conf_ce_bank_dict_v,t1,Tbl:COMPLETE,Col:NONE,Output:[""bank_code"",""bank_name"",""bank_short_name"",""bank_onshore_flag""]"
<-Select Operator [SEL_25] (rows=192 width=273)
" Output:[""_col0"",""_col1""]"
Filter Operator [FIL_24] (rows=192 width=273)
predicate:bank_code is not null
TableScan [TS_0] (rows=192 width=273)
" dwdmdata@dm_ce_f_portrait_credit_line,t,Tbl:COMPLETE,Col:COMPLETE,Output:[""bank_code""]"
""
没有作用的
Plan optimized by CBO.
""
Vertex dependency in root stage
Map 1 <- Map 3 (BROADCAST_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
""
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 2 vectorized
File Output Operator [FS_31]
Select Operator [SEL_30] (rows=105 width=273)
" Output:[""_col0"",""_col1"",""_col2"",""_col3"",""_col4"",""_col5"",""_col6""]"
Group By Operator [GBY_29] (rows=105 width=273)
" Output:[""_col0"",""_col1"",""_col2"",""_col3"",""_col4""],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4"
<-Map 1 [SIMPLE_EDGE] vectorized
SHUFFLE [RS_28]
" PartitionCols:_col0, _col1, _col2, _col3, _col4"
Group By Operator [GBY_27] (rows=211 width=273)
" Output:[""_col0"",""_col1"",""_col2"",""_col3"",""_col4""],keys:_col1, _col2, _col3, _col4, _col5"
Map Join Operator [MAPJOIN_26] (rows=211 width=273)
" Conds:SEL_25._col0=RS_23._col0(Inner),Output:[""_col1"",""_col2"",""_col3"",""_col4"",""_col5""]"
<-Map 3 [BROADCAST_EDGE] vectorized
BROADCAST [RS_23]
PartitionCols:_col0
Select Operator [SEL_22] (rows=1 width=736)
" Output:[""_col0"",""_col1"",""_col2"",""_col3""]"
Filter Operator [FIL_21] (rows=1 width=736)
predicate:bank_code is not null
TableScan [TS_3] (rows=1 width=736)
" dwapsdata@dw_conf_ce_bank_dict_v,t1,Tbl:COMPLETE,Col:NONE,Output:[""bank_code"",""bank_name"",""bank_short_name"",""bank_onshore_flag""]"
<-Select Operator [SEL_25] (rows=192 width=273)
" Output:[""_col0"",""_col1""]"
Filter Operator [FIL_24] (rows=192 width=273)
predicate:bank_code is not null
TableScan [TS_0] (rows=192 width=273)
" dwdmdata@dm_ce_f_portrait_credit_line,t,Tbl:COMPLETE,Col:COMPLETE,Output:[""bank_code""]"
""
对比发现
vectorzied 这个单词一出来我就知道怎么回事了。
hive decimal bug， nvl(decimal,1)=0_cclovezbf的博客-CSDN博客
这个b参数好处没体会到一点，bug到是一堆。
set hive.vectorized.execution.enabled=false; 即可解决中文乱码问题！！！！！！！
其实还有别的办法，但是和concat_ws(array(""))一样比较丑陋，我就不说了