Sqoop抽取Sqlserver数据到hbase
Sqoop抽取Sqlserver数据到hbase
语法：
sqoop import --connect "jdbc:sqlserver://10.96.183.70:1433;database=SPC" --username mes --password mes+123 \
--query \
"select \
random+fdatec+boardsn+idstation+imulti as rowkey, \
fdate, \
idstation, \
imulti, \
cmodel, \
status, \
case when LEN(boardsn)=10 or LEN(boardsn)=11 then \
case when PATINDEX('%[^0-9A-Z]%',boardsn)>0 then 'ErrorSN' \
else boardsn end \
else 'ErrorSN' end as boardsn, \
trans, \
tb, \
path, \
opentime, \
lifecycle, \
case when len(line)=2 then left(line,1)+'0'+right(line,1) else line end as line, \
abside, \
realmodel, \
onlinecheck, \
modifier, \
modifydate, \
yfdate, \
tag, \
steelid, \
vendor, \
createdate, \
rearsq_speed, \
frontsq_pressure, \
rearsq_pressure, \
frontsq_standardspeed, \
frontsq_lowspeed, \
frontsq_highspeed, \
frontsq_standardpressure, \
frontsq_lowpressure, \
frontsq_highpressure, \
rearsq_standardpressure, \
rearsq_lowpressure, \
rearsq_highpressure, \
tableseq_speed, \
mcno, \
machineuserid, \
lotno, \
failsolder, \
sq, \
version, \
abprinter, \
serial, \
lane, \
shift \
from \
( \
select \
RIGHT(convert(varchar(12) , fdate, 108 ),1) random, \
replace(convert(varchar(100),fdate,112)+convert(varchar(100),fdate,8),':', '') as fdatec, \
fdate, \
idstation, \
imulti, \
cmodel, \
status, \
boardsn, \
trans, \
tb, \
path, \
opentime, \
lifecycle, \
case when line='NONE' then 'S'+substring(idstation,1,len(idstation)-1) else 'S'+substring(line,4,len(line)-3) end as line, \
abside, \
realmodel, \
onlinecheck, \
modifier, \
modifydate, \
yfdate, \
tag, \
steelid, \
vendor, \
createdate, \
rearsq_speed, \
frontsq_pressure, \
rearsq_pressure, \
frontsq_standardspeed, \
frontsq_lowspeed, \
frontsq_highspeed, \
frontsq_standardpressure, \
frontsq_lowpressure, \
frontsq_highpressure, \
rearsq_standardpressure, \
rearsq_lowpressure, \
rearsq_highpressure, \
tableseq_speed, \
mcno, \
machineuserid, \
lotno, \
failsolder, \
sq, \
version, \
abprinter, \
serial, \
lane, \
case when convert(varchar,fdate,108)>'20:30:00' \
then convert(varchar(12) ,fdate, 111 )+'N' \
when convert(varchar,fdate,108)>'08:00:00' \
then convert(varchar(12) , fdate, 111 )+'D' \
else convert(varchar(12) , dateadd(dd ,-1, fdate), 111 )+'N' end as shift \
from boardsfc ) as source where \$CONDITIONS" \ 注意在双引号中需要“\”对$CONDITIONS进行转义
--hbase-table "SPI_RESOURCE:SPI_BOARDSFC_KETTLE" --column-family 'Info' \
--hbase-row-key "rowkey" \
--target-dir '/hbase/data/SPI_RESOURCE/SPI_BOARDSFC_KETTLE' -m 3 --split-by imulti
注意事项：抽取过程有报错，具体什么异常忘记截屏了，修改如下两参数解决异常问题。
set mapred.job.map.memory.mb=4096
set mapred.job.reduce.memory.mb=4096