Sqoop的配置和使用

一、 解压sqoop.tar压缩包至指定目录

二、配置

1. 开启Zookeeper
2. 开启集群服务
3. 配置文件( sqoop-env.sh )：

   #Set path to where bin/hadoop is available
   export HADOOP_COMMON_HOME=/usr/local/hadoop-2.7.3
   
   #Set path to where hadoop-*-core.jar is available
   export HADOOP_MAPRED_HOME=/usr/local/hadoop-2.7.3
   
   #set the path to where bin/hbase is available
   export HBASE_HOME=/usr/local/hbase-1.3.2
   
   #Set the path to where bin/hive is available
   export HIVE_HOME=/usr/local/hive
   
   #Set the path for where zookeper config dir is
   export ZOOCFGDIR=/usr/local/zookeeper-3.4.6
   export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.6
   

4. 拷贝jdbc驱动到sqoop的lib目录下

   cp -a mysql-connector-java-5.1.47-bin.jar /usr/local/sqoop-1.4.7/lib/
   

5. 配置环境变量

   // 修改环境变量
   vim /etc/profile
   
   export SQOOP_HOME=/usr/local/sqoop-1.4.7
   export PATH=$PATH:$SQOOP_HOME/bin
   
   // 配置环境变量生效
   source /etc/profile
   

6. 测试sqoop是否能够连接成功

   sqoop list-databases --connect jdbc:mysql://hadoop01:3306/metastore_db \ 
   --username root --password root
   

三、案例

1. RDBMS -> HDFS

1. 全部导入

   sqoop import
   --connect jdbc:mysql://hadoop01:3306/test
   --username root
   --password root
   --table test01
   --target-dir /tmp/test01
   --delete-target-dir
   --num-mappers 1
   --fields-terminated-by "\t"
   

2. 查询导入

   sqoop import 
   --connect jdbc:mysql://hadoop01:3306/test
   --username root 
   --password root 
   --target-dir /tmp/test01
   --delete-target-dir 
   --num-mappers 1 
   --fields-terminated-by "\t" 
   --query 'select name,sex from test01 where id >= 2 and $CONDITIONS;'
   

3. 导入指定列

   sqoop import 
   --connect jdbc:mysql://hadoop01:3306/test
   --username root 
   --password root
   --target-dir /tmp/test01
   --delete-target-dir 
   --num-mappers 1 
   --fields-terminated-by "\t"
   --columns id, sex
   --table test01
   

4. 使用sqoop关键字筛选查询导入数据

   sqoop import 
   --connect jdbc:mysql://hadoop01:3306/test
   --username root 
   --password root
   --target-dir /tmp/test01
   --delete-target-dir 
   --num-mappers 1 
   --fields-terminated-by "\t"
   --table test01
   --where "id=3"
   

2. RDBMS --> Hive

sqoop import 
--connect jdbc:mysql://hadoop01:3306/test 
--username root 
--password root 
--table test01 
--num-mappers 1 
--hive-import 
--fields-terminated-by "\t" 
--hive-overwrite 
--hive-table test.test02

3. Hive/HDFS --> RDBMS

sqoop export 
--connect jdbc:mysql://hadoop01:3306/test 
--username root 
--password root
--table test01
--num-mappers 1 
--export-dir /user/hive/warehouse/test.db/test02
--input-fields-terminated-by "\t" 

4. RDBMS --> HBASE

sqoop import 
--connect jdbc:mysql://hadoop01:3306/zlh 
--username root 
--password root 
--table t_shop
--columns "id,name,price" 
--column-family "infos" 
--hbase-create-table 
--hbase-row-key "id" 
--hbase-table "h_shop" 
--num-mappers 1 
--split-by id