sqoop同步mysql到hdfs

链接：http://pan.baidu.com/s/1gfHnaVL 密码：7j12

mysql-connector version 5.1.32

若在安装版本过程遇到些问题，可参考http://dbspace.blog.51cto.com/6873717/1875955，其中一些问题的解决办法

下载并安装：

cd /usr/local/
tar -zxvf sqoop2-1.99.3-cdh5.0.0.tar.gz
mv sqoop2-1.99.3-cdh5.0.0 sqoop
添加sqoop2到系统环境变量中：
export SQOOP_HOME=/usr/local/sqoop
export CATALINA_BASE=$SQOOP_HOME/server
export PATH=$PATH:/usr/local/sqoop/bin
拷贝mysql驱动包到$SQOOP2_HOME/server/lib下
cp mysql-connector-java-5.1.32-bin.jar /usr/local/sqloop/server/lib/ 
修改配置文件：
vim /usr/local/sqoop/server/conf/sqoop.properties
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop#hadoop的配置文件路径
vim /usr/local/sqoop/server/conf/catalina.properties
把原来58行注释了，这里主要配置了hadoop的jar包的路径信息
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${ca    talina.home}/../lib/*.jar,/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.    jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/shar    e/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*    .jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/sh    are/hadoop/yarn/lib/*.jar
启动\停止sqoop
/usr/local/sqoop/sqoop2-server start/stop
验证是否启动成功：
方式一：jps查看进程: Bootstrap 
[root@mycat sqoop]# jps

25505 SqoopShell
13080 SecondaryNameNode
12878 NameNode
26568 Jps
方式二：方式二：http://192.168.1.114:12000/sqoop/version #SQOOP默认使用的端口为12000在/usr/local/sqoop/server/conf/server.xml中进行设置

####接下来测试mysql到hadoop存储的一个过程
1、客户端登陆
[root@mycat bin]# sqoop2-shell 
Sqoop home directory: /usr/local/sqoop
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> 
2、创建一个mysql链接，在这个版本create 就只有[connection|job],注意不同版本的添加链接方式是不同的.
 查看支持的链接服务
 sqoop:000> show connector
+----+------------------------+-----------------+------------------------------------------------------+
| Id |          Name          |     Version     |                        Class                         |
+----+------------------------+-----------------+------------------------------------------------------+
| 1  | generic-jdbc-connector | 1.99.3-cdh5.0.0 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector |
+----+------------------------+-----------------+------------------------------------------------------+##在1.99.7的版本显示的方式和服务更多。
sqoop:000> create connection --cid 1
Creating connection for connector with id 1
Please fill following values to create new connection object
Name: mysql_to_hadoop

Connection configuration

JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://192.168.1.107:3306/sqoop #这里需要在1.107先添加好库sqoop
Username: sqoop##需要在数据库添加好链接的用户
Password: *******
JDBC Connection Properties: 
There are currently 0 values in the map:
entry# 
Security related configuration options
Max connections:
New connection was successfully created with validation status ACCEPTABLE and persistent id 2
2、创建job
sqoop:000> create job --xid 2 --type import##注意 --xid 2为链接的id号
Creating job for connection with id 2
Please fill following values to create new job object
Name: mysql_to_hadoop

Database configuration

Schema name: sqoop#MySQL的库名
Table name: wangyuan#库下的表
Table SQL statement: 
Table column names: 
Partition column name: 
Nulls in partition column: 
Boundary query: 

Output configuration

Storage type: 
  0 : HDFS
Choose: 0
Output format: 
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
Choose: 0
Compression format: 
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
Choose: 0
Output directory: hdfs://192.168.1.114:9000/home/mysql_to_hdfs2
#注意这个mysql_to_hdfs不能再hadoop的/home/已经存在的，但/home路径要存在,9000端口是在配置hadoop的时候配置，根据实际，或者通过WEB查看http:ip:50070----显示Overview 'mycat:9000' (active)
创建hdfs路径/usr/local/hadoop/bin/hadoop fs -mkidr /home
查看创建目录：/usr/local/hadoop/bin/hadoop fs -ls /home 或者通过WEB查看http:ip:50070
Throttling resources
Extractors: 
Loaders: 
New job was successfully created with validation status FINE  and persistent id 2
sqoop:000> 
启动job
sqoop:000> start job --jid 2
Exception has occurred during processing command 
Exception: org.apache.sqoop.common.SqoopException Message: CLIENT_0001:Server has returned exception
根本不知道这个提示说什么，通过修改设置：
set option --name verbose --value true
sqoop:000> start job --jid 2        
Submission details
Job ID: 2
Server URL: http://localhost:12000/sqoop/
Created by: root
Creation date: 2016-11-23 21:15:27 CST
Lastly updated by: root
External ID: job_1479653943050_0007
	http://haproxy:8088/proxy/application_1479653943050_0007/
Connector schema: Schema{name=wangyuan,columns=[
	FixedPoint{name=id,nullable=null,byteSize=null,unsigned=null},
	Date{name=c_time,nullable=null,fraction=null,timezone=null}]}
2016-11-23 21:15:27 CST: BOOTING  - Progress is not available
返回这样信息OK
查看结果通过WEB

/usr/local/hadoop/bin/hadoop fs -ls /home/