sqoop1.99.3安装与教程
测试环境hadoop版本2.4.0，sqoop版本1.99.3，其他高版本
1.99.4、1.99.5均安装失败。
安装文档 http://sqoop.apache.org/docs/1.99.3/Installation.html
1.下载sqoop-1.99.3-bin-hadoop200.tar.gz
2.解压
tar -xvf sqoop--bin-hadoop.tar.gz
3.配置文件/home/hadoop/sqoop/server/conf
catalina.properties
修改：（依赖jar）
common.loader=common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar
sqoop.properties（设置hadoop配置文件路径）
修改
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
4.添加数据库驱动包
mysql-connector-java-5.1.26-bin.jar --mysql
ojdbc6.jar --oracle
4.启动 /bin/sqoop.sh server start
停止 ./bin/sqoop.sh server stop
5.添加服务端
set server --host 132.77.210.201 --port 12000 --webapp sqoop
6.验证：
-> ./sqoop.sh client
截图
7. 常用命令：
注意 --jid 表示 job ID --xid 表示 连接ID
create connection --cid 1 --创建连接
delete connection --xid 1
update connection --xid 1
create job --xid 1 --type import 创建job 从数据库导入hdfs --xid 1 表示连接id
create job --xid 1 --type export 创建job 从hdfs导入数据库
status job --jid 2 --查看job状态
start job --jid 2 --执行job
update job -jid 2 --修改job
show job --显示job
delete job --jid 4 --删除job --jid 4 要删除的job id
8.链接oracle:
sqoop:000>
create connection --cid 1
Creating connection for connector with id 1
Please fill following values to create new connection object
Name:
wgh_test
Connection configuration
JDBC Driver Class:
oracle.jdbc.driver.OracleDriver
JDBC Connection String:
jdbc:oracle:thin:@132.77.119.185:1528:wghdbts1
Username:
wgh
Password:
***
JDBC Connection Properties:
There are currently 0 values in the map:
entry#
Security related configuration options
Max connections:
10
New connection was successfully created with validation status FINE and persistent id 2
9.修改jop/新建
sqoop:000> update job --jid 1
--修改
-- create job --xid 1 --type import --创建 从数据库导入hdfs
Updating job with id 1
Please update job metadata:
Name:
first_job
Database configuration
Schema name:
wgh --数据库用户名
Table name:
eval_template
Table SQL statement:
Table column names:
EVAL_TEMPLATE_ID,TEMPLATE_NAME --字段
Partition column name:
Nulls in partition column:
Boundary query:
Output configuration
Storage type:
0 : HDFS
Choose:
0
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Choose:
0
Compression format:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
Choose:
0
Output directory:
/test --hdfs文件目录
Throttling resources
Extractors:
Loaders:
Job was successfully updated with status FINE
--创建导入job,从hdfs导入oracle
sqoop:000> create job --xid 1 --type export
Creating job for connection with id 1
Please fill following values to create new job object
Name:
export_eval
Database configuration
Schema name:
wgh
Table name:
eval_template_001
Table SQL statement:
Table column names:
EVAL_TEMPLATE_ID,TEMPLATE_NAME
Stage table name:
Clear stage table:
Input configuration
Input directory:
/test2
Throttling resources
Extractors:
Loaders:
New job was successfully created with validation status FINE and persistent id 4
sqoop:000> show job
+----+--------------------------+--------+-----------+---------+
| Id | Name | Type | Connector | Enabled |
+----+--------------------------+--------+-----------+---------+
| 1 | first_job | IMPORT | 1 | true |
| 2 | export_vip_customer_info | EXPORT | 1 | true |
| 4 | export_eval | EXPORT | 1 | true |
+----+--------------------------+--------+-----------+---------+
sqoop:000>
日志目录：/home/hadoop/sqoop/@LOGDIR@
server启动日志目录：/home/hadoop/sqoop/server/logs