Sqoop版本分Sqoop1和Sqoop2,其中Sqoop1目前最高释出版本为1.4.6,Sqoop2最高释出版本为1.99.7,Sqoop1与Sqoop2相互间不兼容,而且Sqoop2目的并不是作为产品,主要是致力于开发。再者,其对Hadoop的支持版本有些特别要求,比如Hadoop1和Hadoop0.x还有Hadoop2.x的兼容性等。在下载时一般要注意其兼容的Hadoop版本(Sqoop官网上我没有看到相关具体的描述,只是通过下载的文件名辨别与Hadoop的兼容性)。
sqoop2 最新版本 sqoop 1.99.7
http://sqoop.apache.org/docs/1.99.7/index.html
http://mirror.bit.edu.cn/apache/sqoop/1.99.7/
下载文件名:
sqoop-1.99.7-bin-hadoop200.tar.gz
源码:sqoop-1.99.7.tar.gz
sqoop目录
bin:可执行脚本,一般使用sqoop都是通过这个目录中的工具调用,是一些shell或batch脚本。
conf:存放配置文件、目前仅有两个配置文件:sqoop_bootstrap.properties 和 sqoop.properties
docs:目前不清楚具体是什么,可能是帮助文档,不过一般使用sqoop不会用到。
server:里面只有一个lib目录,存了很多jar文件,是sqoop2 的server包。
shell:里面只有一个lib目录,存了很多jar文件,sqoop2的shell包。
tools:里面只有一个lib目录,存了很多jar文件,sqoop2的工具包。
配置sqoop根目录和第三方jar引用路径
[root@hadoop-allinone-200-123 hadoop-2.7.3]# vim /etc/profileexport SQOOP_HOME=/wdcloud/app/sqoop1.99.7export SQOOP_SERVER_EXTRA_LIB=/wdcloud/app/sqoop1.99.7/extra
[root@hadoop-allinone-200-123 sqoop-1.99.7]# mkdir extra
把mysql的驱动jar文件复制到这个目录下。
sqoop核心配置文件
sqoop_bootstrap.properties
配置config支持类,这里一般使用默认值即可:sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider
sqoop.properties
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/wdcloud/app/hadoop-2.7.3/etc/hadoop org.apache.sqoop.security.authentication.type=SIMPLE org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler org.apache.sqoop.security.authentication.anonymous=true
运行
验证配置是否有效
[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-tool verify
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Sqoop tool executor:
Version: 1.99.7
Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
20 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.
开启服务器
[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-server start
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Starting the Sqoop2 server...
0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
22 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
Sqoop2 server started.
#开启服务器后生成了两个目录(在那个目录下运行就在哪个目录下生成)
[root@hadoop-allinone-200-123 sqoop-1.99.7]# ll | grep @
drwxr-xr-x 3 root root 23 Dec 18 22:19 @BASEDIR@
drwxr-xr-x 2 root root 58 Dec 18 22:23 @LOGDIR@
#查看sqoop运行日志:
[root@hadoop-allinone-200-123 sqoop-1.99.7]# ll \@LOGDIR\@/
total 136
-rw-r--r-- 1 root root 165 Dec 18 22:22 audit.log
-rw-r--r-- 1 root root 670 Dec 18 22:21 derbyrepo.log
-rw-r--r-- 1 root root 78957 Dec 18 22:22 sqoop.log
关闭服务器
[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-server stop
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Stopping the Sqoop2 server...
Sqoop2 server stopped.
开启客户端
[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-shell
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>
退出命令行
sqoop:000> :x
[root@hadoop-allinone-200-123 sqoop-1.99.7]#
使用sqoop前请确保hadoop服务和Sqoop2服务器均已启动。其中Hadoop不仅要启动hdfs(NameNode、DataNode),还要启动yarn(NodeManager、ResourceManager),当然,一般还会有一个SecondaryNameNode,用于原始NameNode的备援进程。
在客户端连接server
sqoop:000> set server --host localhost 12000 --webapp sqoop
Usage: set server
sqoop:000> show version -all
client version:
Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
1 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
server version:
Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
API versions:
[v1]
当看到show version -all正确的显示 就说明了链接上了服务器