Sqoop2安装

Sqoop版本分Sqoop1和Sqoop2,其中Sqoop1目前最高释出版本为1.4.6,Sqoop2最高释出版本为1.99.7,Sqoop1与Sqoop2相互间不兼容,而且Sqoop2目的并不是作为产品,主要是致力于开发。再者,其对Hadoop的支持版本有些特别要求,比如Hadoop1和Hadoop0.x还有Hadoop2.x的兼容性等。在下载时一般要注意其兼容的Hadoop版本(Sqoop官网上我没有看到相关具体的描述,只是通过下载的文件名辨别与Hadoop的兼容性)。

sqoop2 最新版本 sqoop 1.99.7
http://sqoop.apache.org/docs/1.99.7/index.html
http://mirror.bit.edu.cn/apache/sqoop/1.99.7/
下载文件名:
sqoop-1.99.7-bin-hadoop200.tar.gz
源码:sqoop-1.99.7.tar.gz

sqoop目录

bin:可执行脚本,一般使用sqoop都是通过这个目录中的工具调用,是一些shell或batch脚本。
conf:存放配置文件、目前仅有两个配置文件:sqoop_bootstrap.properties 和 sqoop.properties
docs:目前不清楚具体是什么,可能是帮助文档,不过一般使用sqoop不会用到。
server:里面只有一个lib目录,存了很多jar文件,是sqoop2 的server包。
shell:里面只有一个lib目录,存了很多jar文件,sqoop2的shell包。
tools:里面只有一个lib目录,存了很多jar文件,sqoop2的工具包。

配置sqoop根目录和第三方jar引用路径

[root@hadoop-allinone-200-123 hadoop-2.7.3]# vim /etc/profileexport SQOOP_HOME=/wdcloud/app/sqoop1.99.7export SQOOP_SERVER_EXTRA_LIB=/wdcloud/app/sqoop1.99.7/extra
  [root@hadoop-allinone-200-123 sqoop-1.99.7]# mkdir extra

把mysql的驱动jar文件复制到这个目录下。

sqoop核心配置文件
sqoop_bootstrap.properties

配置config支持类,这里一般使用默认值即可:sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider

sqoop.properties

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/wdcloud/app/hadoop-2.7.3/etc/hadoop org.apache.sqoop.security.authentication.type=SIMPLE org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler org.apache.sqoop.security.authentication.anonymous=true 

运行
验证配置是否有效

[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-tool verify  
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Sqoop tool executor:
    Version: 1.99.7
    Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
    Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
20   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

开启服务器

[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-server start  
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Starting the Sqoop2 server...
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
22   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Sqoop2 server started.
#开启服务器后生成了两个目录(在那个目录下运行就在哪个目录下生成)
[root@hadoop-allinone-200-123 sqoop-1.99.7]# ll | grep @
drwxr-xr-x 3 root root 23 Dec 18 22:19 @BASEDIR@
drwxr-xr-x 2 root root 58 Dec 18 22:23 @LOGDIR@


#查看sqoop运行日志:
[root@hadoop-allinone-200-123 sqoop-1.99.7]# ll \@LOGDIR\@/
total 136
-rw-r--r-- 1 root root   165 Dec 18 22:22 audit.log
-rw-r--r-- 1 root root   670 Dec 18 22:21 derbyrepo.log
-rw-r--r-- 1 root root 78957 Dec 18 22:22 sqoop.log

关闭服务器

[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-server stop
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Stopping the Sqoop2 server...
Sqoop2 server stopped.

开启客户端

[root@hadoop-allinone-200-123 sqoop-1.99.7]# bin/sqoop2-shell  
Setting conf dir: /wdcloud/app/sqoop-1.99.7/bin/../conf
Sqoop home directory: /wdcloud/app/sqoop-1.99.7
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> 

退出命令行

sqoop:000> :x
[root@hadoop-allinone-200-123 sqoop-1.99.7]# 

使用sqoop前请确保hadoop服务和Sqoop2服务器均已启动。其中Hadoop不仅要启动hdfs(NameNode、DataNode),还要启动yarn(NodeManager、ResourceManager),当然,一般还会有一个SecondaryNameNode,用于原始NameNode的备援进程。

在客户端连接server

sqoop:000> set server --host localhost 12000 --webapp sqoop
Usage: set server
sqoop:000> show version -all
client version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
1    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
server version:
  Sqoop 1.99.7 source revision 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb 
  Compiled by abefine on Tue Jul 19 16:08:27 PDT 2016
API versions:
  [v1]

当看到show version -all正确的显示 就说明了链接上了服务器

你可能感兴趣的:(Sqoop2安装)