【若泽大数据实战第二十天】Sqoop-1.4.6-cdh5.7.0.tar.gz安装 + Sqoop help
Sqoop安装
1 下载
1. 下载地址
http://archive.cloudera.com/cdh5/cdh/5/
2. 环境sqoop-1.4.6-cdh5.7.0.tar.gz版本
2 上传到linux上，并解压
[hadoop@hadoop000 software]$ tar -xzvf sqoop-1.4.6-cdh5.7.0.tar.gz –C~/app
修改用户和用户组
3 Sqoop文件目录结构：
[hadoop@hadoop000 app]$cd sqoop-1.4.6-cdh5.7.0/
[[email protected]]$ ll
total 1880
drwxr-xr-x. 2 hadoophadoop 4096 Jun 25 10:10 bin（放脚本）
-rw-rw-r--. 1 hadoophadoop 60569 Mar 24 2016 build.xml
-rw-rw-r--. 1 hadoophadoop 1126 Mar 24 2016 cdh.build.properties
-rw-rw-r--. 1 hadoophadoop 35614 Mar 24 2016 CHANGELOG.txt
drwxr-xr-x. 4 hadoophadoop 4096 Jun 25 10:10 cloudera
-rw-rw-r--. 1 hadoophadoop 6901 Mar 24 2016 cloudera-pom.xml
-rw-rw-r--. 1 hadoophadoop 9880 Mar 24 2016 COMPILING.txt
drwxr-xr-x. 2 hadoophadoop 4096 Jun 25 10:10 conf（配置文件）
drwxr-xr-x. 5 hadoophadoop 4096 Jun 25 10:10 docs（文档）
drwxr-xr-x. 2 hadoophadoop 4096 Jun 25 10:10 ivy
-rw-rw-r--. 1 hadoophadoop 17386 Mar 24 2016 ivy.xml
drwxr-xr-x. 2 hadoophadoop 4096 Jun 25 10:10 lib（lib包依赖包）
-rw-rw-r--. 1 hadoophadoop 15319 Mar 24 2016 LICENSE.txt
-rw-rw-r--. 1 hadoophadoop 505 Mar 24 2016 NOTICE.txt
-rw-rw-r--. 1 hadoophadoop 18772 Mar 24 2016 pom-old.xml
-rw-rw-r--. 1 hadoophadoop 1096 Mar 24 2016 README.txt
-rw-rw-r--. 1 hadoophadoop 1035867 Mar 24 2016 sqoop-1.4.6-cdh5.7.0.jar（完整jar包）
-rw-rw-r--. 1 hadoophadoop 6554 Mar 24 2016 sqoop-patch-review.py
-rw-rw-r--. 1 hadoophadoop 656201 2016 sqoop-test-1.4.6-cdh5.7.0.jar（测试包）
drwxr-xr-x. 7 hadoophadoop 4096 Mar 24 2016 src（源码）
drwxr-xr-x. 4 hadoophadoop 4096 Jun 25 10:10 testdata
[hadoop@hadoop000 software]$ sudo chown hadoop:hadoop sqoop-1.4.6-cdh5.7.0
4 配置环境变量
[hadoop@hadoop000sqoop]$ sudo vi/etc/profile
添加自己的配置
exportSQOOP_HOME=/home/hadoop/app/sqoop-1.4.6-cdh5.7.0
export PATH=$SQOOP_HOME/bin:$PATH
5 复制sqoop/conf/sqoop-env-template.sh为sqoop-env.sh，添加配置
[hadoop@hadoop000 conf]$ cp sqoop-env-template.sh sqoop-env.sh
#Setpath to where bin/hadoop isavailable （Hadoop安装目录）
exportHADOOP_COMMON_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
#Setpath to where hadoop-*-core.jarisavailable（MapRedice目录）
exportHADOOP_MAPRED_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
#setthe path to where bin/hbaseisavailable
#exportHBASE_HOME=/home/hadoop/app/
#Setthe path to where bin/hive isavailable（Hive目录）
exportHIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.7.0
#Setthe path for where zookeper configdiris
#exportZOOCFGDIR= /opt/software/zookeeper
6 拷贝mysql的jdbc驱动包mysql-connector-java-5.1.27-bin.jar到sqoop/lib目录下
（这里我直接拷贝的hive下面已经下载好的驱动）
[hadoop000 lib]$
sudo
cp/home/hadoop/app/hive-1.1.0-cdh5.7.0/lib/mysql-connector-java-5.1.27-bin.jar .
7 Sqoophelp的使用
首先看看sqoop有没有报错：
[hadoop@hadoop000 bin]$ sqoop
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase importswill fail.
Please set $HBASE_HOME to the root of yourHBase installation.
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobswill fail.
Please set $HCAT_HOME to the root of yourHCatalog installation.
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumuloimports will fail.
Please set $ACCUMULO_HOME to the root of yourAccumulo installation.
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumuloimports will fail.
Please set $ZOOKEEPER_HOME to the root ofyour Zookeeper installation.
Try 'sqoop help' for usage.（这里提示让我用sqoophelp查看）
这里的警告是因为我没有配置hbase，zookeeper，HCatalog
[hadoop@hadoop000 bin]$ sqoop help
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase importswill fail.
Please set $HBASE_HOME to the root of yourHBase installation.
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobswill fail.
Please set $HCAT_HOME to the root of yourHCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulodoes not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of yourAccumulo installation.
Warning:/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumuloimports will fail.
Please set $ZOOKEEPER_HOME to the root ofyour Zookeeper installation.
18/06/25 11:09:55 INFO sqoop.Sqoop: RunningSqoop version: 1.4.6-cdh5.7.0
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate codeto interact with database records
create-hive-table Import a tabledefinition into Hive
eval Evaluate a SQLstatement and display the results
export Export an HDFSdirectory to a database table
help List availablecommands
import Import a tablefrom a database to HDFS
import-all-tables Import tablesfrom a database to HDFS
import-mainframe Import datasetsfrom a mainframe server to HDFS
job Work with savedjobs
list-databases List availabledatabases on a server
list-tables List availabletables in a database
merge Merge resultsof incremental imports
metastore Run astandalone Sqoop metastore
version Display versioninformation
这里version可以查看版本
[hadoop@hadoop000 conf]$ sqoop version
Warning:/opt/software/sqoop/../hbase doesnot exist! HBase imports will fail.
Please set $HBASE_HOME tothe root of your HBaseinstallation.
Warning:/opt/software/sqoop/../hcatalog doesnot exist! HCatalog jobs will fail.
Please set $HCAT_HOME tothe root of your HCatalog Ibnrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrenstallation.
Warning:/opt/software/sqoop/../accumulo doesnot exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME tothe root of your Accumuloinstallation.
Warning:/opt/software/sqoop/../zookeeper doesnot exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME tothe root of your Zookeeperinstallation.
18/01/1412:47:03 INFO sqoop.Sqoop:Running Sqoop version: 1.4.6-cdh5.7.0
Sqoop 1.4.6-cdh5.7.0
git commit id
Compiled by jenkins on Wed Mar 2311:30:51 PDT 2016
登录 MySQL 使用ruozedata_basic03库
mysql> use ruozedata_basic03
Database changed
mysql>
7.1 通过Sqoop查看MySQL中库的情况
[hadoop@hadoop000 bin]$
sqoop list-databases \
--connect jdbc:mysql://localhost:3306 \
--username root --password root
18/06/25 11:56:07 INFOmanager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
mysql
performance_schema
ruozedata_basic03
test
7.2 通过Sqoop查看MySQL中ruozedata_basic03库的表情况
[hadoop@hadoop000 bin]$
sqoop list-tables \
--connect jdbc:mysql://localhost:3306/ruozedata_basic03 \
--username root --password root
18/06/25 11:52:57 INFO manager.MySQLManager:Preparing to use a MySQL streaming resultset.
bucketing_cols
cds
columns_v2
database_params
db_privs
dbs
func_ru
funcs
global_privs
idxs
index_params
part_col_privs
part_col_stats
part_privs
partition_key_vals
partition_keys
partition_params
partitions
roles
sd_params
sds
sequence_table
serde_params
serdes
skewed_col_names
skewed_col_value_loc_map
skewed_string_list
skewed_string_list_values
skewed_values
sort_cols
tab_col_stats
table_params
tbl_col_privs
tbl_privs
tbls
version
大数据课课程推荐：
