大数据环境搭建3：Apache Sqoop环境搭建

1. 下载sqoop安装包

下载地址http://mirror.bit.edu.cn/apache/sqoop/1.4.7/

2. 上传sqoop安装包，并安装

解压安装

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C ../app

3. 配置环境变量

(1) vi /etc/profile

export SQOOP_HOME=/home/wangping/app/sqoop-1.4.7

export PATH=$SQOOP_HOME/bin:$PATH

(2) 使环境变量生效

source /etc/profile

4. 修改sqoop配置文件

cp sqoop-env-template.sh sqoop-env.sh

vi sqoop-env.sh

export HADOOP_COMMON_HOME=/home/wangping/app/hadoop-2.7.7

export HADOOP_MAPRED_HOME=/home/wangping/app/hadoop-2.7.7

export HIVE_HOME=/home/wangping/hive-2.3.5

5. 验证安装

sqoop version

6. 导入mysql-connector

cp mysql-connector-java-8.0.16.jar $SQOOP_HOME/lib

7. sqoop简单使用

(1) 使用帮助

sqoop help

(2) 使用sqoop查看mysql数据库test下的表

sqoop list-tables --connect jdbc:mysql://192.168.108.10:3306/test?characterEncoding=UTF-8 --username root --password '123456'

(3) 在hive的hiveTest中创建一张表

 create table product1(id int, product_name string, product_category string);

(4) 从mysql中导入数据到hive库中

sqoop import --connect jdbc:mysql://192.168.108.10:3306/test --username root --password 123456 --table product --hive-import --hive-table hive_test.product --hive-overwrite -m 1

-m 1指定maptask的数量为1

8. 拍摄快照

拍摄快照，防止日后快速回复环境

9. 错误锦集

（1）main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")

cp hive-site.xml $SQOOP_HOME/conf

（2）ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider

不解决（hadoop的bug不影响操作

（3）

Warning: /home/zkkafka/sqoop/../hcatalog does not exist! HCatalog jobs will fail.  

Please set $HCAT_HOME to the root of your HCatalog installation.  

Warning: /home/zkkafka/sqoop/../accumulo does not exist! Accumulo imports will fail.  

Please set $ACCUMULO_HOME to the root of your Accumulo installation.  

19/05/29 14:52:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7  

19/05/29 14:52:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.  

19/05/29 14:52:46 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException  

java.lang.NullPointerException  

    at org.json.JSONObject.(JSONObject.java:144)  

    at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)  

    at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:785)  

    at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)  

    at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)  

    at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)  

    at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)  

    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)  

    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)  

    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)  

    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)  

    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)  

    at org.apache.sqoop.Sqoop.main(Sqoop.java:252) 

10. 参考文章

https://www.jianshu.com/p/a088713ba26b

https://blog.csdn.net/weixin_34120274/article/details/88509983