【sqoop】Linux环境下sqoop-1.4.7安装

sqoop安装

前提条件：已安装好hadoop+hive+mysql(或其他数据库，这使用mysql)

一、安装

下载和解压
http://ftp.wayne.edu/apache/sqoop/
# tar -zxvf sqoop-1.4.7-bin-hadoop200.tar.gz -C /opt
# mv /opt/sqoop-1.4.7-bin-hadoop200 /opt/sqoop

修改配置文件
cp /opt/sqoop/conf/sqoop-env-template.sh /opt/sqoop/conf/sqoop-env.sh
vi /opt/sqoop/conf/sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop3

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop3

#set the path to where bin/hbase is available
#export HBASE_HOME=

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive3

#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

加入mysql的jdbc驱动包，因之前安装的hive下有这个包，直接从hive的Lib下拷贝到sqoop/lib下即可
cp mysql-connector-java-5.1.46-bin.jar /opt/sqoop/lib/

修改环境变量
vi /etc/profile
##add
export SQOOP_HOME=/opt/sqoop
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
export PATH=$PATH:$JAVA_HOME/bin/:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MYSQL_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin/

source /etc/profile

启动sqoop
$ sqoop

 

二、遇到的问题及解决方法
---------------------------1
hadoop 3.x Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

场景：

解决：
$HADOOP_HOME/etc/mapred-site.xml,增加以下配置

 
   yarn.app.mapreduce.am.env
   HADOOP_MAPRED_HOME=${HADOOP_HOME}
 
 
   mapreduce.map.env
   HADOOP_MAPRED_HOME=${HADOOP_HOME}
 
 
   mapreduce.reduce.env
   HADOOP_MAPRED_HOME=${HADOOP_HOME}
 
 

---------------------------2
ERROR tool.ImportTool: Import failed: Import job failed!

解决：
mapreduce.map.memory.mb
100


mapreduce.map.java.opts
-Xmx300M


mapreduce.reduce.memory.mb
100


mapreduce.reduce.java.opts
-Xmx300M

根据虚拟机的内存设置

---------------------------3
ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

解决：
在配置文件/etc/profile中添加环境变量
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*