前提条件:已安装好hadoop+hive+mysql(或其他数据库,这使用mysql)
一、安装
下载和解压
http://ftp.wayne.edu/apache/sqoop/
# tar -zxvf sqoop-1.4.7-bin-hadoop200.tar.gz -C /opt
# mv /opt/sqoop-1.4.7-bin-hadoop200 /opt/sqoop
修改配置文件
cp /opt/sqoop/conf/sqoop-env-template.sh /opt/sqoop/conf/sqoop-env.sh
vi /opt/sqoop/conf/sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop3
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop3
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive3
#Set the path for where zookeper config dir is
#export ZOOCFGDIR=
加入mysql的jdbc驱动包,因之前安装的hive下有这个包,直接从hive的Lib下拷贝到sqoop/lib下即可
cp mysql-connector-java-5.1.46-bin.jar /opt/sqoop/lib/
修改环境变量
vi /etc/profile
##add
export SQOOP_HOME=/opt/sqoop
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*
export PATH=$PATH:$JAVA_HOME/bin/:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MYSQL_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin/
source /etc/profile
启动sqoop
$ sqoop
二、遇到的问题及解决方法
---------------------------1
hadoop 3.x Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
场景:
解决:
$HADOOP_HOME/etc/mapred-site.xml,增加以下配置
---------------------------2
ERROR tool.ImportTool: Import failed: Import job failed!
解决:
根据虚拟机的内存设置
---------------------------3
ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
解决:
在配置文件/etc/profile中添加环境变量
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*