sqoop2（sqoop-1.99.7）的安装部署

Sqoop 是一个开源工具，主要用于在Hadoop和传统的数据库(Mysql, Oracle,等）进行数据传递，可以将一个关系型数据库中的数据导入Hadoop 的HDFS中，也可以将HDFS中的数据导出到关系型数据库中。之前总结了sqoop1的部署安装，接下来总结一下sqoop2的部署和填坑

一、下载sqoop包

   https://downloads.apache.org/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz

 

二、解压sqoop包并重命名

   tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz

   mv sqoop-1.99.7-bin-hadoop200 sqoop2

三、配置环境变量：vim /etc/profile

  sqoop需要hadoop的环境变量配置：$HADOOP_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_MAPRED_HOME,HADOOP_YARN_HOME

export HADOOP_HOME=/usr/local/hadoop-3.1.2
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME/share/hadoop/common
export HADOOP_HDFS_HOME=$HADOOP_HOME/share/hadoop/hdfs
export HADOOP_MAPRED_HOME=$HADOOP_HOME/share/hadoop/mapreduce
export HADOOP_YARN_HOME=$HADOOP_HOME/share/hadoop/yarn
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HIVE_HOME=/usr/local/hive-3.1.2
export PIG_HOME=/usr/local/pig-0.17.0
export PIG_CLASSPATH=$HADOOP_HOME/conf
export SQOOP_HOME=/usr/local/sqoop2
export SQOOP_SERVER_EXTRA_LIB=/usr/local/sqoop2/extlib/
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$PIG_HOME/bin:$SQOOP_HOME/bin:

  备注：也可以直接把变量定义在sqoop.sh文件中

HADOOP_HOME=/usr/local/hadoop-3.1.2/
HADOOP_COMMON_HOME=$HADOOP_HOME/share/hadoop/common
HADOOP_HDFS_HOME=$HADOOP_HOME/share/hadoop/hdfs
HADOOP_MAPRED_HOME=$HADOOP_HOME/share/hadoop/mapreduce
HADOOP_YARN_HOME=$HADOOP_HOME/share/hadoop/yarn

四、hadoop的core-site.xml和container-executor.cfg配置

  hadoop.proxyuser.sqoop2.hosts
  *


  hadoop.proxyuser.sqoop2.groups
  *




allowed.system.users=sqoop2  #如果有多用 sqoop2,hadoop,root

五、修改sqoop配置文件 sqoop目录下conf/sqoop.properties

1、@LOGDIR@ 修改为 /usr/local/sqoop2/logs，并在sqoop2目录下创建logs文件夹

2、@BASEDIR@修改为/usr/local/sqoop2 

六、添加sqoop2外部引入的jar包

新建目录mkdir /usr/local/sqoop2/extlib/ 把mysql驱动包和derby.jar包存放于该目录下；

开始挖坑了，按照官方文件 添加环境变量 export SQOOP_SERVER_EXTRA_LIB=/usr/local/sqoop2/extlib/

然后source /etc/profile 让文件失效；以上官方部署完成了，然后sqoop2-tool upgrade，直接异常。

   第一没有配置org.apache.sqoop.submission.engine.mapreduce.configuration.directory配置文件目录

       修改配置org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop-3.1.2/etc/hadoop/

  第二次异常找不derby.jar到驱动包

2020-04-13 10:53:44,825 ERROR [org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:55)] Can't finish upgrading RepositoryManager, Driver and ConnectionManager:
org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link
	at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:63)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:86)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:62)
	at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:128)
	at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:128)
	at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
	at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:derby:/usr/local/sqoop2/repository/db;create=true
	at java.sql.DriverManager.getConnection(DriverManager.java:689)
	at java.sql.DriverManager.getConnection(DriverManager.java:208)
	at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:78)
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
	at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:61)

修改sqoop配置文件，强行指定derby.jar包

修改sqoop.properties属性 指定加载jar目录 org.apache.sqoop.connector.external.loadpath=/usr/local/sqoop2/extlib/

具体配置如下

org.apache.sqoop.log4j.debug=false
org.apache.sqoop.log4j.rootLogger=INFO, file
org.apache.sqoop.log4j.category.org.apache.sqoop=INFO
org.apache.sqoop.log4j.appender.file=org.apache.log4j.RollingFileAppender
org.apache.sqoop.log4j.appender.file.File=/usr/local/sqoop2/logs/sqoop.log
org.apache.sqoop.log4j.appender.file.MaxFileSize=25MB
org.apache.sqoop.log4j.appender.file.MaxBackupIndex=5
org.apache.sqoop.log4j.appender.file.layout=org.apache.log4j.PatternLayout
org.apache.sqoop.log4j.appender.file.layout.ConversionPattern=%d{ISO8601} %-5p [%l] %m%n
# Audit logger for default configuration of FileAuditLogger
org.apache.sqoop.log4j.logger.audit=INFO, audit
org.apache.sqoop.log4j.appender.audit=org.apache.log4j.RollingFileAppender
org.apache.sqoop.log4j.appender.audit.File=/usr/local/sqoop2/logs/audit.log
org.apache.sqoop.log4j.appender.audit.MaxFileSize=25MB
org.apache.sqoop.log4j.appender.audit.MaxBackupIndex=5
org.apache.sqoop.log4j.appender.audit.layout=org.apache.log4j.PatternLayout
org.apache.sqoop.log4j.appender.audit.layout.ConversionPattern=%d{ISO8601} %-5p [%l] %m%n

#
# Audit Loggers Configuration
# Multiple audit loggers could be given here. To specify an
# audit logger, you should at least add org.apache.sqoop.
# auditlogger.[LoggerName].class. You could also provide
# more configuration options by using org.apache.sqoop.
# auditlogger.[LoggerName] prefix, then all these options
# are parsed to the logger class.
#
org.apache.sqoop.auditlogger.default.class=org.apache.sqoop.audit.FileAuditLogger
org.apache.sqoop.auditlogger.default.logger=audit

#
# Repository configuration
# The Repository subsystem provides the special prefix which
# is "org.apache.sqoop.repository.sysprop". Any property that
# is specified with this prefix is parsed out and set as a
# system property. For example, if the built in Derby repository
# is being used, the sysprop prefixed properties can be used
# to affect Derby configuration at startup time by setting
# the appropriate system properties.
#

# Repository provider
org.apache.sqoop.repository.provider=org.apache.sqoop.repository.JdbcRepositoryProvider

# Repository upgrade
# If set to true, it will not upgrade the sqoop respository schema, by default it will iniate the upgrade on server start-up
org.apache.sqoop.repository.schema.immutable=false

# JDBC repository provider configuration
#org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.mysql.MySqlRepositoryHandler
#org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
#org.apache.sqoop.repository.jdbc.maximum.connections=10
#org.apache.sqoop.repository.jdbc.url=jdbc:mysql://192.168.15.50:3306/sqoop?createDatabaseIfNotExist=true
#org.apache.sqoop.repository.jdbc.driver=com.mysql.jdbc.Driver
#org.apache.sqoop.repository.jdbc.user=kfyw
#org.apache.sqoop.repository.jdbc.password=123456

org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:/usr/local/sqoop2/repository/db;create=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=

# System properties for embedded Derby configuration
org.apache.sqoop.repository.sysprop.derby.stream.error.file=/usr/local/sqoop2/logs/derbyrepo.log

#
# Sqoop Connector configuration
# If set to true will initiate Connectors config upgrade during server startup
#
org.apache.sqoop.connector.autoupgrade=false

#
# Sqoop Driver configuration
# If set to true will initiate the Driver config upgrade during server startup
#
org.apache.sqoop.driver.autoupgrade=false

# Sleeping period for reloading configuration file (once a minute)
org.apache.sqoop.core.configuration.provider.properties.sleep=60000

#
# Submission engine configuration
#

# Submission engine class
org.apache.sqoop.submission.engine=org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine

# Number of milliseconds, submissions created before this limit will be removed, default is one day
#org.apache.sqoop.submission.purge.threshold=

# Number of milliseconds for purge thread to sleep, by default one day
#org.apache.sqoop.submission.purge.sleep=

# Number of milliseconds for update thread to sleep, by default 5 minutes
#org.apache.sqoop.submission.update.sleep=

#
# Configuration for Mapreduce submission engine (applicable if it's configured)
#

# Hadoop configuration directory
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=//usr/local/hadoop-3.1.2/etc/hadoop/

# Log level for Sqoop Mapper/Reducer
org.apache.sqoop.submission.engine.mapreduce.configuration.loglevel=INFO

#
# Execution engine configuration
#
org.apache.sqoop.execution.engine=org.apache.sqoop.execution.mapreduce.MapreduceExecutionEngine

#
# Authentication configuration
#
org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true
#org.apache.sqoop.security.authentication.type=KERBEROS
#org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.KerberosAuthenticationHandler
#org.apache.sqoop.security.authentication.kerberos.principal=sqoop/_HOST@NOVALOCAL
#org.apache.sqoop.security.authentication.kerberos.keytab=/home/kerberos/sqoop.keytab
#org.apache.sqoop.security.authentication.kerberos.http.principal=HTTP/_HOST@NOVALOCAL
#org.apache.sqoop.security.authentication.kerberos.http.keytab=/home/kerberos/sqoop.keytab
#org.apache.sqoop.security.authentication.enable.doAs=true
#org.apache.sqoop.security.authentication.proxyuser.#USER#.users=*
#org.apache.sqoop.security.authentication.proxyuser.#USER#.groups=*
#org.apache.sqoop.security.authentication.proxyuser.#USER#.hosts=*

# Default user, default value is "sqoop.anonymous.user"
#org.apache.sqoop.security.authentication.default.user=

#
# Authorization configuration
#
#org.apache.sqoop.security.authorization.handler=org.apache.sqoop.security.authorization.DefaultAuthorizationHandler
#org.apache.sqoop.security.authorization.access_controller=org.apache.sqoop.security.authorization.DefaultAuthorizationAccessController
#org.apache.sqoop.security.authorization.validator=org.apache.sqoop.security.authorization.DefaultAuthorizationValidator
#org.apache.sqoop.security.authorization.authentication_provider=org.apache.sqoop.security.authorization.DefaultAuthenticationProvider
#org.apache.sqoop.security.authorization.server_name=SqoopServer1

#
# SSL/TLS configuration
#
#org.apache.sqoop.security.tls.enabled=false
#org.apache.sqoop.security.tls.protocol="TLSv1.2"
#org.apache.sqoop.security.tls.keystore=
#org.apache.sqoop.security.tls.keystore_password=

#
# Repository Encryption
#

#org.apache.sqoop.security.repo_encryption.enabled=true
#org.apache.sqoop.security.repo_encryption.password=
#org.apache.sqoop.security.repo_encryption.password_generator=
#org.apache.sqoop.security.repo_encryption.hmac_algorithm=HmacSHA256
#org.apache.sqoop.security.repo_encryption.cipher_algorithm=AES
#org.apache.sqoop.security.repo_encryption.cipher_key_size=16
#org.apache.sqoop.security.repo_encryption.cipher_spec=AES/CBC/PKCS5Padding
#org.apache.sqoop.security.repo_encryption.initialization_vector_size=16
#org.apache.sqoop.security.repo_encryption.pbkdf2_algorithm=PBKDF2WithHmacSHA1
#org.apache.sqoop.security.repo_encryption.pbkdf2_rounds=4000


# External connectors load path
# "/path/to/external/connectors/": Add all the connector JARs in the specified folder
#
org.apache.sqoop.connector.external.loadpath=/usr/local/sqoop2/extlib/

# Sqoop application classpath
# ":" separated list of jars to be included in sqoop.
#
org.apache.sqoop.classpath.extra=

# Sqoop extra classpath to be included with all jobs
# ":" separated list of jars to be included in map job classpath.
#
org.apache.sqoop.classpath.job=

#
# Jetty Server configuration
#
#org.apache.sqoop.jetty.thread.pool.worker.max=500
#org.apache.sqoop.jetty.thread.pool.worker.min=5
#org.apache.sqoop.jetty.thread.pool.worker.alive.time=60
#org.apache.sqoop.jetty.port=12000

# Blacklisted Connectors
# ":" separated list of connector names as specified in their
# sqoopconnector.properties file
org.apache.sqoop.connector.blacklist=

配置完后，重新sqoop2-tool upgrade 和sqoop2-tool verify 成功了

Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly

七、补充部分，如果你不想使用derby而改用mysql，可以修改配置如下

org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.mysql.MySqlRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:mysql://192.168.15.50:3306/sqoop?createDatabaseIfNotExist=true
org.apache.sqoop.repository.jdbc.driver=com.mysql.jdbc.Driver
org.apache.sqoop.repository.jdbc.user=kfyw
org.apache.sqoop.repository.jdbc.password=123456

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=//usr/local/hadoop-3.1.2/etc/hadoop/
org.apache.sqoop.connector.external.loadpath=/usr/local/sqoop2/extlib/

 当然这么配置好了，sqoop2-tool upgrade 和sqoop2-tool verify 时也会报错

2020-04-13 11:15:22,916 INFO  [org.apache.sqoop.repository.mysql.MySqlRepositoryHandler.initialize(MySqlRepositoryHandler.java:68)] MySqlRepositoryHandler initialized.
2020-04-13 11:15:22,918 INFO  [org.apache.sqoop.repository.JdbcRepositoryProvider.initializeRepositoryHandler(JdbcRepositoryProvider.java:160)] JdbcRepositoryProvider initialized
2020-04-13 11:15:22,918 INFO  [org.apache.sqoop.repository.JdbcRepositoryProvider.initialize(JdbcRepositoryProvider.java:67)] JdbcRepository initialized.
2020-04-13 11:15:22,918 INFO  [org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:127)] Creating or updating respository at bootup
2020-04-13 11:15:23,230 INFO  [org.apache.sqoop.repository.JdbcRepository$1.doIt(JdbcRepository.java:131)] Creating repository schema objects
2020-04-13 11:15:23,238 INFO  [org.apache.sqoop.repository.mysql.MySqlRepositoryHandler.createOrUpgradeRepository(MySqlRepositoryHandler.java:122)] Detected repository version: 0
2020-04-13 11:15:23,259 ERROR [org.apache.sqoop.repository.common.CommonRepositoryHandler.runQuery(CommonRepositoryHandler.java:2685)] Can't execute query: CREATE DATABASE IF NOT EXISTS"SQOOP"
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"SQOOP"' at line 1
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
	at com.mysql.jdbc.Util.getInstance(Util.java:383)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1062)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4208)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4140)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2597)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2758)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2826)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1302)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.sqoop.repository.common.CommonRepositoryHandler.runQuery(CommonRepositoryHandler.java:2672)
	at org.apache.sqoop.repository.mysql.MySqlRepositoryHandler.createOrUpgradeRepository(MySqlRepositoryHandler.java:129)
	at org.apache.sqoop.repository.JdbcRepository$1.doIt(JdbcRepository.java:132)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:91)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:62)
	at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:128)
	at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:128)
	at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
	at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)
2020-04-13 11:15:23,263 INFO  [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2020-04-13 11:15:23,265 ERROR [org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:55)] Can't finish upgrading RepositoryManager, Driver and ConnectionManager:
org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to run specified query - CREATE DATABASE IF NOT EXISTS"SQOOP"
	at org.apache.sqoop.repository.common.CommonRepositoryHandler.runQuery(CommonRepositoryHandler.java:2686)
	at org.apache.sqoop.repository.mysql.MySqlRepositoryHandler.createOrUpgradeRepository(MySqlRepositoryHandler.java:129)
	at org.apache.sqoop.repository.JdbcRepository$1.doIt(JdbcRepository.java:132)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:91)
	at org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:62)
	at org.apache.sqoop.repository.JdbcRepository.createOrUpgradeRepository(JdbcRepository.java:128)
	at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:128)
	at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
	at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
	at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"SQOOP"' at line 1
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
	at com.mysql.jdbc.Util.getInstance(Util.java:383)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1062)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4208)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4140)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2597)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2758)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2826)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1302)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.sqoop.repository.common.CommonRepositoryHandler.runQuery(CommonRepositoryHandler.java:2672)
	... 9 more
2020-04-13 11:15:23,266 INFO  [org.

解决办法：在mysql中设置全局变量：set global sql_mode ='ANSI_QUOTES';或者修改mysql配置文件sql_mode=ANSI_QUOTES

 

第二部分:sqoop操作说明

[hadoop@hadoop-01 sbin]$ sqoop.sh client 登录客户端 
Setting conf dir: /usr/local/sqoop2/bin/../conf
Sqoop home directory: /usr/local/sqoop2
Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000> set server --host 127.0.0.1 --port 12000 --webapp sqoop  #连接服务器端
Server is set successfully
sqoop:000> set option --name verbose --value true   #设置提示报错
Verbose option was changed to true
sqoop:000> show connector    #查看连接器
0    [main] WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+------------------------+---------+------------------------------------------------------------+----------------------+
|          Name          | Version |                           Class                            | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector       | FROM/TO              |
| kite-connector         | 1.99.7  | org.apache.sqoop.connector.kite.KiteConnector              | FROM/TO              |
| oracle-jdbc-connector  | 1.99.7  | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO              |
| ftp-connector          | 1.99.7  | org.apache.sqoop.connector.ftp.FtpConnector                | TO                   |
| hdfs-connector         | 1.99.7  | org.apache.sqoop.connector.hdfs.HdfsConnector              | FROM/TO              |
| kafka-connector        | 1.99.7  | org.apache.sqoop.connector.kafka.KafkaConnector            | TO                   |
| sftp-connector         | 1.99.7  | org.apache.sqoop.connector.sftp.SftpConnector              | TO                   |
+------------------------+---------+------------------------------------------------------------+----------------------+
sqoop:000> show link  #查看link
+----------------+------------------------+---------+
|      Name      |     Connector Name     | Enabled |
+----------------+------------------------+---------+
| sx_hadoop      | generic-jdbc-connector | true    |
| sx_hadoop_hdfs | hdfs-connector         | true    |
+----------------+------------------------+---------+
sqoop:000> show job  #查看JOB
+----+----------------+------------------------------------+---------------------------------+---------+
| Id |      Name      |           From Connector           |          To Connector           | Enabled |
+----+----------------+------------------------------------+---------------------------------+---------+
| 3  | clean_user_job | sx_hadoop (generic-jdbc-connector) | sx_hadoop_hdfs (hdfs-connector) | true    |
+----+----------------+------------------------------------+---------------------------------+---------+

1、创建 mysql连接器

   sqoop:000>create link -connector generic-jdbc-connector