解决:Sqoop导入数据到Hive连接hiveserver2卡住

问题

  • 执行sqoop job要将mysql中数据导入hive表,执行到连接hiveserver2时就卡住不动
  • 单独执行连接hiveserver2可以正常连接

Import options:

sqoop import --connect jdbc:mysql://xxx:3306/test --username xxxxx --password xxxxxx 
--table t_user 
--fields-terminated-by '\001' 
--hive-import 
--hive-database sqoop2hive
--hive-table t_user_hive 
--hive-overwrite 
--delete-target-dir --m 1

Output:

......
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/07/27 11:49:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.7.1.1.0-565
20/07/27 11:49:12 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
20/07/27 11:49:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
20/07/27 11:49:13 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
......
20/07/27 11:49:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/bf26bf5b6aa581837cbdf0d60c4867e1/t_user.jar
20/07/27 11:49:16 INFO tool.ImportTool: Destination directory t_user is not present, hence not deleting.
20/07/27 11:49:16 WARN manager.MySQLManager: It looks like you are importing from mysql.
20/07/27 11:49:16 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
......
20/07/27 11:49:16 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/07/27 11:49:16 INFO client.RMProxy: Connecting to ResourceManager at xxx/10.23.98.75:8032
20/07/27 11:49:16 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/root/.staging/job_1595209123778_0044
20/07/27 11:49:19 INFO db.DBInputFormat: Using read commited transaction isolation
......
20/07/27 11:49:19 INFO impl.YarnClientImpl: Submitted application application_1595209123778_0044
20/07/27 11:49:19 INFO mapreduce.Job: The url to track the job: http://xxx:8088/proxy/application_1595209123778_0044/
20/07/27 11:49:19 INFO mapreduce.Job: Running job: job_1595209123778_0044
20/07/27 11:49:26 INFO mapreduce.Job: Job job_1595209123778_0044 running in uber mode : false
20/07/27 11:49:26 INFO mapreduce.Job:  map 0% reduce 0%
20/07/27 11:49:32 INFO mapreduce.Job:  map 100% reduce 0%
20/07/27 11:49:32 INFO mapreduce.Job: Job job_1595209123778_0044 completed successfully
20/07/27 11:49:33 INFO mapreduce.Job: Counters: 33
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=248550
		FILE: Number of read operations=0
	    ......
		HDFS: Number of write operations=2
		HDFS: Number of bytes read erasure-coded=0
	Job Counters 
		Launched map tasks=1
		Other local map tasks=1
		.......
		Total megabyte-milliseconds taken by all map tasks=3757056
	Map-Reduce Framework
		Map input records=9
		......
		Peak Map Virtual memory (bytes)=2851627008
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=104
20/07/27 11:49:33 INFO mapreduce.ImportJobBase: Transferred 104 bytes in 16.5599 seconds (6.2802 bytes/sec)
20/07/27 11:49:33 INFO mapreduce.ImportJobBase: Retrieved 9 records.
20/07/27 11:49:33 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t_user` AS t LIMIT 1
20/07/27 11:49:33 INFO hive.HiveImport: Loading uploaded data into Hive
20/07/27 11:49:33 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
20/07/27 11:49:33 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.3266817/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20/07/27 11:49:33 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.3266817/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
......
20/07/27 11:49:35 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
20/07/27 11:49:35 INFO hive.HiveImport: SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
20/07/27 11:49:37 INFO hive.HiveImport: Connecting to jdbc:hive2://xxx:2181,xxx:2181,xxx:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2

就停在这:
20/07/27 11:49:37 INFO hive.HiveImport:Connecting to jdbc:hive2://xxx:2181,xxx:2181,xxx:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
连接不上 hiveserver2。

解决

在hive conf目录(一般在/etc/hive/conf)新建(如果没有)一个beeline-hs2-connection.xml文件:





 beeline.hs2.connection.user
 hive


 beeline.hs2.connection.password
 hive


解决:Sqoop导入数据到Hive连接hiveserver2卡住_第1张图片

熄灯

参考:
cloudera社区
apache wiki

你可能感兴趣的:(cloudera,data,platform)