troubleshooting-sqoop mysql导入hive 报：GC overhead limit exceeded

Halting due to Out Of Memory Error...
18/09/13 21:42:17 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000001_0, Status : FAILED
Error: Java heap space
18/09/13 21:43:26 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000000_1, Status : FAILED
Error: Java heap space
18/09/13 21:44:04 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000001_1, Status : FAILED
Error: GC overhead limit exceeded
18/09/13 21:45:07 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000000_2, Status : FAILED
Error: Java heap space
18/09/13 21:45:22 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000001_2, Status : FAILED
Error: GC overhead limit exceeded

 

解决办法

mysql connection连接字符串加上：
?dontTrackOpenResources=true&defaultFetchSize=1000&useCursorFetch=true
参考：
https://stackoverflow.com/questions/26484873/cloudera-settings-sqoop-import-gives-java-heap-space-error-and-gc-overhead-limit

异常日志

18/09/13 21:41:49 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000000_0, Status : FAILED
[2018-09-13 21:41:48.551]Exception from container-launch.
Container id: container_1536756558194_0031_01_000002
Exit code: 255
Shell output: main : command provided 1
main : run as user is chenweidong
main : requested yarn user is chenweidong
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file /data/yarn/nm/nmPrivate/application_1536756558194_0031/container_1536756558194_0031_01_000002/container_1536756558194_0031_01_000002.pid.tmp
Writing to cgroup task files...
Creating local dirs...
Launching container...
Getting exit code file...
Creating script paths...

[2018-09-13 21:41:48.553]Container exited with a non-zero exit code 255. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Thu Sep 13 21:40:41 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Halting due to Out Of Memory Error...

[2018-09-13 21:41:48.554]Container exited with a non-zero exit code 255. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Thu Sep 13 21:40:41 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Halting due to Out Of Memory Error...
18/09/13 21:42:17 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000001_0, Status : FAILED
Error: Java heap space
18/09/13 21:43:26 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000000_1, Status : FAILED
Error: Java heap space
18/09/13 21:44:04 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000001_1, Status : FAILED
Error: GC overhead limit exceeded
18/09/13 21:45:07 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000000_2, Status : FAILED
Error: Java heap space
18/09/13 21:45:22 INFO mapreduce.Job: Task Id : attempt_1536756558194_0031_m_000001_2, Status : FAILED
Error: GC overhead limit exceeded
18/09/13 21:46:37 INFO mapreduce.Job: map 100% reduce 0%
18/09/13 21:46:37 INFO mapreduce.Job: Job job_1536756558194_0031 failed with state FAILED due to: Task failed task_1536756558194_0031_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

转载于:https://www.cnblogs.com/chwilliam85/p/9693276.html