yarn:hadoop YarnChild OutOfMemoryError Java heap space || GCOverhead limit

做kylin的时候

商品表 txt文件 49M 一共1千万数据
类别表 1.7M 10万条数据

然后在进行hive去重的时候报错

报错原因是
有几个

 org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
 

以及

B.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:310)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy15.create(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:2146)
	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1804)
	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1728)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:438)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:434)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:434)
2019-05-14 23:33:57,661 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.zip.ZipCoder.getBytes(ZipCoder.java:80)
	at java.util.zip.ZipFile.getEntry(ZipFile.java:316)
	at java.util.jar.JarFile.getEntry(JarFile.java:240)
	at java.util.jar.JarFile.getJarEntry(JarFile.java:223)
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1054)

解决方法

# 原本4G
yarn.scheduler.maximum-allocation-mb=6G
# 原本4G  增加到6G 后来增加到7G
yarn.nodemanager.resource.memory-mb=7G


# 原本1G  增加到2G 这是数据量大的时候 报错这个
mapreduce.map.memory.mb=2G
# 原本1G  增加到2G 。。。。6G
mapreduce.reduce.memory.mb=6G