Hadoop datanode隔一段时间就挂掉 报错：java.lang.OutOfMemoryError: GC overhead limit exceeded
背景：
之前数据量增加，集群开始报错10000 millis timeout while waiting for channel to be ready for write，后来发现是hdfs写数据超时，通过修改配置文件，集群正常。
https://blog.csdn.net/dehu_zhou/article/details/81533802
集群运行几天后，发现又出现新的问题了，datanode经常一批批的挂掉，每次还不是一样的节点，继续查看日志；
发现datanode日志一直打印：
2018-08-09 09:49:42,830 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1102ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1450ms
2018-08-09 09:49:44,371 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1040ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1396ms
2018-08-09 09:49:47,343 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1169ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1542ms
2018-08-09 09:49:50,286 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1042ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1444ms
2018-08-09 09:50:15,703 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2197ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=2679ms
2018-08-09 09:50:18,903 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1595ms由于本人对JVM也不懂，只能怀疑是内存不足引起，GC超时，好吧，笨人使用笨方法，增加datanode的内存
在datanode的haoop-env.sh里面配置（根据自己集群调整参数）
export HADOOP_DATANODE_OPTS="-Xmx3g -Xms3g -Xmn2g -Xloggc:$HADOOP_LOG_DIR/datanode_gc.log"配置完成，拷贝到所有节点，刚好集群没有任务要执行，重启集群，然后等待集群重启。
。。。。。。。。。。。。。。。。。
等了20分钟，发现集群还没有起来，还在加载edit文件，平时没有那么慢，，，赶紧去看namenode日志，发现namenode也报错了：
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:549)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:422)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:403)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:359)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:545)
... 10 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.io.UnixFileSystem.resolve(UnixFileSystem.java:108)
at java.io.File.(File.java:262)
at java.io.File.listFiles(File.java:1212)
at org.apache.hadoop.fs.FileUtil.listFiles(FileUtil.java:1162)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.compileReport(DirectoryScanner.java:595)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.compileReport(DirectoryScanner.java:610)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.compileReport(DirectoryScanner.java:610)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.compileReport(DirectoryScanner.java:610)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:585)
at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:570)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more 纳尼，又是JVM问题，欺负我不懂JVM啊，，，好吧，继续使用笨方法，
修改namendoe节点的hadoop-env.sh
# The maximum amount of heap to use, in MB. Default is 1000.
export HADOOP_HEAPSIZE=10240
export HADOOP_NAMENODE_INIT_HEAPSIZE="100000"继续重启namenode，等3分钟左右，发现集群成功启动，截止到现在，已经成功运行100多个任务，运行3天了，说明集群暂时正常。
以上纯属个人愚见，而且都是使用的笨方法，欢迎有更好方法的大神交流，谢谢！
附hadoop env可以配置的一些参数：
感谢大神分享，原文链接：https://blog.csdn.net/zxln007/article/details/79356654
| 环境参数 | 参数设置 | 参数说明 | 备注 |
| HADOOP_OPTS | -Djava.net.preferIPv4Stack=true | 优先使用IPV4，禁用IPV6 | |
| HADOOP_NAMENODE_OPTS | Xms140G | 初始堆内存大小 | |
| Xmx140G | 最大堆内存大小 | ||
| XX:NewRatio=8 | 设置年轻代和年老代的比值。如:为3，表示年轻代与年老代比值为1：3，年轻代占整个年轻代年老代和的1/4 | ||
| XX:SurvivorRatio=4 | 年轻代中Eden区与两个Survivor区的比值。注意Survivor区有两个。如：3，表示Eden：Survivor=3：2，一个Survivor区占整个年轻代的1/5 | ||
| XX:MaxPermSize=200M | 设置持久代大小 | ||
| XX:+UseParNewGC | ParNew收集器其实就是Serial收集器的多线程版本。新生代并行，老年代串行；新生代复制算法、老年代标记-压缩 | ||
| XX:+UseConcMarkSweepGC | 设置CMS收集器（并发收集器） | ||
| XX:+CMSParallelRemarkEnabled | 为了减少第二次暂停的时间，开启并行remark | ||
| XX:MaxDirectMemorySize=512M | 此参数的含义是当Direct ByteBuffer分配的堆外内存到达指定大小后，即触发Full GC | ||
| XX:CMSInitiatingOccupancyFraction=75 | 设置CMS收集器在老年代空间被使用多少后触发垃圾收集，默认值为68%，仅在使用CMS时生效 | ||
| XX:ConcGCThreads=8 | |||
| verbose:gc | |||
| XX:+PrintGCDetails | 打印GC详情 | ||
| XX:+PrintGCDateStamps | 打印GC时间戳 | ||
| Xloggc:/home/bigdata/hadoop/logs/namenode.gc.log | gc日志的目录及文件名 | ||
| Dcom.sun.management.jmxremote | |||
| HADOOP_DATANODE_OPTS | Xms3G | 初始堆内存 |
|
| Xmx3G | 最大堆内存 | ||
| Xmn512M | 设置年轻代大小 | ||
| XX:MaxDirectMemorySize=512M | 此参数的含义是当Direct ByteBuffer分配的堆外内存到达指定大小后，即触发Full GC | ||
| XX:+UseParNewGC | |||
| XX:+UseConcMarkSweepGC | |||
| XX:CMSInitiatingOccupancyFraction=75 |
附:网上找到的大神的一个配置，感谢大神分享
原文链接： http://blog.csdn.net/maijiyouzou/article/details/23740225
JVM_OPTS="-server -verbose:gc
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=9
-XX:GCLogFileSize=256m
-XX:+DisableExplicitGC
-XX:+UseCompressedOops
-XX:SoftRefLRUPolicyMSPerMB=0
-XX:+UseFastAccessorMethods
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:CMSInitiatingOccupancyFraction=70
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:+CMSClassUnloadingEnabled
-XX:CMSMaxAbortablePrecleanTime=301
-XX:+CMSScavengeBeforeRemark
-XX:PermSize=160m
-XX:GCTimeRatio=19
-XX:SurvivorRatio=2
-XX:MaxTenuringThreshold=100" //这个参数很重要，MaxTenuringThreshold这个参数用于控制对象能经历多少次Minor GC才晋升到旧生代，默认值是15
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="$JVM_OPTS -Xmx80g -Xms80g -Xmn21g -Xloggc:$HADOOP_LOG_DIR/namenode_gc.log"
export HADOOP_SECONDARYNAMENODE_OPTS="$JVM_OPTS -Xmx80g -Xms80g -Xmn21g"
export HADOOP_DATANODE_OPTS="$JVM_OPTS -Xmx3g -Xms3g -Xmn2g -Xloggc:$HADOOP_LOG_DIR/datanode_gc.log"
export HADOOP_BALANCER_OPTS="$JVM_OPTS -Xmx1g -Xms1g -Xmn512m -Xloggc:$HADOOP_LOG_DIR/balancer_gc.log"
export HADOOP_JOBTRACKER_OPTS="$JVM_OPTS -Xmx1g -Xms1g -Xmn512m -Xloggc:$HADOOP_LOG_DIR/jobtracker_gc.log"
export HADOOP_TASKTRACKER_OPTS="$JVM_OPTS -Xmx1g -Xms1g -Xmn512m -Xloggc:$HADOOP_LOG_DIR/tasktracker_gc.log"
export HADOOP_CLIENT_OPTS="$JVM_OPTS -Xmx512m -Xms512m -Xmn256m"