Hadoop集群环境下HBase集群slave节点总是掉问题

HBase表格设计：

create'DATA',{NAME=>'0',VERSIONS=>3,BLOCKSIZE=>'65535',BLOCKCACHE=>'false',IN_MEMORY=>'true',MIN_VERSIONS=>'1',TTL=>'18000'},{NAME=>'1',VERSIONS=>3,BLOCKSIZE=>'65535',BLOCKCACHE=>'false',IN_MEMORY=>'true',MIN_VERSIONS=>'1',TTL=>'18000'},{NAME=>'2',VERSIONS=>3,BLOCKSIZE=>'65535',BLOCKCACHE=>'false',IN_MEMORY=>'true',MIN_VERSIONS=>'1',TTL=>'18000'}

 

HBase：批量put

出错：

2017年 09月 14日星期四 09:12:23 CST Startingregionserver on slave1.hadoop

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 7897

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) 7897

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

2017-09-14 09:12:25,625INFO  [main] util.VersionInfo: HBase1.2.6

2017-09-14 09:12:25,626INFO  [main] util.VersionInfo: Sourcecode repositoryfile:///home/busbey/projects/hbase/hbase-assembly/target/hbase-1.2.6revision=Unknown

2017-09-14 09:12:25,626INFO  [main] util.VersionInfo: Compiledby busbey on Mon May 29 02:25:32 CDT 2017

2017-09-14 09:12:25,626INFO  [main] util.VersionInfo: Fromsource with checksum 7e8ce83a648e252758e9dae1fbe779c9

2017-09-14 09:12:26,747INFO  [main] util.ServerCommandLine:env:JAVA_HOME=/usr/local/jdk1.7

2017-09-14 09:12:26,747INFO  [main] util.ServerCommandLine:env:SHLVL=3

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_LOG_DIR=/home/softs/hbase/bin/../logs

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_HOME=/home/softs/hbase/bin/..

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:KDEDIRS=/usr

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_OPTS=-XX:+UseConcMarkSweepGC  -XX:PermSize=128m -XX:MaxPermSize=128m-Dhbase.log.dir=/home/softs/hbase/bin/../logs-Dhbase.log.file=hbase-root-regionserver-slave1.hadoop.log-Dhbase.home.dir=/home/softs/hbase/bin/.. -Dhbase.id.str=root-Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_ENV_INIT=true

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:SSH_CLIENT=10.9.148.193 51362 22

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:MAIL=/var/mail/root

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:SELINUX_ROLE_REQUESTED=

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_ZNODE_FILE=/tmp/hbase-root-regionserver.znode

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:PWD=/home/softs/hbase

2017-09-14 09:12:26,748INFO  [main] util.ServerCommandLine:env:HBASE_MASTER_OPTS= -XX:PermSize=128m -XX:MaxPermSize=128m

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:LOGNAME=root

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:CVS_RSH=ssh

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_MANAGES_ZK=false

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:G_BROKEN_FILENAMES=1

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:QTDIR=/usr/lib64/qt-3.3

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_NICENESS=0

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_REST_OPTS=

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:MALLOC_ARENA_MAX=4

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:SSH_CONNECTION=10.9.148.193 51362 10.9.148.190 22

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:SELINUX_USE_CURRENT_RANGE=

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:SHELL=/bin/bash

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine: env:SELINUX_LEVEL_REQUESTED=

2017-09-14 09:12:26,749INFO  [main] util.ServerCommandLine:env:HBASE_REGIONSERVER_OPTS= -XX:PermSize=128m -XX:MaxPermSize=128m

2017-09-14 09:12:26,750INFO  [main] util.ServerCommandLine:env:HBASE_ROOT_LOGGER=INFO,RFA

2017-09-14 09:12:26,752INFO  [main] util.ServerCommandLine:env:CLASSPATH=/home/softs/hbase/conf:/usr/local/jdk1.7/lib/tools.jar:/home/softs/hbase/bin/..:/home/softs/hbase/bin/../lib/activation-1.1.jar:/home/softs/hbase/bin/../lib/aopalliance-1.0.jar:/home/softs/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/asm-3.1.jar:/home/softs/hbase/bin/../lib/avro-1.7.4.jar:/home/softs/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/softs/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/softs/hbase/bin/../lib/commons-cli-1.2.jar:/home/softs/hbase/bin/../lib/commons-codec-1.9.jar:/home/softs/hbase/bin/../lib/commons-collections-3.2.2.jar:/home/softs/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/softs/hbase/bin/../lib/commons-configuration-1.6.jar:/home/softs/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/softs/hbase/bin/../lib/commons-digester-1.8.jar:/home/softs/hbase/bin/../lib/commons-el-1.0.jar:/home/softs/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/softs/hbase/bin/../lib/commons-io-2.4.jar:/home/softs/hbase/bin/../lib/commons-lang-2.6.jar:/home/softs/hbase/bin/../lib/commons-logging-1.2.jar:/home/softs/hbase/bin/../lib/commons-math-2.2.jar:/home/softs/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/softs/hbase/bin/../lib/commons-net-3.1.jar:/home/softs/hbase/bin/../lib/disruptor-3.3.0.jar:/home/softs/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/softs/hbase/bin/../lib/guava-12.0.1.jar:/home/softs/hbase/bin/../lib/guice-3.0.jar:/home/softs/hbase/bin/../lib/guice-servlet-3.0.jar:/home/softs/hbase/bin/../lib/hadoop-annotations-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-auth-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-hdfs-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-api-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-client-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-examples-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-external-blockcache-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-prefix-tree-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-procedure-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-protocol-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-resource-bundle-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-rest-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-shell-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-thrift-1.2.6.jar:/home/softs/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/softs/hbase/bin/../lib/httpclient-4.2.5.jar:/home/softs/hbase/bin/../lib/httpcore-4.4.1.jar:/home/softs/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/softs/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/home/softs/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/softs/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/softs/hbase/bin/../lib/javax.inject-1.jar:/home/softs/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/softs/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/softs/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/softs/hbase/bin/../lib/jcodings-1.0.8.jar:/home/softs/hbase/bin/../lib/jersey-client-1.9.jar:/home/softs/hbase/bin/../lib/jersey-core-1.9.jar:/home/softs/hbase/bin/../lib/jersey-guice-1.9.jar:/home/softs/hbase/bin/../lib/jersey-json-1.9.jar:/home/softs/hbase/bin/../lib/jersey-server-1.9.jar:/home/softs/hbase/bin/../lib/jets3t-0.9.0.jar:/home/softs/hbase/bin/../lib/jettison-1.3.3.jar:/home/softs/hbase/bin/../lib/jetty-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/softs/hbase/bin/../lib/joni-2.1.2.jar:/home/softs/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/softs/hbase/bin/../lib/jsch-0.1.42.jar:/home/softs/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/junit-4.12.jar:/home/softs/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/softs/hbase/bin/../lib/libthrift-0.9.3.jar:/home/softs/hbase/bin/../lib/log4j-1.2.17.jar:/home/softs/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/softs/hbase/bin/../lib/netty-all-4.0.23.Final.jar:/home/softs/hbase/bin/../lib/paranamer-2.3.jar:/home/softs/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5.jar:/home/softs/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/softs/hbase/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/softs/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/softs/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/softs/hbase/bin/../lib/xmlenc-0.52.jar:/home/softs/hbase/bin/../lib/xz-1.0.jar:/home/softs/hbase/bin/../lib/zookeeper-3.4.6.jar:

2017-09-14 09:12:26,762INFO  [main] util.ServerCommandLine:env:HBASE_THRIFT_OPTS=

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:QTINC=/usr/lib64/qt-3.3/include

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:USER=root

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:QTLIB=/usr/lib64/qt-3.3/lib

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:KDE_IS_PRELINKED=1

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_SECURITY_LOGGER=INFO,RFAS

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HOME=/root

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:LESSOPEN=||/usr/bin/lesspipe.sh %s

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_START_FILE=/tmp/hbase-root-regionserver.autorestart

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_LOGFILE=hbase-root-regionserver-slave1.hadoop.log

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:LANG=zh_CN.UTF-8

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass

2017-09-14 09:12:26,763INFO  [main] util.ServerCommandLine:env:HBASE_LOG_PREFIX=hbase-root-regionserver-slave1.hadoop

2017-09-14 09:12:26,764INFO  [main] util.ServerCommandLine:env:HBASE_IDENT_STRING=root

2017-09-14 09:12:26,764INFO  [main] util.ServerCommandLine:vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation,vmVersion=24.45-b08

2017-09-14 09:12:26,765INFO  [main] util.ServerCommandLine:vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p,-XX:+UseConcMarkSweepGC, -XX:PermSize=128m, -XX:MaxPermSize=128m, -Dhbase.log.dir=/home/softs/hbase/bin/../logs,-Dhbase.log.file=hbase-root-regionserver-slave1.hadoop.log,-Dhbase.home.dir=/home/softs/hbase/bin/.., -Dhbase.id.str=root,-Dhbase.root.logger=INFO,RFA, -Dhbase.security.logger=INFO,RFAS]

2017-09-14 09:12:26,969WARN  [main] util.NativeCodeLoader:Unable to load native-hadoop library for your platform... using builtin-javaclasses where applicable

2017-09-14 09:12:27,474INFO  [main] regionserver.RSRpcServices:regionserver/slave1.hadoop/10.9.148.190:16020 server-side HConnectionretries=350

2017-09-14 09:12:27,790INFO  [main] ipc.SimpleRpcScheduler:Using deadline as user call queue, count=3

2017-09-14 09:12:27,810INFO  [main] ipc.RpcServer:regionserver/slave1.hadoop/10.9.148.190:16020: started 10 reader(s) listeningon port=16020

2017-09-14 09:12:29,678INFO  [main] fs.HFileSystem: Addedintercepting call to namenode#getBlockLocations so can do block reorderingusing class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks

2017-09-14 09:12:29,988INFO  [main]zookeeper.RecoverableZooKeeper: Process identifier=regionserver:16020connecting to ZooKeeperensemble=master:2181,slave1:2181,slave2:2181,slave3:2181

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:host.name=slave1.hadoop

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.version=1.7.0_45

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.vendor=Oracle Corporation

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.home=/usr/local/jdk1.7/jre

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.class.path=/home/softs/hbase/conf:/usr/local/jdk1.7/lib/tools.jar:/home/softs/hbase/bin/..:/home/softs/hbase/bin/../lib/activation-1.1.jar:/home/softs/hbase/bin/../lib/aopalliance-1.0.jar:/home/softs/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/softs/hbase/bin/../lib/asm-3.1.jar:/home/softs/hbase/bin/../lib/avro-1.7.4.jar:/home/softs/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/softs/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/softs/hbase/bin/../lib/commons-cli-1.2.jar:/home/softs/hbase/bin/../lib/commons-codec-1.9.jar:/home/softs/hbase/bin/../lib/commons-collections-3.2.2.jar:/home/softs/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/softs/hbase/bin/../lib/commons-configuration-1.6.jar:/home/softs/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/softs/hbase/bin/../lib/commons-digester-1.8.jar:/home/softs/hbase/bin/../lib/commons-el-1.0.jar:/home/softs/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/softs/hbase/bin/../lib/commons-io-2.4.jar:/home/softs/hbase/bin/../lib/commons-lang-2.6.jar:/home/softs/hbase/bin/../lib/commons-logging-1.2.jar:/home/softs/hbase/bin/../lib/commons-math-2.2.jar:/home/softs/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/softs/hbase/bin/../lib/commons-net-3.1.jar:/home/softs/hbase/bin/../lib/disruptor-3.3.0.jar:/home/softs/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/softs/hbase/bin/../lib/guava-12.0.1.jar:/home/softs/hbase/bin/../lib/guice-3.0.jar:/home/softs/hbase/bin/../lib/guice-servlet-3.0.jar:/home/softs/hbase/bin/../lib/hadoop-annotations-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-auth-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-hdfs-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-api-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-client-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-annotations-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-client-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-common-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-examples-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-external-blockcache-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop2-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-hadoop-compat-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-it-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-prefix-tree-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-procedure-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-protocol-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-resource-bundle-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-rest-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-server-1.2.6-tests.jar:/home/softs/hbase/bin/../lib/hbase-shell-1.2.6.jar:/home/softs/hbase/bin/../lib/hbase-thrift-1.2.6.jar:/home/softs/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/softs/hbase/bin/../lib/httpclient-4.2.5.jar:/home/softs/hbase/bin/../lib/httpcore-4.4.1.jar:/home/softs/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/softs/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/home/softs/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/softs/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/softs/hbase/bin/../lib/javax.inject-1.jar:/home/softs/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/softs/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/softs/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/softs/hbase/bin/../lib/jcodings-1.0.8.jar:/home/softs/hbase/bin/../lib/jersey-client-1.9.jar:/home/softs/hbase/bin/../lib/jersey-core-1.9.jar:/home/softs/hbase/bin/../lib/jersey-guice-1.9.jar:/home/softs/hbase/bin/../lib/jersey-json-1.9.jar:/home/softs/hbase/bin/../lib/jersey-server-1.9.jar:/home/softs/hbase/bin/../lib/jets3t-0.9.0.jar:/home/softs/hbase/bin/../lib/jettison-1.3.3.jar:/home/softs/hbase/bin/../lib/jetty-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/softs/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/softs/hbase/bin/../lib/joni-2.1.2.jar:/home/softs/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/softs/hbase/bin/../lib/jsch-0.1.42.jar:/home/softs/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/softs/hbase/bin/../lib/junit-4.12.jar:/home/softs/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/softs/hbase/bin/../lib/libthrift-0.9.3.jar:/home/softs/hbase/bin/../lib/log4j-1.2.17.jar:/home/softs/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/softs/hbase/bin/../lib/netty-all-4.0.23.Final.jar:/home/softs/hbase/bin/../lib/paranamer-2.3.jar:/home/softs/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/softs/hbase/bin/../lib/servlet-api-2.5.jar:/home/softs/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/softs/hbase/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/softs/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/softs/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/softs/hbase/bin/../lib/xmlenc-0.52.jar:/home/softs/hbase/bin/../lib/xz-1.0.jar:/home/softs/hbase/bin/../lib/zookeeper-3.4.6.jar:

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.io.tmpdir=/tmp

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:java.compiler=

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:os.name=Linux

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:os.arch=amd64

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:os.version=2.6.32-696.el6.x86_64

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:user.name=root

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:user.home=/root

2017-09-14 09:12:30,006INFO  [main] zookeeper.ZooKeeper: Clientenvironment:user.dir=/home/softs/hbase

2017-09-14 09:12:30,008INFO  [main] zookeeper.ZooKeeper:Initiating client connection,connectString=master:2181,slave1:2181,slave2:2181,slave3:2181sessionTimeout=90000 watcher=regionserver:160200x0,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbase

2017-09-14 09:12:30,075INFO [main-SendThread(slave1.hadoop:2181)] zookeeper.ClientCnxn: Openingsocket connection to server slave1.hadoop/10.9.148.190:2181. Will not attemptto authenticate using SASL (unknown error)

2017-09-14 09:12:30,080INFO [main-SendThread(slave1.hadoop:2181)] zookeeper.ClientCnxn: Socketconnection established to slave1.hadoop/10.9.148.190:2181, initiating session

2017-09-14 09:12:30,125INFO [main-SendThread(slave1.hadoop:2181)] zookeeper.ClientCnxn: Sessionestablishment complete on server slave1.hadoop/10.9.148.190:2181, sessionid =0x15e7df074580001, negotiated timeout = 40000

2017-09-14 09:12:30,153INFO  [RpcServer.responder] ipc.RpcServer:RpcServer.responder: starting

2017-09-14 09:12:30,155INFO  [RpcServer.listener,port=16020]ipc.RpcServer: RpcServer.listener,port=16020: starting

2017-09-14 09:12:30,353INFO  [main] mortbay.log: Logging toorg.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2017-09-14 09:12:30,358INFO  [main] http.HttpRequestLog: Httprequest log for http.requests.regionserver is not defined

2017-09-14 09:12:30,376INFO  [main] http.HttpServer: Addedglobal filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)

2017-09-14 09:12:30,377INFO  [main] http.HttpServer: Addedglobal filter 'clickjackingprevention'(class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)

2017-09-14 09:12:30,380INFO  [main] http.HttpServer: Addedfilter static_user_filter(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter)to context regionserver

2017-09-14 09:12:30,380INFO  [main] http.HttpServer: Addedfilter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter)to context logs

2017-09-14 09:12:30,380INFO  [main] http.HttpServer: Addedfilter static_user_filter(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter)to context static

2017-09-14 09:12:30,400INFO  [main] http.HttpServer: Jetty boundto port 16030

2017-09-14 09:12:30,400INFO  [main] mortbay.log: jetty-6.1.26

2017-09-14 09:12:31,015INFO  [main] mortbay.log: [email protected]:16030

2017-09-14 09:12:31,195INFO [regionserver/slave1.hadoop/10.9.148.190:16020]zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x19a4fdf0connecting to ZooKeeperensemble=master:2181,slave1:2181,slave2:2181,slave3:2181

2017-09-14 09:12:31,203INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]zookeeper.ZooKeeper: Initiating client connection,connectString=master:2181,slave1:2181,slave2:2181,slave3:2181sessionTimeout=90000 watcher=hconnection-0x19a4fdf00x0,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbase

2017-09-14 09:12:31,210INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Opening socket connection to serverslave2.hadoop/10.9.148.191:2181. Will not attempt to authenticate using SASL(unknown error)

2017-09-14 09:12:31,215INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Socket connection established toslave2.hadoop/10.9.148.191:2181, initiating session

2017-09-14 09:12:31,272INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Session establishment complete on serverslave2.hadoop/10.9.148.191:2181, sessionid = 0x25e7df015ce0002, negotiatedtimeout = 40000

2017-09-14 09:12:31,555INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: ClusterId : 51cb0d44-3b86-4bfd-b195-b19ad9712430

2017-09-14 09:12:31,699INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.MemStoreFlusher: globalMemStoreLimit=194.2 M,globalMemStoreLimitLowMark=184.4 M, maxHeap=485.4 M

2017-09-14 09:12:31,728INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: CompactionChecker runs every 10sec

2017-09-14 09:12:31,839INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled

2017-09-14 09:12:31,839INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled

2017-09-14 09:12:31,854INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: reportForDuty tomaster=master.hadoop,16000,1505351542568 with port=16020,startcode=1505351548182

2017-09-14 09:12:32,153INFO [regionserver/slave1.hadoop/10.9.148.190:16020] hfile.CacheConfig:Allocating LruBlockCache size=194.15 MB, blockSize=64 KB

2017-09-14 09:12:32,175INFO [regionserver/slave1.hadoop/10.9.148.190:16020] hfile.CacheConfig:Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=210048,freeSize=203370992, maxSize=203581040, heapSize=210048, minSize=193401984,minFactor=0.95, multiSize=96700992, multiFactor=0.5, singleSize=48350496,singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false,cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 09:12:32,377INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.WALFactory:Instantiating WALProvider of type classorg.apache.hadoop.hbase.wal.DefaultWALProvider

2017-09-14 09:12:32,470INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.FSHLog: WALconfiguration: blocksize=128 MB, rollsize=121.60 MB,prefix=slave1.hadoop%2C16020%2C1505351548182.default, suffix=,logDir=hdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182,archiveDir=hdfs://master:9000/hbase/oldWALs

2017-09-14 09:12:32,790INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.FSHLog: Slow synccost: 267 ms, current pipeline: []

2017-09-14 09:12:32,792INFO [regionserver/slave1.hadoop/10.9.148.190:16020] wal.FSHLog: New WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470

2017-09-14 09:12:32,846INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metricsevery 5000 milliseconds

2017-09-14 09:12:32,906INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.ReplicationSourceManager: Current list of replicators:[slave1.hadoop,16020,1504077821200, slave2.hadoop,16020,1505351523724,slave1.hadoop,16020,1505351548182, slave3.hadoop,16020,1505351532331] otherRSs: [slave2.hadoop,16020,1505351523724, slave1.hadoop,16020,1505351548182,slave3.hadoop,16020,1505351532331]

2017-09-14 09:12:33,058INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HeapMemoryManager: Starting HeapMemoryTuner chore.

2017-09-14 09:12:33,067INFO  [SplitLogWorker-slave1:16020]regionserver.SplitLogWorker: SplitLogWorker slave1.hadoop,16020,1505351548182starting

2017-09-14 09:12:33,079INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Serving as slave1.hadoop,16020,1505351548182,RpcServer on slave1.hadoop/10.9.148.190:16020, sessionid=0x15e7df074580001

2017-09-14 09:12:33,105INFO [regionserver/slave1.hadoop/10.9.148.190:16020]quotas.RegionServerQuotaManager: Quota support disabled

2017-09-14 09:13:20,702INFO  [ReplicationExecutor-0]replication.ReplicationQueuesZKImpl: Atomically movingslave1.hadoop,16020,1504077821200's WALs to my queue

2017-09-14 09:17:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0

2017-09-14 09:22:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0

2017-09-14 09:27:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0

2017-09-14 09:32:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0

2017-09-14 09:37:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0

2017-09-14 09:42:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0

2017-09-14 09:47:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0

2017-09-14 09:52:32,173 INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0

2017-09-14 09:57:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0

2017-09-14 10:02:06,073INFO [PriorityRpcServer.handler=1,queue=1,port=16020]regionserver.RSRpcServices: OpenCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 10:02:06,352INFO  [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]hfile.CacheConfig: Created cacheConfig for 0:blockCache=LruBlockCache{blockCount=0, currentSize=210048, freeSize=203370992,maxSize=203581040, heapSize=210048, minSize=193401984, minFactor=0.95,multiSize=96700992, multiFactor=0.5, singleSize=48350496, singleFactor=0.25},cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false,cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,prefetchOnOpen=false

2017-09-14 10:02:06,361INFO  [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 10:02:06,388INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1] hfile.CacheConfig:Created cacheConfig for 1: blockCache=LruBlockCache{blockCount=0,currentSize=210048, freeSize=203370992, maxSize=203581040, heapSize=210048,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 10:02:06,389INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 10:02:06,405INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1] hfile.CacheConfig:Created cacheConfig for 2: blockCache=LruBlockCache{blockCount=0,currentSize=210048, freeSize=203370992, maxSize=203581040, heapSize=210048,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false,cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false,cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 10:02:06,406INFO [StoreOpener-3f1b89971cc74cb7bfab4e4b0a3bf2c9-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 10:02:06,474INFO  [RS_OPEN_REGION-slave1:16020-0]regionserver.HRegion: Onlined 3f1b89971cc74cb7bfab4e4b0a3bf2c9; nextsequenceid=2

2017-09-14 10:02:06,509INFO [PostOpenDeployTasks:3f1b89971cc74cb7bfab4e4b0a3bf2c9]regionserver.HRegionServer: Post open deploy tasks for CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 10:02:06,636INFO [PostOpenDeployTasks:3f1b89971cc74cb7bfab4e4b0a3bf2c9]hbase.MetaTableAccessor: Updated rowCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. with server=slave1.hadoop,16020,1505351548182

2017-09-14 10:02:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0

2017-09-14 10:07:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0

2017-09-14 10:12:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0

2017-09-14 10:12:32,995INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470with entries=1, filesize=359 B; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920

2017-09-14 10:12:32,998INFO  [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]wal.FSHLog: Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505351552470

2017-09-14 10:17:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0

2017-09-14 10:22:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0

2017-09-14 10:27:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0

2017-09-14 10:32:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0

2017-09-14 10:37:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0

2017-09-14 10:42:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0

2017-09-14 10:47:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0,cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0

2017-09-14 10:49:36,562INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=124.17 MB

2017-09-14 10:49:42,835INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=85700, memsize=124.2 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/d4ed3074dde745d882ea3f6d896f4dd7

2017-09-14 10:49:43,108INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/d4ed3074dde745d882ea3f6d896f4dd7,entries=494658, sequenceid=85700, filesize=63.9 M

2017-09-14 10:49:43,111INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~124.17 MB/130198992,currentsize=4.25 MB/4459608 for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.in 6549ms, sequenceid=85700, compaction requested=false

2017-09-14 10:52:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=27, hits=0, hitRatio=0, cachingAccesses=27,cachingHits=0, cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0

2017-09-14 10:53:05,853INFO [regionserver/slave1.hadoop/10.9.148.190:16020.leaseChecker]regionserver.RSRpcServices: Scanner 5 lease expired on regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 10:54:48,951INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920with entries=149352, filesize=121.61 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887

2017-09-14 10:56:49,722INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=119.05 MB

2017-09-14 10:56:55,601INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=169579, memsize=119.1 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/f0a1cfa73bd5407fa13fea322063b3a9

2017-09-14 10:56:55,672INFO  [MemStoreFlusher.0]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/f0a1cfa73bd5407fa13fea322063b3a9,entries=477138, sequenceid=169579, filesize=61.3 M

2017-09-14 10:56:55,675INFO  [MemStoreFlusher.0]regionserver.HRegion: Finished memstore flush of ~119.05 MB/124833880,currentsize=9.36 MB/9814568 for regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 5954ms,sequenceid=169579, compaction requested=false

2017-09-14 10:57:22,752INFO  [sync.3] wal.FSHLog: Slow synccost: 480 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 10:57:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=205.13 KB, freeSize=193.95 MB, max=194.15 MB,blockCount=0, accesses=27, hits=0, hitRatio=0, cachingAccesses=27,cachingHits=0, cachingHitsRatio=0,evictions=629, evicted=0, evictedPerRun=0.0

2017-09-14 11:02:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=462.47 KB, freeSize=193.70 MB, max=194.15 MB,blockCount=2, accesses=33, hits=4, hitRatio=12.12%, , cachingAccesses=33,cachingHits=4, cachingHitsRatio=12.12%, evictions=659, evicted=0,evictedPerRun=0.0

2017-09-14 11:03:21,439INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB; 0=114.39MB

2017-09-14 11:03:27,045INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=250053, memsize=114.4 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/3906d45c77aa4d0392cfabe93def9f59

2017-09-14 11:03:27,112INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/3906d45c77aa4d0392cfabe93def9f59,entries=459096, sequenceid=250053, filesize=58.8 M

2017-09-14 11:03:27,114INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~114.39 MB/119943872,currentsize=13.92 MB/14598040 for regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 5675ms, sequenceid=250053,compaction requested=true

2017-09-14 11:03:27,126INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:03:27,127INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. into tmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp,totalSize=183.9 M

2017-09-14 11:03:45,559INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 (all) file(s) in 0 ofCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. into59850935db0840eabce90c676f6701b9(size=183.8 M), total size for store is 183.8M. This selection was in queue for 0sec, and took 18sec to execute.

2017-09-14 11:03:45,632INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.,storeName=0, fileCount=3, fileSize=183.9 M, priority=7, time=517255043298492;duration=18sec

2017-09-14 11:07:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=462.52 KB, freeSize=193.70 MB, max=194.15 MB,blockCount=2, accesses=2957, hits=4, hitRatio=0.14%, , cachingAccesses=35,cachingHits=4, cachingHitsRatio=11.43%, evictions=689, evicted=2,evictedPerRun=0.0029027576092630625

2017-09-14 11:07:53,465INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887with entries=150665, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399

2017-09-14 11:10:00,083 INFO  [MemStoreFlusher.0] regionserver.HRegion:Flushing 2/3 column families, memstore=128.00 MB; 2=18.66 MB; 0=109.34 MB

2017-09-14 11:10:05,464INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=326980, memsize=109.3 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/ac9ee478e810460494cfef143290c341

2017-09-14 11:10:06,214INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=326980, memsize=18.7 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/3a98a0c896774fec8719b2d2b1df2115

2017-09-14 11:10:06,272INFO  [MemStoreFlusher.0]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/ac9ee478e810460494cfef143290c341,entries=435900, sequenceid=326980, filesize=56.6 M

2017-09-14 11:10:06,309INFO  [MemStoreFlusher.0]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/2/3a98a0c896774fec8719b2d2b1df2115,entries=94980, sequenceid=326980, filesize=7.1 M

2017-09-14 11:10:06,319INFO  [MemStoreFlusher.0]regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218448,currentsize=479.64 KB/491152 for regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 6235ms,sequenceid=326980, compaction requested=false

2017-09-14 11:12:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=497.18 KB, freeSize=193.66 MB, max=194.15 MB,blockCount=3, accesses=6952, hits=2933, hitRatio=42.19%, ,cachingAccesses=3516, cachingHits=2933, cachingHitsRatio=83.42%, evictions=719,evicted=2, evictedPerRun=0.0027816412039101124

2017-09-14 11:17:32,174 INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=8622, hits=2942, hitRatio=34.12%, ,cachingAccesses=5186, cachingHits=2942, cachingHitsRatio=56.73%, evictions=749,evicted=2, evictedPerRun=0.0026702270843088627

2017-09-14 11:17:50,905INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=122.52 MB

2017-09-14 11:17:57,353INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=409609, memsize=122.5 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/35e19cbf6a47452b9d485b684db1df29

2017-09-14 11:17:57,408INFO  [MemStoreFlusher.0] regionserver.HStore:Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/35e19cbf6a47452b9d485b684db1df29,entries=468186, sequenceid=409609, filesize=65.8 M

2017-09-14 11:17:57,410INFO  [MemStoreFlusher.0] regionserver.HRegion:Finished memstore flush of ~122.52 MB/128470400, currentsize=5.87 MB/6157376for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in 6505ms,sequenceid=409609, compaction requested=true

2017-09-14 11:17:57,519INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitTransaction: Starting split of regionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:17:58,195INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Running close preflush ofCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:17:58,196INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Flushing 3/3 column families, memstore=6.20 MB

2017-09-14 11:17:58,383INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410060, memsize=692.0 K,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/0ff7b107866e4677bd84e3b2b61dcd01

2017-09-14 11:17:58,645INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410060, memsize=5.5 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/25ccf0ebf5554b18bbbb9a5935dbdabf

2017-09-14 11:17:58,686INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/0ff7b107866e4677bd84e3b2b61dcd01,entries=2496, sequenceid=410060, filesize=387.1 K

2017-09-14 11:17:58,711INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/2/25ccf0ebf5554b18bbbb9a5935dbdabf,entries=27750, sequenceid=410060, filesize=2.2 M

2017-09-14 11:17:58,713INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Finished memstore flush of ~6.20 MB/6497168,currentsize=81.49 KB/83448 for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.in 518ms, sequenceid=410060, compaction requested=true

2017-09-14 11:17:58,713INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Flushing 3/3 column families, memstore=81.49 KB

2017-09-14 11:17:58,770INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410115, memsize=79.1 K,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/a88c6b8fcf734c22bac0e8b57e9e433d

2017-09-14 11:17:58,828INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.DefaultStoreFlusher: Flushed, sequenceid=410115, memsize=2.4 K,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/.tmp/21c20a3e4c324e5da987a6aba80d71d0

2017-09-14 11:17:58,868INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Added hdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/0/a88c6b8fcf734c22bac0e8b57e9e433d,entries=294, sequenceid=410115, filesize=47.5 K

2017-09-14 11:17:58,898INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/3f1b89971cc74cb7bfab4e4b0a3bf2c9/2/21c20a3e4c324e5da987a6aba80d71d0,entries=12, sequenceid=410115, filesize=5.7 K

2017-09-14 11:17:58,900INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: Finished memstore flush of ~81.49 KB/83448, currentsize=0B/0 for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. in187ms, sequenceid=410115, compaction requested=true

2017-09-14 11:17:58,906INFO [StoreCloserThread-CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.-1]regionserver.HStore: Closed 0

2017-09-14 11:17:58,909INFO [StoreCloserThread-CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.-1]regionserver.HStore: Closed 1

2017-09-14 11:17:58,911INFO [StoreCloserThread-CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.-1]regionserver.HStore: Closed 2

2017-09-14 11:17:58,964INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegion: ClosedCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.

2017-09-14 11:17:58,965INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitTransaction: Preparing to split 8 storefiles for region CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9.using 2 threads

2017-09-14 11:17:59,718INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1] hfile.CacheConfig:Created cacheConfig for 0: blockCache=LruBlockCache{blockCount=4,currentSize=640872, freeSize=202940168, maxSize=203581040, heapSize=640872,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:17:59,719INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:17:59,720INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1] hfile.CacheConfig:Created cacheConfig for 0: blockCache=LruBlockCache{blockCount=4,currentSize=640872, freeSize=202940168, maxSize=203581040, heapSize=640872,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:17:59,722INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:17:59,946INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1] hfile.CacheConfig:Created cacheConfig for 1: blockCache=LruBlockCache{blockCount=5,currentSize=772752, freeSize=202808288, maxSize=203581040, heapSize=772752,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:17:59,947INFO [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttlepoint 2684354560; major period 604800000, major jitter 0.500000, min localityto compact 0.000000

2017-09-14 11:18:00,005INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1] hfile.CacheConfig:Created cacheConfig for 1: blockCache=LruBlockCache{blockCount=5,currentSize=772752, freeSize=202808288, maxSize=203581040, heapSize=772752,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:18:00,006INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:18:00,012INFO  [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]hfile.CacheConfig: Created cacheConfig for 2:blockCache=LruBlockCache{blockCount=5, currentSize=772752, freeSize=202808288,maxSize=203581040, heapSize=772752, minSize=193401984, minFactor=0.95,multiSize=96700992, multiFactor=0.5, singleSize=48350496, singleFactor=0.25},cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false,cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,prefetchOnOpen=false

2017-09-14 11:18:00,014INFO  [StoreOpener-c5ddefa1371f4c7d731310cc082cdfb7-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:18:00,018INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1] hfile.CacheConfig:Created cacheConfig for 2: blockCache=LruBlockCache{blockCount=5,currentSize=772752, freeSize=202808288, maxSize=203581040, heapSize=772752,minSize=193401984, minFactor=0.95, multiSize=96700992, multiFactor=0.5,singleSize=48350496, singleFactor=0.25}, cacheDataOnRead=false,cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false

2017-09-14 11:18:00,019INFO [StoreOpener-7726b6f586c6b46ad1334186a21be68b-1]compactions.CompactionConfiguration: size [134217728, 9223372036854775807,9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000;throttle point 2684354560; major period 604800000, major jitter 0.500000, minlocality to compact 0.000000

2017-09-14 11:18:00,047INFO [slave1.hadoop,16020,1505351548182-daughterOpener=c5ddefa1371f4c7d731310cc082cdfb7]regionserver.HRegion: Onlined c5ddefa1371f4c7d731310cc082cdfb7; nextsequenceid=410117

2017-09-14 11:18:00,137INFO [slave1.hadoop,16020,1505351548182-daughterOpener=7726b6f586c6b46ad1334186a21be68b]regionserver.HRegion: Onlined 7726b6f586c6b46ad1334186a21be68b; nextsequenceid=410116

2017-09-14 11:18:00,139INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegionServer: Post open deploy tasks forCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 11:18:00,151INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]hbase.MetaTableAccessor: Updated rowCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.with server=slave1.hadoop,16020,1505351548182

2017-09-14 11:18:00,151INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.HRegionServer: Post open deploy tasks forCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:18:00,154INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 11:18:00,154INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 5 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into tmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp,totalSize=306.7 M

2017-09-14 11:18:00,161INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]hbase.MetaTableAccessor: Updated row CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.with server=slave1.hadoop,16020,1505351548182

2017-09-14 11:18:00,163INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 0 in region CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:18:00,164INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 5 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=306.7 M

2017-09-14 11:18:00,380INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitRequest: Region split, hbase:meta updated, and report tomaster. Parent=CAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9., newregions: CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7..Split took 2sec

2017-09-14 11:18:00,380INFO [regionserver/slave1.hadoop/10.9.148.190:16020-splits-1505359077426]regionserver.SplitRequest: Split transaction journal:

         STARTED at 1505359077432

         PREPARED at 1505359077518

         BEFORE_PRE_SPLIT_HOOK at 1505359077519

         AFTER_PRE_SPLIT_HOOK at 1505359077520

         SET_SPLITTING at 1505359077530

         CREATE_SPLIT_DIR at 1505359078195

         CLOSED_PARENT_REGION at 1505359078965

         OFFLINED_PARENT at 1505359078965

         STARTED_REGION_A_CREATION at 1505359079332

         STARTED_REGION_B_CREATION at 1505359079430

         PONR at 1505359079518

         OPENED_REGION_A at 1505359080139

         OPENED_REGION_B at 1505359080139

         BEFORE_POST_SPLIT_HOOK at 1505359080334

         AFTER_POST_SPLIT_HOOK at 1505359080334

         COMPLETED at 1505359080334

2017-09-14 11:18:00,486WARN [B.defaultRpcServer.handler=0,queue=0,port=16020]regionserver.RSRpcServices: 7 encountered RegionCAR_DATA,,1505354523942.3f1b89971cc74cb7bfab4e4b0a3bf2c9. is not online onslave1.hadoop,16020,1505351548182, closing ...

2017-09-14 11:18:33,335INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 5 (all) file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. into42cb5c4b051d42c58de7f84feb6fdf45(size=151.2 M), total size for store is 151.2M. This selection was in queue for 0sec, and took 33sec to execute.

2017-09-14 11:18:33,337INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=0, fileCount=5, fileSize=306.7 M, priority=5, time=518128079182940;duration=33sec

2017-09-14 11:18:33,365INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 2 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:18:33,366INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 3 file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=9.3 M

2017-09-14 11:18:33,402INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 5 (all) file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into a81a9c60452b4c5497dd8055d0504a90(size=155.5 M), total size for store is155.5 M. This selection was in queue for 0sec, and took 33sec to execute.

2017-09-14 11:18:33,402INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,storeName=0, fileCount=5, fileSize=306.7 M, priority=5, time=518128071001392; duration=33sec

2017-09-14 11:18:34,570INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 3 (all) file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. into 3e87be46c78f40ddb8f0e1ce4c5ff4e9(size=9.3M), total size for store is 9.3 M. This selection was in queue for 0sec, andtook 1sec to execute.

2017-09-14 11:18:34,570INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=2, fileCount=3, fileSize=9.3 M, priority=7, time=518161282561103;duration=1sec

2017-09-14 11:20:36,895INFO  [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]wal.FSHLog: Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399with entries=139255, filesize=121.61 MB; new WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840

2017-09-14 11:20:36,899INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505355152920

2017-09-14 11:20:36,912INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505357688887

2017-09-14 11:22:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=883.21 KB, freeSize=193.29 MB, max=194.15 MB,blockCount=6, accesses=15629, hits=2973, hitRatio=19.02%, ,cachingAccesses=7171, cachingHits=2972, cachingHitsRatio=41.44%, evictions=779,evicted=3, evictedPerRun=0.00385109125636518

2017-09-14 11:27:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=892.95 KB, freeSize=193.28 MB, max=194.15 MB,blockCount=7, accesses=15860, hits=2975, hitRatio=18.76%, ,cachingAccesses=7402, cachingHits=2974, cachingHitsRatio=40.18%, evictions=809,evicted=3, evictedPerRun=0.0037082817871123552

2017-09-14 11:28:28,526INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=184.4 M, Region memstore size=96.4 M

2017-09-14 11:28:28,526INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=96.42 MB

2017-09-14 11:28:33,268INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher:Flushed, sequenceid=470830, memsize=89.3 M, hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/2e35213036664f35a5de111d46452a82

2017-09-14 11:28:33,701INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=470830, memsize=7.1 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/d31a5152e1e34e809607ec73f38c8abd

2017-09-14 11:28:33,768INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/2e35213036664f35a5de111d46452a82,entries=330288, sequenceid=470830, filesize=49.4 M

2017-09-14 11:28:33,819INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/d31a5152e1e34e809607ec73f38c8abd,entries=33966, sequenceid=470830, filesize=3.0 M

2017-09-14 11:28:33,822INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~96.42 MB/101105000,currentsize=220.79 KB/226088 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 5295ms,sequenceid=470830, compaction requested=false

2017-09-14 11:32:31,667INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840with entries=130189, filesize=121.62 MB; new WAL /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596

2017-09-14 11:32:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1.03 MB, freeSize=193.12 MB, max=194.15 MB,blockCount=9, accesses=15864, hits=2977, hitRatio=18.77%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=839,evicted=3, evictedPerRun=0.0035756854340434074

2017-09-14 11:32:58,727INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=128.00 MB

2017-09-14 11:33:05,481INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=488903, memsize=128.0 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/cfb52cbc8dd54dbca1a143cca485d2ec

2017-09-14 11:33:05,533INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/cfb52cbc8dd54dbca1a143cca485d2ec,entries=472692, sequenceid=488903, filesize=70.8 M

2017-09-14 11:33:05,536INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218984,currentsize=245.87 KB/251768 for region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 6808ms, sequenceid=488903, compaction requested=false

2017-09-14 11:33:30,606INFO  [sync.2] wal.FSHLog: Slow synccost: 350 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010, 10.9.148.192:50010]

2017-09-14 11:37:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1.03 MB, freeSize=193.12 MB, max=194.15 MB,blockCount=9, accesses=15864, hits=2977, hitRatio=18.77%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=869,evicted=3, evictedPerRun=0.00345224398188293

2017-09-14 11:40:48,609INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heap pressure.Total Memstore size=184.4 M, Region memstore size=113.8 M

2017-09-14 11:40:48,610INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=113.80 MB

2017-09-14 11:40:54,396INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=541946, memsize=107.6 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/94682411f19e49019f75d8bb8f1e3984

2017-09-14 11:40:54,720INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=541946, memsize=6.2 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/bfeedf84f0e1402c82a28af10303e032

2017-09-14 11:40:54,774INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/94682411f19e49019f75d8bb8f1e3984,entries=397416, sequenceid=541946, filesize=59.5 M

2017-09-14 11:40:54,806INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/bfeedf84f0e1402c82a28af10303e032,entries=29256, sequenceid=541946, filesize=2.7 M

2017-09-14 11:40:54,808INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~113.80 MB/119329776,currentsize=275.40 KB/282008 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 6198ms,sequenceid=541946, compaction requested=true

2017-09-14 11:40:54,817 INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 2 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:40:54,818INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 3 file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=14.9 M

2017-09-14 11:40:56,495INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 3 (all) file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intoc03d152596014599bb0534b02b376bc9(size=15.0 M), total size for store is 15.0 M.This selection was in queue for 0sec, and took 1sec to execute.

2017-09-14 11:40:56,496INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=2, fileCount=3, fileSize=14.9 M, priority=7, time=519502733345301;duration=1sec

2017-09-14 11:42:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1021.62 KB, freeSize=193.15 MB, max=194.15 MB,blockCount=8, accesses=16100, hits=2977, hitRatio=18.49%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=899,evicted=4, evictedPerRun=0.004449388012290001

2017-09-14 11:43:31,904INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596with entries=128535, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833

2017-09-14 11:43:31,908INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505358473399

2017-09-14 11:43:31,921INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505359236840

2017-09-14 11:46:53,797INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=127.2 M

2017-09-14 11:46:53,798INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=127.24 MB

2017-09-14 11:46:57,445INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=184.7 M, Region memstore size=57.3 M

2017-09-14 11:46:57,447INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 3/3 column families, memstore=57.32 MB

2017-09-14 11:47:03,191INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=566552, memsize=127.2 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/c050d9857d0b4a94ac4b092d13e871ec

2017-09-14 11:47:03,258INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=577750, memsize=53.9 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/f6c5cdbf035f41f285829c0f222ae04f

2017-09-14 11:47:03,265INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/c050d9857d0b4a94ac4b092d13e871ec,entries=465870, sequenceid=566552, filesize=70.8 M

2017-09-14 11:47:03,267INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~127.24 MB/133416160,currentsize=195.73 KB/200424 for region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 9469ms, sequenceid=566552, compaction requested=true

2017-09-14 11:47:03,273INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 11:47:03,274INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp,totalSize=297.1 M

2017-09-14 11:47:03,583 INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=577750, memsize=3.4 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/cbf89107dec741fa88678491cd1b8d45

2017-09-14 11:47:03,666INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/f6c5cdbf035f41f285829c0f222ae04f,entries=199266, sequenceid=577750, filesize=29.8 M

2017-09-14 11:47:03,734INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/cbf89107dec741fa88678491cd1b8d45,entries=15528, sequenceid=577750, filesize=1.5 M

2017-09-14 11:47:03,754 INFO  [MemStoreFlusher.0] regionserver.HRegion:Finished memstore flush of ~57.32 MB/60106416, currentsize=112.47 KB/115168 forregion CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 6307ms,sequenceid=577750, compaction requested=true

2017-09-14 11:47:03,802INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:47:03,802INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 4 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=289.8 M

2017-09-14 11:47:11,363INFO  [sync.2] wal.FSHLog: Slow synccost: 100 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:47:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=1021.62 KB, freeSize=193.15 MB, max=194.15 MB,blockCount=8, accesses=21074, hits=2977, hitRatio=14.13%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=929,evicted=4, evictedPerRun=0.004305704962462187

2017-09-14 11:47:42,495INFO  [sync.4] wal.FSHLog: Slow synccost: 730 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:47:43,082INFO  [sync.1] wal.FSHLog: Slow synccost: 478 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:48:00,596INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 (all) file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into 7733f6a209b5440c8064d7ae0e78175f(size=297.0 M), total size for store is297.0 M. This selection was in queue for 0sec, and took 57sec to execute.

2017-09-14 11:48:00,597INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,storeName=0, fileCount=3, fileSize=297.1 M, priority=7, time=519871190566306;duration=57sec

2017-09-14 11:48:00,796INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 4 (all) file(s) in 0 of CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.into afd20ea26d33422da8958863af014add(size=289.7 M), total size for store is289.7 M. This selection was in queue for 0sec, and took 56sec to execute.

2017-09-14 11:48:00,797INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=0, fileCount=4, fileSize=289.8 M, priority=6, time=519871719170783;duration=56sec

2017-09-14 11:48:06,588INFO  [sync.1] wal.FSHLog: Slow synccost: 823 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:52:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25426, hits=2979, hitRatio=11.72%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=959,evicted=8, evictedPerRun=0.008342023007571697

2017-09-14 11:54:46,673INFO  [sync.3] wal.FSHLog: Slow synccost: 1861 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 11:55:40,288INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833with entries=129272, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218

2017-09-14 11:55:40,293INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505359951596

2017-09-14 11:57:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25426, hits=2979, hitRatio=11.72%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%, evictions=989,evicted=8, evictedPerRun=0.008088978938758373

2017-09-14 11:57:48,416INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of region CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=94.7 M

2017-09-14 11:57:48,416INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=94.68 MB

2017-09-14 11:57:53,120INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=637089, memsize=88.8 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/9ec71e99d8fd47ff88b0b2a6f091a779

2017-09-14 11:57:53,445INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=637089, memsize=5.9 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/e7b8472f595243dbbc217367710e9665

2017-09-14 11:57:53,488INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/9ec71e99d8fd47ff88b0b2a6f091a779,entries=329706, sequenceid=637089, filesize=48.9 M

2017-09-14 11:57:53,521INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/e7b8472f595243dbbc217367710e9665,entries=26298, sequenceid=637089, filesize=2.7 M

2017-09-14 11:57:53,524INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~94.68 MB/99279648,currentsize=198.28 KB/203040 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 5107ms, sequenceid=637089,compaction requested=true

2017-09-14 11:57:53,530INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HRegion: Starting compaction on 2 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 11:57:53,531INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Starting compaction of 3 file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=19.2 M

2017-09-14 11:57:55,737INFO [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.HStore: Completed compaction of 3 (all) file(s) in 2 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intobfb1adeac5444446a7f780c555234b58(size=19.2 M), total size for store is 19.2 M.This selection was in queue for 0sec, and took 2sec to execute.

2017-09-14 11:57:55,737INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-longCompactions-1505351551702]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=2, fileCount=3, fileSize=19.2 M, priority=7, time=520521447730712;duration=2sec

2017-09-14 12:02:04,880INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=128.00 MB

2017-09-14 12:02:11,614INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=644924, memsize=128.0 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/f974666048d74d739008b1cd889f49c0

2017-09-14 12:02:11,657INFO  [MemStoreFlusher.1] regionserver.HStore:Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/f974666048d74d739008b1cd889f49c0,entries=470202, sequenceid=644924, filesize=71.1 M

2017-09-14 12:02:11,658INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~128.00 MB/134218592,currentsize=247.68 KB/253624 for regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 6778ms, sequenceid=644924, compaction requested=false

2017-09-14 12:02:32,173INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1019, evicted=8, evictedPerRun=0.007850834168493748

2017-09-14 12:07:15,856INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218with entries=129538, filesize=121.61 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362035790

2017-09-14 12:07:15,860INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833tohdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505360611833

2017-09-14 12:07:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1049, evicted=8, evictedPerRun=0.007626310922205448

2017-09-14 12:09:31,021INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=184.4 M, Region memstore size=111.6 M

2017-09-14 12:09:31,021INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=111.64 MB

2017-09-14 12:09:33,770INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Flush of region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.6 M, Region memstoresize=72.9 M

2017-09-14 12:09:33,771INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 3/3 column families, memstore=72.91 MB

2017-09-14 12:09:39,622INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=706940, memsize=106.8 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/37f4c026cf2f40b0a344c7e3048babd3

2017-09-14 12:09:40,530INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=706940, memsize=4.8 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/4a18796ab17a44cab5886eece6f37f7d

2017-09-14 12:09:40,705INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/37f4c026cf2f40b0a344c7e3048babd3,entries=396984, sequenceid=706940, filesize=58.8 M

2017-09-14 12:09:40,810INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/4a18796ab17a44cab5886eece6f37f7d,entries=22092, sequenceid=706940, filesize=2.1 M

2017-09-14 12:09:40,828INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~111.64 MB/117063960,currentsize=171.20 KB/175304 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 9806ms,sequenceid=706940, compaction requested=true

2017-09-14 12:09:41,121INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=689755, memsize=72.9 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/73ad5e6ad61d42c89b2f09a9182fd46a

2017-09-14 12:09:41,171INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/73ad5e6ad61d42c89b2f09a9182fd46a,entries=268962, sequenceid=689755, filesize=40.3 M

2017-09-14 12:09:41,174INFO  [MemStoreFlusher.0]regionserver.HRegion: Finished memstore flush of ~72.91 MB/76450064,currentsize=110.16 KB/112808 for region CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 7402ms, sequenceid=689755, compaction requested=true

2017-09-14 12:10:06,563INFO  [sync.4] wal.FSHLog: Slow synccost: 174 ms, current pipeline: [10.9.148.190:50010, 10.9.148.192:50010,10.9.148.191:50010]

2017-09-14 12:12:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1079, evicted=8, evictedPerRun=0.007414272520691156

2017-09-14 12:17:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=25730, hits=2979, hitRatio=11.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1109, evicted=8, evictedPerRun=0.007213706150650978

2017-09-14 12:18:24,734INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362035790with entries=129386, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362704656

2017-09-14 12:18:24,738INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Archivinghdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218to hdfs://master:9000/hbase/oldWALs/slave1.hadoop%2C16020%2C1505351548182.default.1505361340218

2017-09-14 12:19:31,849INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=93.2 M

2017-09-14 12:19:31,850INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=93.23 MB

2017-09-14 12:19:36,804INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=747393, memsize=93.2 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/55cf50752f9f4615a2bc8355335fb9e3

2017-09-14 12:19:36,845INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/55cf50752f9f4615a2bc8355335fb9e3,entries=345804, sequenceid=747393, filesize=51.4 M

2017-09-14 12:19:36,848INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~93.23 MB/97763144,currentsize=166.49 KB/170488 for regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.in 4998ms, sequenceid=747393, compaction requested=true

2017-09-14 12:19:36,854INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 12:19:36,855INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp,totalSize=162.8 M

2017-09-14 12:19:52,353INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 file(s) in 0 ofCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.into 1204be22ee6c48a6b1d92025a09f822d(size=162.7 M), total size for store is459.7 M. This selection was in queue for 0sec, and took 15sec to execute.

2017-09-14 12:19:52,353INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,storeName=0, fileCount=3, fileSize=162.8 M, priority=6, time=521824771615235;duration=15sec

2017-09-14 12:19:56,029INFO  [sync.4] wal.FSHLog: Slow synccost: 168 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 12:22:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=28313, hits=2979, hitRatio=10.52%, , cachingAccesses=7406,cachingHits=2976, cachingHitsRatio=40.18%, evictions=1139, evicted=8,evictedPerRun=0.007023705169558525

2017-09-14 12:24:04,205INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 1/3 column families, memstore=128.00 MB;0=122.86 MB

2017-09-14 12:24:10,663INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=786337, memsize=122.9 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/bdc237ad953f43508e96c4ed9747addc

2017-09-14 12:24:10,715INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/bdc237ad953f43508e96c4ed9747addc,entries=452208, sequenceid=786337, filesize=68.1 M

2017-09-14 12:24:10,717INFO  [MemStoreFlusher.1]regionserver.HRegion: Finished memstore flush of ~122.86 MB/128828080,currentsize=5.29 MB/5549560 for regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. in 6512ms,sequenceid=786337, compaction requested=true

2017-09-14 12:24:10,726INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HRegion: Starting compaction on 0 in regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 12:24:10,726INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Starting compaction of 3 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. intotmpdir=hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp,totalSize=175.8 M

2017-09-14 12:24:27,231INFO  [sync.1] wal.FSHLog: Slow synccost: 143 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 12:24:28,437INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.HStore: Completed compaction of 3 file(s) in 0 ofCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. into 2e56407c59f64b1e96640f783dfaa18c(size=175.6M), total size for store is 465.3 M. This selection was in queue for 0sec, andtook 17sec to execute.

2017-09-14 12:24:28,437INFO [regionserver/slave1.hadoop/10.9.148.190:16020-shortCompactions-1505358207118]regionserver.CompactSplitThread: Completed compaction: Request =regionName=CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.,storeName=0, fileCount=3, fileSize=175.8 M, priority=6, time=522098643140405;duration=17sec

2017-09-14 12:27:32,174INFO  [LruBlockCacheStatsExecutor]hfile.LruBlockCache: totalSize=625.85 KB, freeSize=193.54 MB, max=194.15 MB,blockCount=4, accesses=31103, hits=2979, hitRatio=9.58%, ,cachingAccesses=7406, cachingHits=2976, cachingHitsRatio=40.18%,evictions=1169, evicted=8, evictedPerRun=0.006843456067144871

2017-09-14 12:27:47,672INFO  [sync.1] wal.FSHLog: Slow synccost: 3569 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010,10.9.148.192:50010]

2017-09-14 12:30:05,873INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Rolled WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505362704656with entries=129330, filesize=121.62 MB; new WAL/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805

2017-09-14 12:32:09,616INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=184.4 M, Region memstoresize=114.2 M

2017-09-14 12:32:09,616INFO  [MemStoreFlusher.1]regionserver.HRegion: Flushing 3/3 column families, memstore=114.24 MB

2017-09-14 12:32:11,680INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Flush of region CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.due to global heap pressure. Total Memstore size=184.6 M, Region memstoresize=70.3 M

2017-09-14 12:32:11,680INFO  [MemStoreFlusher.0]regionserver.HRegion: Flushing 3/3 column families, memstore=70.28 MB

2017-09-14 12:32:25,470WARN  [ResponseProcessor for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920] hdfs.DFSClient:DFSOutputStream ResponseProcessor exception for block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920

java.io.IOException: Badresponse ERROR for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920 from datanode10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828)

2017-09-14 12:32:25,472WARN  [DataStreamer for file/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920] hdfs.DFSClient:Error Recovery for block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920in pipeline 10.9.148.190:50010, 10.9.148.192:50010, 10.9.148.191:50010: baddatanode 10.9.148.192:50010

2017-09-14 12:32:25,518WARN  [ResponseProcessor for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922] hdfs.DFSClient:DFSOutputStream ResponseProcessor exception for block BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922

java.io.IOException: Badresponse ERROR for block BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922from datanode 10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828)

2017-09-14 12:32:25,523WARN  [DataStreamer for file /hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805block BP-200595776-10.9.148.193-1503627647735:blk_1073744575_22920]hdfs.DFSClient: DataStreamer Exception

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,523WARN  [sync.0] hdfs.DFSClient: Errorwhile syncing

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,524ERROR [sync.0] wal.FSHLog: Error syncing, request close of WAL

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,524INFO  [sync.0] wal.FSHLog: Slow synccost: 10542 ms, current pipeline: [10.9.148.190:50010, 10.9.148.191:50010]

2017-09-14 12:32:25,582WARN  [DataStreamer for file/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/f4717749ff1c40beb6639677f2e107ceblock BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922] hdfs.DFSClient:Error Recovery for block BP-200595776-10.9.148.193-1503627647735:blk_1073744577_22922in pipeline 10.9.148.190:50010, 10.9.148.192:50010, 10.9.148.191:50010: baddatanode 10.9.148.192:50010

2017-09-14 12:32:25,584WARN  [ResponseProcessor for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921] hdfs.DFSClient:DFSOutputStream ResponseProcessor exception for block BP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921

java.io.IOException: Badresponse ERROR for block BP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921from datanode 10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:828)

2017-09-14 12:32:25,585WARN  [DataStreamer for file/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/d12b4c633d2a406ea84a248b4a1a54abblock BP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921]hdfs.DFSClient: Error Recovery for blockBP-200595776-10.9.148.193-1503627647735:blk_1073744576_22921 in pipeline10.9.148.190:50010, 10.9.148.192:50010, 10.9.148.191:50010: bad datanode10.9.148.192:50010

2017-09-14 12:32:25,649INFO  [Thread-647] hdfs.DFSClient:Exception in createBlockOutputStream

java.io.IOException: Badconnect ack with firstBadLink as 10.9.148.192:50010

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2017-09-14 12:32:25,649INFO  [Thread-647] hdfs.DFSClient:Abandoning BP-200595776-10.9.148.193-1503627647735:blk_1073744578_22923

2017-09-14 12:32:25,799INFO  [Thread-647] hdfs.DFSClient:Excluding datanode 10.9.148.192:50010

2017-09-14 12:32:25,863INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Slow sync cost: 316 ms, current pipeline: [10.9.148.190:50010,10.9.148.191:50010]

2017-09-14 12:32:25,864ERROR [sync.1] wal.FSHLog: Error syncing, request close of WAL

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,865ERROR [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] wal.FSHLog:Failed close of WAL writerhdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805,unflushedEntries=1

org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.checkIfSyncFailed(FSHLog.java:1650)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1667)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:858)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:709)

         atorg.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)

         at java.lang.Thread.run(Thread.java:744)

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,865FATAL [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.HRegionServer: ABORTING region server slave1.hadoop,16020,1505351548182:Failed log close in log roller

org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException:hdfs://master:9000/hbase/WALs/slave1.hadoop,16020,1505351548182/slave1.hadoop%2C16020%2C1505351548182.default.1505363405805,unflushedEntries=1

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:907)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:709)

         atorg.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.checkIfSyncFailed(FSHLog.java:1650)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1667)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:858)

         ... 3 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:25,954FATAL [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []

2017-09-14 12:32:26,184INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller] regionserver.HRegionServer:Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" :"java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48602144

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 289356608

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 917,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 1,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

    "TotalCallTime_TimeRangeCount_10000-30000": 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

    "QueueCallTime_TimeRangeCount_0-1": 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

   "QueueCallTime_TimeRangeCount_3-10" : 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

   "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 9,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

    "RequestSize_SizeRangeCount_10-100": 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

    "ResponseSize_SizeRangeCount_10-100": 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

   "ResponseSize_SizeRangeCount_1000-10000" : 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

   "ProcessCallTime_TimeRangeCount_0-1" : 776706,

   "ProcessCallTime_TimeRangeCount_1-3" : 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

   "ProcessCallTime_TimeRangeCount_100-300" : 10,

   "ProcessCallTime_TimeRangeCount_300-1000" : 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.NotServingRegionException" : 6,

    "sentBytes" : 353375774,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277608984,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

   "exceptions.UnknownScannerException" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" :"master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 172728,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 795950,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

    "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

   "ScanNext_TimeRangeCount_600000-inf" : 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:26,283INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.HRegionServer: STOPPED: Failed log close in log roller

2017-09-14 12:32:26,283INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.SplitLogWorker: Sending interrupt to stop the worker thread

2017-09-14 12:32:26,284INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Stopping infoServer

2017-09-14 12:32:26,283INFO [regionserver/slave1.hadoop/10.9.148.190:16020.logRoller]regionserver.LogRoller: LogRoller exiting.

2017-09-14 12:32:26,300INFO  [SplitLogWorker-slave1:16020]regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.

2017-09-14 12:32:26,301INFO  [SplitLogWorker-slave1:16020]regionserver.SplitLogWorker: SplitLogWorker slave1.hadoop,16020,1505351548182exiting

2017-09-14 12:32:26,324INFO [regionserver/slave1.hadoop/10.9.148.190:16020] mortbay.log: [email protected]:16030

2017-09-14 12:32:26,327INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.

2017-09-14 12:32:26,327WARN [B.defaultRpcServer.handler=14,queue=2,port=16020] ipc.RpcServer:(responseTooSlow):{"processingtimems":10793,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","client":"10.9.148.17:2345","param":"region=CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.,for 1 actions and 1st rowkey=LGXCD4D31H01012051498898607","starttimems":1505363534981,"queuetimems":0,"class":"HRegionServer","responsesize":970,"method":"Multi"}

2017-09-14 12:32:28,947INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=826921, memsize=62.5 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/f4717749ff1c40beb6639677f2e107ce

2017-09-14 12:32:29,143INFO  [Thread-651] hdfs.DFSClient:Exception in createBlockOutputStream

java.io.IOException: Badconnect ack with firstBadLink as 10.9.148.192:50010

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1377)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2017-09-14 12:32:29,144INFO  [Thread-651] hdfs.DFSClient:Abandoning BP-200595776-10.9.148.193-1503627647735:blk_1073744580_22927

2017-09-14 12:32:29,163INFO  [Thread-651] hdfs.DFSClient:Excluding datanode 10.9.148.192:50010

2017-09-14 12:32:29,997INFO  [MemStoreFlusher.0]regionserver.DefaultStoreFlusher: Flushed, sequenceid=826921, memsize=7.7 M,hasBloomFilter=true, into tmp filehdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/.tmp/112a6705ac76495b8ae9200ba60d0897

2017-09-14 12:32:30,005INFO  [MemStoreFlusher.1]regionserver.DefaultStoreFlusher: Flushed, sequenceid=817996, memsize=114.2 M,hasBloomFilter=true, into tmp file hdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/.tmp/d12b4c633d2a406ea84a248b4a1a54ab

2017-09-14 12:32:30,052INFO  [MemStoreFlusher.0]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/0/f4717749ff1c40beb6639677f2e107ce,entries=231420, sequenceid=826921, filesize=34.5 M

2017-09-14 12:32:30,056INFO  [MemStoreFlusher.1]regionserver.HStore: Addedhdfs://master:9000/hbase/data/default/CAR_DATA/c5ddefa1371f4c7d731310cc082cdfb7/0/d12b4c633d2a406ea84a248b4a1a54ab,entries=423588, sequenceid=817996, filesize=63.0 M

2017-09-14 12:32:30,057WARN [regionserver/slave1.hadoop/10.9.148.190:16020.append-pool1-t1]wal.FSHLog: Append sequenceId=818069, requesting roll of WAL

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,060FATAL [MemStoreFlusher.1] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region:CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: AppendsequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,062FATAL [MemStoreFlusher.1] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,079INFO  [MemStoreFlusher.1]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48860760

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 231470104

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

    "QueueCallTime_TimeRangeCount_3-10": 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

    "ResponseSize_SizeRangeCount_1000-10000": 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

    "ProcessCallTime_TimeRangeCount_0-1": 776706,

   "ProcessCallTime_TimeRangeCount_1-3" : 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

    "ProcessCallTime_TimeRangeCount_100-300": 10,

   "ProcessCallTime_TimeRangeCount_300-1000" : 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

    "exceptions.NotServingRegionException": 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

    "exceptions.UnknownScannerException": 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" : "master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

   "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

    "ScanNext_TimeRangeCount_600000-inf": 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,082FATAL [MemStoreFlusher.1] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region:CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException:Append sequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,082FATAL [MemStoreFlusher.1] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,084INFO  [MemStoreFlusher.0] regionserver.HStore:Addedhdfs://master:9000/hbase/data/default/CAR_DATA/7726b6f586c6b46ad1334186a21be68b/2/112a6705ac76495b8ae9200ba60d0897,entries=36204, sequenceid=826921, filesize=3.4 M

2017-09-14 12:32:30,086FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region: CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: AppendsequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,086FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,093INFO  [MemStoreFlusher.0]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48864520

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 233210456

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

    "QueueCallTime_TimeRangeCount_3-10": 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

    "ResponseSize_SizeRangeCount_1000-10000": 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

    "ProcessCallTime_TimeRangeCount_0-1": 776706,

   "ProcessCallTime_TimeRangeCount_1-3" : 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

    "ProcessCallTime_TimeRangeCount_100-300": 10,

   "ProcessCallTime_TimeRangeCount_300-1000" : 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

    "exceptions.NotServingRegionException": 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

    "exceptions.UnknownScannerException": 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" : "master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

   "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

    "ScanNext_TimeRangeCount_600000-inf": 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,094INFO  [MemStoreFlusher.1]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" :"java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48862664

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 232753112

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

   "TotalCallTime_TimeRangeCount_300-1000" : 9,

    "TotalCallTime_TimeRangeCount_1000-3000": 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

   "QueueCallTime_TimeRangeCount_3-10" : 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

   "ResponseSize_SizeRangeCount_1000-10000" : 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

   "ProcessCallTime_TimeRangeCount_0-1" : 776706,

    "ProcessCallTime_TimeRangeCount_1-3": 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

   "ProcessCallTime_TimeRangeCount_100-300" : 10,

    "ProcessCallTime_TimeRangeCount_300-1000": 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.NotServingRegionException" : 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

   "exceptions.UnknownScannerException" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" :"master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

    "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

   "ScanNext_TimeRangeCount_600000-inf" : 209,

   "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,096FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1505351548182: Replay of WAL required. Forcing servershutdown

org.apache.hadoop.hbase.DroppedSnapshotException:region: CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2429)

         atorg.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2108)

         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2070)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1961)

         atorg.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1887)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:514)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:219)

         at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75)

         atorg.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:248)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: AppendsequenceId=818069, requesting roll of WAL

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1970)

         atorg.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1814)

         at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1724)

         atcom.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)

         atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         ... 1 more

Caused by:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,097INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Excluding unflushable regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.- trying to find a different region to flush.

2017-09-14 12:32:30,100FATAL [MemStoreFlusher.0] regionserver.HRegionServer: RegionServer abort:loaded coprocessors are: []

2017-09-14 12:32:30,100INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: Flush of regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. due to global heappressure. Total Memstore size=164.7 K, Region memstore size=48.8 K

2017-09-14 12:32:30,109INFO  [MemStoreFlusher.0]regionserver.HRegionServer: Dump of metrics as JSON on abort: {

  "beans" : [ {

    "name" : "java.lang:type=Memory",

    "modelerType" :"sun.management.MemoryImpl",

    "NonHeapMemoryUsage" : {

      "committed" : 140312576,

      "init" : 136773632,

      "max" : 184549376,

      "used" : 48864944

    },

    "ObjectPendingFinalizationCount": 0,

    "Verbose" : false,

    "HeapMemoryUsage" : {

      "committed" : 508952576,

      "init" : 32822848,

      "max" : 508952576,

      "used" : 234635400

    },

    "ObjectName" :"java.lang:type=Memory"

  } ],

  "beans" : [ {

    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",

    "modelerType" :"RegionServer,sub=IPC",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "queueSize" : 0,

    "numCallsInGeneralQueue" : 0,

    "numCallsInReplicationQueue" : 0,

    "numCallsInPriorityQueue" : 0,

    "numOpenConnections" : 2,

    "numActiveHandler" : 0,

    "TotalCallTime_num_ops" :1279658,

    "TotalCallTime_min" : 0,

    "TotalCallTime_max" : 10793,

    "TotalCallTime_mean" :1.5656612938769578,

    "TotalCallTime_median" : 1.0,

    "TotalCallTime_75th_percentile" :2.0,

    "TotalCallTime_90th_percentile" :2.0,

    "TotalCallTime_95th_percentile" :3.0,

    "TotalCallTime_99th_percentile" :4.0,

   "TotalCallTime_TimeRangeCount_0-1" : 752938,

   "TotalCallTime_TimeRangeCount_1-3" : 511814,

   "TotalCallTime_TimeRangeCount_3-10" : 4948,

   "TotalCallTime_TimeRangeCount_10-30" : 8252,

   "TotalCallTime_TimeRangeCount_30-100" : 1684,

   "TotalCallTime_TimeRangeCount_100-300" : 10,

    "TotalCallTime_TimeRangeCount_300-1000": 9,

   "TotalCallTime_TimeRangeCount_1000-3000" : 1,

   "TotalCallTime_TimeRangeCount_3000-10000" : 1,

   "TotalCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.FailedSanityCheckException" : 0,

    "exceptions.RegionMovedException": 0,

    "QueueCallTime_num_ops" :1279658,

    "QueueCallTime_min" : 0,

    "QueueCallTime_max" : 31,

    "QueueCallTime_mean" :0.020814155032047626,

    "QueueCallTime_median" : 0.0,

    "QueueCallTime_75th_percentile" :0.0,

    "QueueCallTime_90th_percentile" :0.0,

    "QueueCallTime_95th_percentile" :0.0,

    "QueueCallTime_99th_percentile" :1.0,

   "QueueCallTime_TimeRangeCount_0-1" : 1279599,

   "QueueCallTime_TimeRangeCount_1-3" : 47,

    "QueueCallTime_TimeRangeCount_3-10": 7,

   "QueueCallTime_TimeRangeCount_10-30" : 3,

   "QueueCallTime_TimeRangeCount_30-100" : 2,

   "exceptions.ScannerResetException" : 0,

    "authenticationFailures" : 0,

    "authorizationFailures" : 0,

    "authenticationFallbacks" : 0,

    "exceptions" : 18,

    "RequestSize_num_ops" : 1279658,

    "RequestSize_min" : 12,

    "RequestSize_max" : 148,

    "RequestSize_mean" :118.57426828105635,

    "RequestSize_median" : 17.0,

    "RequestSize_75th_percentile" :113.0,

    "RequestSize_90th_percentile" :141.0,

    "RequestSize_95th_percentile" :141.0,

    "RequestSize_99th_percentile" :141.0,

   "RequestSize_SizeRangeCount_10-100" : 44840,

   "RequestSize_SizeRangeCount_100-1000" : 1234818,

    "ResponseSize_num_ops" : 1279658,

    "ResponseSize_min" : 2,

    "ResponseSize_max" : 9418,

    "ResponseSize_mean" :10.767688710577357,

    "ResponseSize_median" : 54.0,

    "ResponseSize_75th_percentile" :54.0,

    "ResponseSize_90th_percentile" :54.0,

    "ResponseSize_95th_percentile" :54.0,

    "ResponseSize_99th_percentile" :54.0,

   "ResponseSize_SizeRangeCount_0-10" : 1234824,

   "ResponseSize_SizeRangeCount_10-100" : 44612,

   "ResponseSize_SizeRangeCount_100-1000" : 8,

   "ResponseSize_SizeRangeCount_1000-10000" : 214,

    "authenticationSuccesses" : 0,

    "authorizationSuccesses" : 10,

    "ProcessCallTime_num_ops" :1279658,

    "ProcessCallTime_min" : 0,

    "ProcessCallTime_max" : 10793,

    "ProcessCallTime_mean" :1.54484713884491,

    "ProcessCallTime_median" : 1.0,

    "ProcessCallTime_75th_percentile": 2.0,

    "ProcessCallTime_90th_percentile": 2.0,

    "ProcessCallTime_95th_percentile": 3.0,

    "ProcessCallTime_99th_percentile": 5.529999999999973,

   "ProcessCallTime_TimeRangeCount_0-1" : 776706,

    "ProcessCallTime_TimeRangeCount_1-3": 488343,

   "ProcessCallTime_TimeRangeCount_3-10" : 4662,

   "ProcessCallTime_TimeRangeCount_10-30" : 8249,

   "ProcessCallTime_TimeRangeCount_30-100" : 1676,

   "ProcessCallTime_TimeRangeCount_100-300" : 10,

    "ProcessCallTime_TimeRangeCount_300-1000": 9,

   "ProcessCallTime_TimeRangeCount_1000-3000" : 1,

   "ProcessCallTime_TimeRangeCount_3000-10000" : 1,

   "ProcessCallTime_TimeRangeCount_10000-30000" : 1,

   "exceptions.NotServingRegionException" : 6,

    "sentBytes" : 353384194,

   "exceptions.RegionTooBusyException" : 0,

    "receivedBytes" : 1277614353,

   "exceptions.OutOfOrderScannerNextException" : 0,

   "exceptions.multiResponseTooLarge" : 0,

   "exceptions.UnknownScannerException" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Replication",

    "modelerType" :"RegionServer,sub=Replication",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "sink.appliedOps" : 0,

    "sink.appliedBatches" : 0,

    "sink.ageOfLastAppliedOp" : 0

  } ],

  "beans" : [ {

    "name" :"Hadoop:service=HBase,name=RegionServer,sub=Server",

    "modelerType" :"RegionServer,sub=Server",

    "tag.zookeeperQuorum" :"master:2181,slave1:2181,slave2:2181,slave3:2181",

    "tag.serverName" :"slave1.hadoop,16020,1505351548182",

    "tag.clusterId" :"51cb0d44-3b86-4bfd-b195-b19ad9712430",

    "tag.Context" :"regionserver",

    "tag.Hostname" :"slave1.hadoop",

    "regionCount" : 2,

    "storeCount" : 6,

    "hlogFileCount" : 3,

    "hlogFileSize" : 255049009,

    "storeFileCount" : 6,

    "memStoreSize" : 171208,

    "storeFileSize" : 992339241,

    "regionServerStartTime" :1505351548182,

    "totalRequestCount" : 1279661,

    "readRequestCount" : 815700,

    "writeRequestCount" : 1234808,

    "checkMutateFailedCount" : 0,

    "checkMutatePassedCount" : 0,

    "storeFileIndexSize" : 33784,

    "staticIndexSize" : 854328,

    "staticBloomSize" : 1449984,

    "mutationsWithoutWALCount" : 0,

    "mutationsWithoutWALSize" : 0,

    "percentFilesLocal" : 100.0,

   "percentFilesLocalSecondaryRegions" : 0.0,

    "splitQueueLength" : 0,

    "compactionQueueLength" : 0,

    "flushQueueLength" : 0,

    "blockCacheFreeSize" : 202676584,

    "blockCacheCount" : 6,

    "blockCacheSize" : 904456,

    "blockCacheHitCount" : 4551,

    "blockCacheHitCountPrimary" :4548,

    "blockCacheMissCount" : 28393,

    "blockCacheMissCountPrimary" :28392,

    "blockCacheEvictionCount" : 8,

    "blockCacheEvictionCountPrimary": 7,

    "blockCacheCountHitPercent" :13.814350962638855,

    "blockCacheExpressHitPercent" :50.64588189125061,

    "blockCacheFailedInsertionCount": 0,

    "updatesBlockedTime" : 0,

    "flushedCellsCount" : 4256538,

    "compactedCellsCount" : 9341430,

    "majorCompactedCellsCount" : 0,

    "flushedCellsSize" : 1201779824,

    "compactedCellsSize" :1296131819,

    "majorCompactedCellsSize" : 0,

    "blockedRequestCount" : 0,

    "splitSuccessCount" : 1,

    "splitRequestCount" : 1,

    "Append_num_ops" : 0,

    "Append_min" : 0,

    "Append_max" : 0,

    "Append_mean" : 0.0,

    "Append_median" : 0.0,

    "Append_75th_percentile" : 0.0,

    "Append_90th_percentile" : 0.0,

    "Append_95th_percentile" : 0.0,

    "Append_99th_percentile" : 0.0,

    "Delete_num_ops" : 0,

    "Delete_min" : 0,

    "Delete_max" : 0,

    "Delete_mean" : 0.0,

    "Delete_median" : 0.0,

    "Delete_75th_percentile" : 0.0,

    "Delete_90th_percentile" : 0.0,

    "Delete_95th_percentile" : 0.0,

    "Delete_99th_percentile" : 0.0,

    "Mutate_num_ops" : 1234811,

    "Mutate_min" : 0,

    "Mutate_max" : 10793,

    "Mutate_mean" :1.5622698534431585,

    "Mutate_median" : 1.0,

    "Mutate_75th_percentile" : 2.0,

    "Mutate_90th_percentile" : 2.0,

    "Mutate_95th_percentile" : 2.0,

    "Mutate_99th_percentile" : 27.0,

    "Mutate_TimeRangeCount_0-1" :753523,

    "Mutate_TimeRangeCount_1-3" :469035,

    "Mutate_TimeRangeCount_3-10" :2655,

    "Mutate_TimeRangeCount_10-30" :7953,

    "Mutate_TimeRangeCount_30-100" :1630,

    "Mutate_TimeRangeCount_100-300" :4,

    "Mutate_TimeRangeCount_300-1000": 8,

    "Mutate_TimeRangeCount_1000-3000": 1,

   "Mutate_TimeRangeCount_3000-10000" : 1,

   "Mutate_TimeRangeCount_10000-30000" : 1,

    "ScanNext_num_ops" : 44822,

    "ScanNext_min" : 0,

    "ScanNext_max" : 2098048,

    "ScanNext_mean" :11041.560126723483,

    "ScanNext_median" : 1280.0,

    "ScanNext_75th_percentile" :1280.0,

    "ScanNext_90th_percentile" :1280.0,

    "ScanNext_95th_percentile" :1280.0,

    "ScanNext_99th_percentile" :1280.0,

    "ScanNext_TimeRangeCount_0-1" :4,

   "ScanNext_TimeRangeCount_1000-3000" : 44608,

   "ScanNext_TimeRangeCount_300000-600000" : 1,

   "ScanNext_TimeRangeCount_600000-inf" : 209,

    "slowDeleteCount" : 0,

    "slowIncrementCount" : 0,

    "FlushTime_num_ops" : 16,

    "FlushTime_min" : 4999,

    "FlushTime_max" : 9808,

    "FlushTime_mean" : 6602.75,

    "FlushTime_median" : 6411.0,

    "FlushTime_75th_percentile" :6802.0,

    "FlushTime_90th_percentile" :9572.1,

    "FlushTime_95th_percentile" :9808.0,

    "FlushTime_99th_percentile" :9808.0,

   "FlushTime_TimeRangeCount_3000-10000" : 16,

    "Get_num_ops" : 6,

    "Get_min" : 1,

    "Get_max" : 27,

    "Get_mean" : 14.5,

    "Get_median" : 15.0,

    "Get_75th_percentile" : 21.75,

    "Get_90th_percentile" : 27.0,

    "Get_95th_percentile" : 27.0,

    "Get_99th_percentile" : 27.0,

    "Get_TimeRangeCount_0-1" : 1,

    "Get_TimeRangeCount_3-10" : 1,

    "Get_TimeRangeCount_10-30" : 4,

    "Replay_num_ops" : 0,

    "Replay_min" : 0,

    "Replay_max" : 0,

    "Replay_mean" : 0.0,

    "Replay_median" : 0.0,

    "Replay_75th_percentile" : 0.0,

    "Replay_90th_percentile" : 0.0,

    "Replay_95th_percentile" : 0.0,

    "Replay_99th_percentile" : 0.0,

    "slowGetCount" : 0,

    "slowAppendCount" : 0,

    "slowPutCount" : 3,

    "SplitTime_num_ops" : 1,

    "SplitTime_min" : 2954,

    "SplitTime_max" : 2954,

    "SplitTime_mean" : 2954.0,

    "SplitTime_median" : 2954.0,

    "SplitTime_75th_percentile" :2954.0,

    "SplitTime_90th_percentile" :2954.0,

    "SplitTime_95th_percentile" :2954.0,

    "SplitTime_99th_percentile" :2954.0,

   "SplitTime_TimeRangeCount_1000-3000" : 1,

    "Increment_num_ops" : 0,

    "Increment_min" : 0,

    "Increment_max" : 0,

    "Increment_mean" : 0.0,

    "Increment_median" : 0.0,

    "Increment_75th_percentile" :0.0,

    "Increment_90th_percentile" :0.0,

    "Increment_95th_percentile" :0.0,

    "Increment_99th_percentile" : 0.0

  } ]

}

2017-09-14 12:32:30,112INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: Excluding unflushable regionCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b. - trying to find adifferent region to flush.

2017-09-14 12:32:30,112INFO [regionserver/slave1.hadoop/10.9.148.190:16020]snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManagerabruptly.

2017-09-14 12:32:30,113INFO [regionserver/slave1.hadoop/10.9.148.190:16020]flush.RegionServerFlushTableProcedureManager: Stopping region server flushprocedure manager abruptly.

2017-09-14 12:32:30,113INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: aborting server slave1.hadoop,16020,1505351548182

2017-09-14 12:32:30,113INFO [regionserver/slave1.hadoop/10.9.148.190:16020] client.ConnectionManager$HConnectionImplementation:Closing zookeeper sessionid=0x25e7df015ce0002

2017-09-14 12:32:30,113INFO  [MemStoreFlusher.1]regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting

2017-09-14 12:32:30,114INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher:Flush of regionCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.due to global heap pressure. Total Memstore size=164.7 K, Region memstoresize=115.9 K

2017-09-14 12:32:30,114INFO  [MemStoreFlusher.0]regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting

2017-09-14 12:32:30,129INFO [regionserver/slave1.hadoop/10.9.148.190:16020] zookeeper.ZooKeeper:Session: 0x25e7df015ce0002 closed

2017-09-14 12:32:30,129INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-EventThread]zookeeper.ClientCnxn: EventThread shut down

2017-09-14 12:32:30,129INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Waiting on 2 regions to close

2017-09-14 12:32:30,138INFO  [StoreCloserThread-CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.-1]regionserver.HStore: Closed 0

2017-09-14 12:32:30,139INFO [StoreCloserThread-CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.-1]regionserver.HStore: Closed 1

2017-09-14 12:32:30,139INFO [StoreCloserThread-CAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.-1]regionserver.HStore: Closed 2

2017-09-14 12:32:30,143ERROR [RS_CLOSE_REGION-slave1:16020-0] regionserver.HRegion: Memstore size is118664

2017-09-14 12:32:30,146INFO  [RS_CLOSE_REGION-slave1:16020-0]regionserver.HRegion: ClosedCAR_DATA,LGXCD4C39H01254321498841392,1505359077518.c5ddefa1371f4c7d731310cc082cdfb7.

2017-09-14 12:32:30,148 INFO [StoreCloserThread-CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.-1]regionserver.HStore: Closed 0

2017-09-14 12:32:30,148INFO [StoreCloserThread-CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.-1]regionserver.HStore: Closed 1

2017-09-14 12:32:30,151INFO [StoreCloserThread-CAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.-1]regionserver.HStore: Closed 2

2017-09-14 12:32:30,152ERROR [RS_CLOSE_REGION-slave1:16020-1] regionserver.HRegion: Memstore size is49952

2017-09-14 12:32:30,154INFO  [RS_CLOSE_REGION-slave1:16020-1]regionserver.HRegion: ClosedCAR_DATA,,1505359077518.7726b6f586c6b46ad1334186a21be68b.

2017-09-14 12:32:30,336INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: stopping server slave1.hadoop,16020,1505351548182;all regions closed.

2017-09-14 12:32:30,336WARN [regionserver/slave1.hadoop/10.9.148.190:16020] wal.ProtobufLogWriter:Failed to write trailer, non-fatal, continuing...

java.io.IOException:Failed to replace a bad datanode on the existing pipeline due to no more gooddatanodes being available to try. (Nodes: current=[10.9.148.190:50010,10.9.148.191:50010], original=[10.9.148.190:50010, 10.9.148.191:50010]). Thecurrent failed datanode replacement policy is DEFAULT, and a client mayconfigure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'in its configuration.

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

2017-09-14 12:32:30,348ERROR [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: Shutdown / close of WAL failed:java.io.IOException: Failed to replace a bad datanode on the existing pipelinedue to no more good datanodes being available to try. (Nodes:current=[10.9.148.190:50010, 10.9.148.191:50010], original=[10.9.148.190:50010,10.9.148.191:50010]). The current failed datanode replacement policy isDEFAULT, and a client may configure this via'dfs.client.block.write.replace-datanode-on-failure.policy' in itsconfiguration.

2017-09-14 12:32:30,358INFO [regionserver/slave1.hadoop/10.9.148.190:16020] regionserver.Leases:regionserver/slave1.hadoop/10.9.148.190:16020 closing leases

2017-09-14 12:32:30,358INFO [regionserver/slave1.hadoop/10.9.148.190:16020] regionserver.Leases:regionserver/slave1.hadoop/10.9.148.190:16020 closed leases

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020] hbase.ChoreService:Chore service for: slave1.hadoop,16020,1505351548182 had [[ScheduledChore:Name: slave1.hadoop,16020,1505351548182-MemstoreFlusherChore Period: 10000Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for regionslave1.hadoop,16020,1505351548182 Period: 120000 Unit: MILLISECONDS]] onshutdown

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Split Thread to finish...

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Merge Thread to finish...

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Large Compaction Thread tofinish...

2017-09-14 12:32:30,359INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.CompactSplitThread: Waiting for Small Compaction Thread tofinish...

2017-09-14 12:32:30,409INFO [regionserver/slave1.hadoop/10.9.148.190:16020] ipc.RpcServer: Stoppingserver on 16020

2017-09-14 12:32:30,410INFO  [RpcServer.listener,port=16020]ipc.RpcServer: RpcServer.listener,port=16020: stopping

2017-09-14 12:32:30,413INFO  [RpcServer.responder]ipc.RpcServer: RpcServer.responder: stopped

2017-09-14 12:32:30,413INFO  [RpcServer.responder]ipc.RpcServer: RpcServer.responder: stopping

2017-09-14 12:32:30,462INFO [regionserver/slave1.hadoop/10.9.148.190:16020] zookeeper.ZooKeeper:Session: 0x15e7df074580001 closed

2017-09-14 12:32:30,462INFO [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: stopping server slave1.hadoop,16020,1505351548182;zookeeper connection closed.

2017-09-14 12:32:30,462INFO  [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: regionserver/slave1.hadoop/10.9.148.190:16020exiting

2017-09-14 12:32:30,462ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting

java.lang.RuntimeException:HRegionServer Aborted

         atorg.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)

         atorg.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)

         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

         atorg.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2697)

2017-09-14 12:32:30,463INFO  [main-EventThread] zookeeper.ClientCnxn:EventThread shut down

2017-09-14 12:32:30,511INFO  [Thread-4]regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true;fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@7ed32b58

2017-09-14 12:32:30,511INFO  [Thread-4]regionserver.ShutdownHook: Starting fs shutdown hook thread.

2017-09-14 12:32:30,530INFO  [Thread-4]regionserver.ShutdownHook: Shutdown hook finished.

 

 

 

 

 

 

 

 

 

 

 

 

重新配置hadoop、HBase、重新批量put

错误日志：

2017-08-30 15:09:14,445WARN  [DataStreamer for file/hbase/WALs/slave1.hadoop,16020,1504076498637/slave1.hadoop%2C16020%2C1504076498637..meta.1504076507048.metablock BP-200595776-10.9.148.193-1503627647735:blk_1073742324_20651]hdfs.DFSClient: DataStreamer Exception

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):Client (=DFSClient_NONMAPREDUCE_-1470364669_1) is not the lease owner(=DFSClient_NONMAPREDUCE_-908925855_1:/hbase/WALs/slave1.hadoop,16020,1504076498637-splitting/slave1.hadoop%2C16020%2C1504076498637..meta.1504076507048.meta(inode 16723) [Lease.  Holder:DFSClient_NONMAPREDUCE_-1470364669_1, pending creates: 1]

         atorg.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2624)

         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2539)

         atorg.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:863)

         atorg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:530)

         atorg.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)

         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)

         at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)

         at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)

         at java.security.AccessController.doPrivileged(NativeMethod)

         at javax.security.auth.Subject.doAs(Subject.java:415)

         atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)

         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

 

         at org.apache.hadoop.ipc.Client.call(Client.java:1411)

         at org.apache.hadoop.ipc.Client.call(Client.java:1364)

         atorg.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

         at com.sun.proxy.$Proxy18.getAdditionalDatanode(UnknownSource)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)

         atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:606)

         atorg.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

         atorg.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

         at com.sun.proxy.$Proxy18.getAdditionalDatanode(UnknownSource)

         atorg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:393)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)

         atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

         atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:606)

         atorg.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)

         at com.sun.proxy.$Proxy19.getAdditionalDatanode(UnknownSource)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1028)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)

         atorg.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)

         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)

解决办法，调整xcievers参数

默认是4096，改为8192

vi/home/dwhftp/opt/hadoop/conf/hdfs-site.xml

dfs.datanode.max.xcievers

8192

dfs.datanode.max.xcievers参数说明：

一个 Hadoop HDFS Datanode 有一个同时处理文件的上限. 这个参数叫 xcievers (Hadoop的作者把这个单词拼错了). 在你加载之前，先确认下你有没有配置这个文件conf/hdfs-site.xml里面的xceivers参数，至少要有4096:

dfs.datanode.max.xcievers

4096

Hadoop设置更改（vihdfs-site.xml）：增加绿色部分

 

   dfs.namenode.secondary.http-address

   master.hadoop:50090

 

 

   dfs.namenode.name.dir

    file:/home/softs/hadoop/tmp/dfs/name

 

 

   dfs.datanode.data.dir

   file:/home/softs/hadoop/tmp/dfs/data

 

 

    dfs.replication

    1

 

 

    dfs.permissions

    false

 

 

   dfs.datanode.handler.count

    3

 

 

   dfs.client.socket-timeout

    60000

 

 

    dfs.support.append

    true

 

 

   dfs.datanode.max.xcievers

    8192

 

错误日志：

2017-09-01 09:39:30,827WARN [regionserver/slave1.hadoop/10.9.148.190:16020] util.Sleeper: We slept30263ms instead of 3000ms, this is likely due to a long garbage collectingpause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired

2017-09-01 09:39:30,826INFO [main-SendThread(slave2.hadoop:2181)] zookeeper.ClientCnxn: Clientsession timed out, have not heard from server in 36049ms for sessionid0x15e3200dc5c0000, closing socket connection and attempting reconnect

2017-09-01 09:39:30,826INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave3.hadoop:2181)]zookeeper.ClientCnxn: Client session timed out, have not heard from server in30345ms for sessionid 0x15e3200dc5c0003, closing socket connection andattempting reconnect

2017-09-01 09:39:46,287INFO [main-SendThread(master.hadoop:2181)] zookeeper.ClientCnxn: Openingsocket connection to server master.hadoop/10.9.148.193:2181. Will not attemptto authenticate using SASL (unknown error)

2017-09-01 09:39:46,287INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Opening socket connection to serverslave2.hadoop/10.9.148.191:2181. Will not attempt to authenticate using SASL(unknown error)

2017-09-01 09:39:46,287INFO [main-SendThread(master.hadoop:2181)] zookeeper.ClientCnxn: Socketconnection established to master.hadoop/10.9.148.193:2181, initiating session

2017-09-01 09:39:46,287INFO  [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Socket connection established toslave2.hadoop/10.9.148.191:2181, initiating session

2017-09-01 09:39:47,772INFO [main-SendThread(master.hadoop:2181)] zookeeper.ClientCnxn: Unable toreconnect to ZooKeeper service, session 0x15e3200dc5c0000 has expired, closingsocket connection

2017-09-01 09:39:50,548INFO [regionserver/slave1.hadoop/10.9.148.190:16020-SendThread(slave2.hadoop:2181)]zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session0x15e3200dc5c0003 has expired, closing socket connection

2017-09-01 09:40:02,048FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1504077821200: regionserver:16020-0x15e3200dc5c0000,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbaseregionserver:16020-0x15e3200dc5c0000 received expired from ZooKeeper, aborting

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-01 09:40:02,048WARN [regionserver/slave1.hadoop/10.9.148.190:16020-EventThread]client.ConnectionManager$HConnectionImplementation: This client just lost it'ssession with ZooKeeper, closing it. It will be recreated next time someoneneeds it

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-01 09:40:02,048FATAL [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1504077821200: org.apache.hadoop.hbase.YouAreDeadException:Server REPORT rejected; currently processing slave1.hadoop,16020,1504077821200as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         at java.lang.Thread.run(Thread.java:744)

 

org.apache.hadoop.hbase.YouAreDeadException:org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currentlyprocessing slave1.hadoop,16020,1504077821200 as dead server

         at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         atorg.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         at java.lang.Thread.run(Thread.java:744)

 

         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)

         atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

         atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

         atjava.lang.reflect.Constructor.newInstance(Constructor.java:526)

         atorg.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

         atorg.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

         atorg.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:962)

         at java.lang.Thread.run(Thread.java:744)

Caused by:org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException):org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currentlyprocessing slave1.hadoop,16020,1504077821200 as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         at java.lang.Thread.run(Thread.java:744)

 

         atorg.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)

         atorg.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)

         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:8994)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1151)

         ... 2 more

描述及解决方法：

为什么regionserver 和Zookeeper的session expired? 可能的原因有

1. 网络不好。

2. Java full GC， 这会block所有的线程。如果时间比较长，也会导致session expired.

怎么办？

1. 将Zookeeper的timeout时间加长。

2. 配置“hbase.regionserver.restart.on.zk.expire” 为true。 这样子，遇到ZooKeepersession expired ， regionserver将选择 restart 而不是 abort

具体的配置是，在hbase-site.xml中加入

       

                hbase.rootdir

               hdfs://master:9000/hbase

       

       

               hbase.cluster.distributed

                true

       

       

                hbase.master

               master

       

       

               hbase.zookeeper.quorum

               master,slave1,slave2,slave3

       

       

               hbase.zookeeper.property.dataDir

               /home/softs/zookeeper/data

       

       

               zookeeper.session.timeout

                90000

                ZooKeepersession timeout.

                        HBase passes this tothe zk quorum as suggested maximum time for a

                        session.  See http://hadoop.apache.org/zooke ... sions

                        “The client sends a requested timeout, theserver responds with the

                        timeout that it cangive the client. The current implementation

                        requires that thetimeout be a minimum of 2 times the tickTime

                        (as set in the serverconfiguration) and a maximum of 20 times

                        the tickTime.” Set thezk ticktime with hbase.zookeeper.property.tickime.

                        In milliseconds.

               

       

       

               hbase.regionserver.restart.on.zk.expire

                true

               

                        Zookeeper sessionexpired will force regionserver exit.

                        Enable this will makethe regionserver restart.

               

       

HBase Java Full GC错误：

错误日志：

2017-09-0109:40:02,048 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING regionserver slave1.hadoop,16020,1504077821200: regionserver:16020-0x15e3200dc5c0000,quorum=master:2181,slave1:2181,slave2:2181,slave3:2181, baseZNode=/hbaseregionserver:16020-0x15e3200dc5c0000 received expired from ZooKeeper, aborting

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-0109:40:02,048 WARN [regionserver/slave1.hadoop/10.9.148.190:16020-EventThread] client.ConnectionManager$HConnectionImplementation:This client just lost it's session with ZooKeeper, closing it. It will berecreated next time someone needs it

org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:702)

         atorg.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:613)

         atorg.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

         atorg.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

2017-09-0109:40:02,048 FATAL [regionserver/slave1.hadoop/10.9.148.190:16020]regionserver.HRegionServer: ABORTING region serverslave1.hadoop,16020,1504077821200: org.apache.hadoop.hbase.YouAreDeadException:Server REPORT rejected; currently processing slave1.hadoop,16020,1504077821200as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         atorg.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         atjava.lang.Thread.run(Thread.java:744)

 

org.apache.hadoop.hbase.YouAreDeadException:org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processingslave1.hadoop,16020,1504077821200 as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         atjava.lang.Thread.run(Thread.java:744)

 

         atsun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

         atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

         atjava.lang.reflect.Constructor.newInstance(Constructor.java:526)

         atorg.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)

         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)

         atorg.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:330)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1153)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:962)

         atjava.lang.Thread.run(Thread.java:744)

Causedby:org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException):org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currentlyprocessing slave1.hadoop,16020,1504077821200 as dead server

         atorg.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:434)

         atorg.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:339)

         atorg.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:339)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8617)

         atorg.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)

         atorg.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)

         atorg.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)

         atjava.lang.Thread.run(Thread.java:744)

 

         atorg.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1267)

         atorg.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)

         atorg.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)

         atorg.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerReport(RegionServerStatusProtos.java:8994)

         atorg.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1151)

         ... 2 more

解决方法：

看起来是GC时间太长了，导致ZK 心跳 连接超时，最终通知这个server下线。

HBase集群总是掉的原因：

1.网络不好，导致RegionServer 和 HMaster通信超时，RegionServer被认为已经挂掉，从而退出集群 --网络问题，无法从软件方面解决，关于通信超时的设置下面做个简单介绍

2.Java full GC ，这个过程会block所有的线程，如果此事件过长，导致Session expired 会话过期，导致退出集群--下文会阐述

3.各节点时间不一致，导致RegionServer 退出。--hbase.master.maxclockskew 增大容忍度，默认是30s，但不要太大，毕竟时间不一致是不正常现象，可将所有节点和同一服务器时间做同步，也可以和时间服务器同步。（这个在安装zookeeper的时候，就讲过，时间最好一致，要么安装集群时钟管理软件，要么手动设置，超过30sHBase集群就难以启动了）

修改HBase（./conf/hbase-env.sh）:

#exportHBASE_OPTS="-XX:+UseConcMarkSweepGC"

exportHBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC-XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -Xmn256m"

解释：

具体参数含义如下：

-XX:+UseCompressedOops

压缩指针，解决内存占用

-XX:+UseParNewGC

设置年轻代为并行收集

-XX:+UseConcMarkSweepG

 使用CMS内存收集

-XX:+CMSClassUnloadingEnabled

相对于并行收集器，CMS收集器默认不会对永久代进行垃圾回收。如果希望对永久代进行垃圾回收，可用设置标志-XX:+CMSClassUnloadingEnabled。 在早期JVM版本中，要求设置额外的标志-XX:+CMSPermGenSweepingEnabled。注意，即使没有设置这个标志，一旦永久代耗尽空 间也会尝试进行垃圾回收，但是收集不会是并行的，而再一次进行Full GC。

-XX:+UseCMSCompactAtFullCollection

使用并发收集器时,开启对年老代的压缩.

-XX:CMSFullGCsBeforeCompaction

由于并发收集器不对内存空间进行压缩,整理,所以运行一段时间以后会产生”碎片”,使得运行效率降低.此值设置运行多少次GC以后对内存空间进行压缩,整理.

-XX:+CMSParallelRemarkEnabled

降低标记停顿

-XX:CMSInitiatingOccupancyFraction=75

使用cms作为垃圾回收使用75％后开始CMS收集

-XX:SoftRefLRUPolicyMSPerMB

每兆堆空闲空间中SoftReference的存活时间

 

 

4.推测原因1--compaction、split过于频繁：

 

              由于配置里面hfile最大文件大小设置为1G，所以compaction、split比较频繁，资源消耗比较大，导致gc暂停时间过长，出现写hdfs错误，导致regionsever挂掉

                 推测原因2-- hdfs压力过大，datanode超负荷：

 

               由于集群运行各种任务，hdfs读写压力大，datanode的负载比较高，导致regionserver写hdfs异常，宕机

         解决办法：1、hfile大小改为100G，禁止系统自己做major compaction。2、给datanode多一些内存，调整rpc线程数。

因为本人搭建的集群的slave性能很差，磁盘、内存都不是很大，所以在这里并没有设置HFile的大小

 

参考网址：http://wenda.chinahadoop.cn/question/4872