2018-10-20 20:38:01,799 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.HRegionServer: STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server linux02,16020,1540039076246 has been rejected; Reported time is too far out of sync with master. Time difference of 447463ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:410)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:276)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:363)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
2018-10-20 20:38:01,799 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.HRegionServer: Stopping infoServer
2018-10-20 20:38:01,810 INFO [regionserver/linux02/192.168.3.21:16020] mortbay.log: Stopped [email protected]:16030
2018-10-20 20:38:01,814 INFO [regionserver/linux02/192.168.3.21:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2018-10-20 20:38:01,814 INFO [regionserver/linux02/192.168.3.21:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2018-10-20 20:38:01,815 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.HRegionServer: aborting server linux02,16020,1540039076246
2018-10-20 20:38:01,815 INFO [regionserver/linux02/192.168.3.21:16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x266917799a70004
2018-10-20 20:38:01,819 INFO [regionserver/linux02/192.168.3.21:16020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-10-20 20:38:01,819 INFO [regionserver/linux02/192.168.3.21:16020] zookeeper.ZooKeeper: Session: 0x266917799a70004 closed
2018-10-20 20:38:01,822 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.HRegionServer: stopping server linux02,16020,1540039076246; all regions closed.
2018-10-20 20:38:01,824 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.Leases: regionserver/linux02/192.168.3.21:16020 closing leases
2018-10-20 20:38:01,824 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.Leases: regionserver/linux02/192.168.3.21:16020 closed leases
2018-10-20 20:38:01,826 INFO [regionserver/linux02/192.168.3.21:16020] hbase.ChoreService: Chore service for: linux02,16020,1540039076246 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown
2018-10-20 20:38:01,827 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2018-10-20 20:38:01,827 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2018-10-20 20:38:01,827 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2018-10-20 20:38:01,827 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2018-10-20 20:38:01,827 INFO [regionserver/linux02/192.168.3.21:16020] ipc.RpcServer: Stopping server on 16020
2018-10-20 20:38:01,828 INFO [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: stopping
2018-10-20 20:38:01,830 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2018-10-20 20:38:01,830 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2018-10-20 20:38:01,877 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-10-20 20:38:01,877 INFO [regionserver/linux02/192.168.3.21:16020] zookeeper.ZooKeeper: Session: 0x166917e6c350006 closed
2018-10-20 20:38:01,877 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.HRegionServer: stopping server linux02,16020,1540039076246; zookeeper connection closed.
2018-10-20 20:38:01,877 INFO [regionserver/linux02/192.168.3.21:16020] regionserver.HRegionServer: regionserver/linux02/192.168.3.21:16020 exiting
2018-10-20 20:38:01,877 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2721)
2018-10-20 20:38:01,882 INFO [Thread-4] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@7ca33c24
2018-10-20 20:38:01,883 INFO [Thread-4] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2018-10-20 20:38:01,886 INFO [Thread-4] regionserver.ShutdownHook: Shutdown hook finished.。。。
造成的原因:时间同步问题,hbase集群必须要进行时间同步,否则响应时间超过30000ms时,节点会自动挂掉。
网上查了好久,说什么的都有,有添加配置文件的操作,试过了没用,可以先查看一下自己集群的时间是否同步,如果不同步那就一定是时间同步问题。
操作步骤如下:
先查看自己的集群时间是否同步,使用如下命令:
$ date
配置时间服务器
1.检查时区
$ date -R
+0800东八区区时
2.检查软件包
如果ntp工具不存在,则需要使用yum安装
$ sudo rpm -qa | grep ntp
如果不存在则安装
$ sudo yum -y install ntp
3.先以网络时间为标准,纠正集群的时间服务器的时间
$ sudo ntpdate pool.ntp.org
最后重启hbase即可。。。 ^ - ^