Hbase的regionServer无法启动报ClockOutOfSyncException解决方法

阅读更多

2014-04-11 02:41:20,142 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop3,60020,1397155279126 has been rejected; Reported time is too far out of sync with master.  Time difference of 48018ms > max allowed of 30000ms
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
 at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2051)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:736)
 at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop3,60020,1397155279126 has been rejected; Reported time is too far out of sync with master.  Time difference of 48018ms > max allowed of 30000ms
 at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:235)
 at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:169)
 at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1047)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)

 

1、方案1
在hbase-site.xml添加配置

        hbase.master.maxclockskew
        180000
        Time difference of regionserver from master
 

2、方案2
写一个同步时间的shell脚本

#!/bin/bash
echo "start"
for ((i=2;i<=3;i++))
do
#ssh hadoop$i 'date -s "2014-04-13 01:11:00"; clock -w'
ssh hadoop$i 'clock -w'
done
echo "complete"