the leap second may crash servers

1.what is leap second

设在巴黎的国际地球自转服务组织宣布，由于地球自转放缓的原因，今年全球将迎来史上第26次闰秒，时间在格林尼治时间6月30日午夜，科学家们将再度给钟表额外增加1秒钟时间，届时原子钟将会在23时59分59秒后停留一秒钟，让原子钟等上地球自转变慢的步伐。

 
  that is:

 

     case 'world time' is slower than atom time

  how long a second in scale in this case
 

2.impacts

  yesterday,the leap second was come here at 8'o clock moring,one related thing were occured:

  high system cpu,user cpu for all processes lied in that node which run a 'ntp' service

   and the kern.log

Jul  1 07:59:59 xxx-01 kernel: [14897543.632649] Clock: inserting leap second 23:59:60 UTC

 

3.solutions

 a.restore time from hardware

  first,my fellow checked that but found nothing!then a idea come to his mind:leap second!so he used this command to restore time from haredware

hwclock -s

   then the processes in that node all restored to be normal.

 b.restart processes

 c.add a jump second (experiment,like google)

 d.stop ntp at that time

 

4.root cause

  a dead lock occurs when get hardware clock,see [2]

 

The kernel attempts to print a message when a leap second is inserted or removed.  This can cause kernel versions prior to 2.6.29 to hang, due to a deadlock on xtime_lock.  See http://lkml.org/lkml/2009/1/2/373 for a trace and explanation.

   note:in my case,it only occured in os linux-3.2 which run with 'ntp' service,but for other os linux-3.5 which run  with 'ntp' also lucky happened nothing!

 

ref:

闰秒变更对服务器等的影响

[2]Leap second message can hang the kernel

全球迎第26次闰秒 7月1日闰秒怎么来的？