snowflaker 问题
snowflaker 问题
之前由于使用了tiDB，需要实现一个snow flaker 算法。找了一下轮子没找到。看着也简单，于是自己写了一个。然后发现了有问题。
python RLock
python 的RLockRLock
就是一个线程获取了锁之后可以再次获得锁。比如：
如下主线程可以在for循环中取得锁。
import threading
class _RLock:
def __init__(self):
self._lock = threading.Lock()
self._owner = 0
self._count = 0
def acquired(self, block=True, timeout=-1):
me = threading.get_ident()
if self._owner == me:
self._count += 1
print(f"acquired count = {self._count}")
return 1
rc = self._lock.acquire(block, timeout)
if rc:
self._count = 1
print(f"acquired count = {self._count}")
self._owner = me
return rc
def release(self):
me = threading.get_ident()
if self._owner != me:
raise RuntimeError("can not release the un-acquired the lock")
self._count = count = self._count - 1
print(f"release count = {self._count}")
if not count:
self._owner = None
self._lock.release()
def worker(lock):
print("====非可重入锁所在线程释放RLock内部锁====")
pid = threading.current_thread()
print(f"current pid = {pid}")
try:
lock.release()
except Exception as e:
print(str(e))
print("====非可重入锁所在线程释放RLock内部锁失败====")
print("waiting......")
lock.acquired()
print(f"RLock所在线程已释放所有锁，worker线程获取RLock内部锁成功")
if __name__ == '__main__':
print("生成一个R锁对象")
rlock = RLock()
print("主线程获取RLock")
z = rlock.acquired()
print(f"current pid = {threading.current_thread()}")
print("创建并启动一个worker线程")
t = threading.Thread(target=worker, args=(rlock,))
t.start()
print("主线程开始获取RLock锁")
for i in range(5):
rlock.acquired()
print("主线程释放RLock锁")
for i in range(6):
rlock.release()
print("主线程释放完所有的RLock锁")
结果：
生成一个R锁对象
主线程获取RLock
acquired count = 1
current pid = <_MainThread(MainThread, started 140736157979584)>
创建并启动一个worker线程
====非可重入锁所在线程释放RLock内部锁====
主线程开始获取RLock锁
acquired count = 2
acquired count = 3
acquired count = 4
acquired count = 5
acquired count = 6
主线程释放RLock锁
release count = 5
release count = 4
release count = 3
release count = 2
current pid = <Thread(Thread-1, started 123145446993920)>
can not release the un-acquired the lock
====非可重入锁所在线程释放RLock内部锁失败====
waiting......
release count = 1
release count = 0
主线程释放完所有的RLock锁
acquired count = 1
RLock所在线程已释放所有锁，worker线程获取RLock内部锁成功
在之前的snowflaker实现中使用了读写锁并且还递归了一下：
def get_id(self):
with self.lock:
now = get_timestamp()
# 时间回退 异常
if now < self.last_timestamp:
raise Exception('SnowFlake now:%s small than last_timestamp:%s' % now, self.last_timestamp)
time_stamp_count = now - self.start
# 同一秒内 sequence 累加, 否则清0
if now == self.last_timestamp:
self.sequence += 1
else:
self.sequence = 0
self.last_timestamp = now
# 已经累计到最大值 需要休眠50ms 再次调用 最大休眠0.5 秒 否则不合理
if self.sequence == self.sequence_max_value:
for i in xrange(self.sleep_max_count):
time.sleep(self.sleep_time)
current_timestamp = get_timestamp()
if current_timestamp > self.last_timestamp:
return self.get_id()
raise Exception('SnowFlake sleep time not match next second :%s last_timestamp:%s' %
(self.sleep_max_count * self.sleep_time, self.last_timestamp))
current_id = time_stamp_count << self.time_stamp_left_shift | \
self.work_id << self.work_node_id_shift | self.sequence
return current_id
忘记了项目中使用的是gevent模式，并不是线程模式。在一个时间片切换到了其他协程可能有隐晦的问题。最好改为写锁，编程更加轻松。。
第二个问题在于
if self.sequence == self.sequence_max_value:
for i in xrange(self.sleep_max_count):
time.sleep(self.sleep_time)
current_timestamp = get_timestamp()
if current_timestamp > self.last_timestamp:
return self.get_id()
raise Exception('SnowFlake sleep time not match next second :%s last_timestamp:%s' %
(self.sleep_max_count * self.sleep_time, self.last_timestamp))
当 self.sequence 达到最大值之后我们应该先判断再sleep. 并且这里备注了只sleep 500ms。需要改成累加为1s。否则仍然容易出现错误。
并且这里有个bug，前面已经累加了sequence。其他协程进入的时候sequence > max_value.并且sequence 会截断最后十位所以就会导致重复。
新版本：
def get_id(self):
with self.lock:
now = get_timestamp()
# 时间回退 异常
if now < self.last_timestamp:
raise Exception('SnowFlake now:%s small than last_timestamp:%s' % now, self.last_timestamp)
time_stamp_count = now - self.start
# 同一秒内 sequence 累加, 否则清0
if now == self.last_timestamp:
self.sequence += 1
else:
self.sequence = 0
self.last_timestamp = now
# 已经累计到最大值 每次休眠50ms 直到下一秒 最大休眠1 秒
if self.sequence >= self.sequence_max_value:
# wait for next second
if get_timestamp() == now:
for i in range(self.sleep_max_count):
time.sleep(self.sleep_time)
if get_timestamp() > now:
break
current_id = time_stamp_count << self.time_stamp_left_shift | \
self.work_id << self.work_node_id_shift | self.sequence
return current_id