C++高性能服务框架revolver:同时支持100万个的定时事件的定时器
在高性能的服务器程序当中，定时器是必不可少的部件，而且定时器的效率是直接影响到服务的性能。在众多的开源项目中，定时器设计都有各有各的方法，例如ACE和libEvent都采用了最小堆的算法实现，还有其他的开源项目采用平衡二叉树来做定时的器管理算法。不管是最小堆还是平衡二叉树，其定时器扫描都是O(1),但定时器插入和删除都是O(logN)的复杂度。在定时事件少的情况下，这种算法是足够的，如果超过上百万的定时事件，效率会成为瓶颈。所以revolver在定时器的实现上并没有使用通用的平衡二叉树和最小堆，而是采用了轮转HASH算法来做定时器管理。
什么是轮转HASH算法？轮转HASH是通过4个时间轮的转动来触发定时事件，就像时钟的秒针轮、分针轮、时针轮之间的关系一样。如图：
每个轮的有256个刻度，4个轮刚好是一个uint32_t整型数。最小轮的一刻度表示一个reactor的event loop时间（5ms）
1.定时器的扫描
1）获取当前的系统时间cur_ts，判断与上次扫描时刻prev_ts之间差距超过最小刻度，计算需要转动最小刻度的数量
scale = (cur_ts - prev_ts) / loop_delay;
2) 进行第一轮的轮转，转动scale个刻度，所有转动的刻度中的定时事件全部进行触发。如果转动后没有超过最大刻度256，轮转结束。
3）如果超过最大刻度256，那么从它的父轮(第二轮)上轮转1刻度。将父轮上轮转的刻度所有的定时事件全部按时间映射到子轮上。相当于重复2步。这是个迭代过程，如果父轮到了256，那么继续轮转父轮的父轮。直到到达第四轮转动为止。这和秒钟走一圈就是分针走一个刻度的原理是一样的。
代码如下：
template<class HANDLER, class FUNCTOR, class LOCK>
uint32_t CTimerQueue_T<HANDLER, FUNCTOR, LOCK>::expire()
{
BASE_GUARD_RETURN(LOCK, cf_mon, mutex_, 0);
uint32_t ret = SELECT_DELAY; //默认20MS
CBaseTimeValue cur_timer = CBaseTimeValue::get_time_value();
if(cur_timer > prev_time_)
{
uint32_t scale = static_cast<uint32_t>((cur_timer.msec() - prev_time_.msec()) / SELECT_DELAY);
if(scale > 0)
{
ret = revolver(scale);
prev_time_ = cur_timer;
}
}
return ret;
}
template<class HANDLER, class FUNCTOR, class LOCK>
uint32_t CTimerQueue_T<HANDLER, FUNCTOR, LOCK>::revolver(uint32_t scale)
{
//std::cout << "pos, first = " << rings_[0].get_pos() << ", second = " << rings_[1].get_pos()
// << ", third = " << rings_[2].get_pos() << ", fourth = " << rings_[3].get_pos() <<std::endl;
uint32_t ret = SELECT_DELAY;
uint8_t index = 0;
uint32_t rewind_scale = scale;
while(rewind_scale > 0)
{
index = 0;
if(rings_[index].cycle(rewind_scale, this)) //扫描第一轮
{
index ++;
uint32_t sc = 1;
while(rings_[index].cycle(sc, this))//扫描下一轮，刻度只往前推进1格
{
sc = 1;
index ++;
if(index >= RINGS_SIZE)
{
start_time_ = CBaseTimeValue::get_time_value();
break;
}
}
}
}
return ret;
}
2.定时事件的插入
1）首先计算需触发的时刻与定时器轮当前时刻的距离
d = (cur_ts - start_ts + delay) / loop_delay;
2)分别计算在4个轮的位置
first = (uint8_t)(timeout_stamp_ / FIRST_ROUND);
second = (uint8_t)((timeout_stamp_ % FIRST_ROUND) / SECOND_ROUND);
third = (uint8_t)((timeout_stamp_ % SECOND_ROUND) / THIRD_ROUND);
fourth = (uint8_t) (timeout_stamp_ % THIRD_ROUND);
second = (uint8_t)((timeout_stamp_ % FIRST_ROUND) / SECOND_ROUND);
third = (uint8_t)((timeout_stamp_ % SECOND_ROUND) / THIRD_ROUND);
fourth = (uint8_t) (timeout_stamp_ % THIRD_ROUND);
3）通过计算得到的为止，保存到对应的轮刻度上，插入就完毕了。例子，如果计算的first = 0, second = 2, third = 30, fouth = 1，就会保存到第3轮的第2刻度上。
代码如下：
template<class HANDLER, class FUNCTOR, class LOCK>
uint32_t CTimerQueue_T<HANDLER, FUNCTOR, LOCK>::schedule(HANDLER handler, const void *act, uint32_t delay, uint32_t interval)
{
BASE_GUARD_RETURN(LOCK, cf_mon, mutex_, 0);
BaseTimerNode_T<HANDLER>* timer_obj = node_pool_.pop_obj();
if(timer_obj != NULL)
{
uint32_t timer_id = get_free_node();
CBaseTimeValue cur_timer = CBaseTimeValue::get_time_value();
//计算距离
uint64_t distance = delay / SELECT_DELAY; //直接以当前时间作为坐标，相差一个扫描间隔20MS
if(cur_timer > start_time_)
distance = (cur_timer.msec() - start_time_.msec() + delay) / SELECT_DELAY;
distance = distance % (UNINT32_MAX);
timer_obj->set(handler, act, (uint32_t)(core_max(distance, 1)), interval, timer_id);
heap_[timer_id] = timer_obj;
used_num_ ++;
//插入事件
insert_node(timer_obj);
upcall_functor().registration(timer_obj->get_handler(), timer_id);
return timer_id;
}
return 0;
}
template<class HANDLER, class FUNCTOR, class LOCK>
void CTimerQueue_T<HANDLER, FUNCTOR, LOCK>::insert_node(BaseTimerNode_T<HANDLER>* node)
{
uint32_t timer_id = node->get_timer_id();
uint8_t poss[RINGS_SIZE] = {0};
//获取位置
node->get_revolver_pos(poss[RINGS_SIZE - 1], poss[RINGS_SIZE - 2], poss[RINGS_SIZE - 3], poss[RINGS_SIZE - 4]);
uint8_t index = RINGS_SIZE - 1;
//进行插入
while(!rings_[index].add_element(poss[index], timer_id))
{
if(index == 0)
break ;
index --;
}
}
3.定时事件的删除
删除和插入计算差不多步骤
1）首先计算需要删除定时事件触发的时刻与定时器轮当前时刻的距离
d = (cur_ts - start_ts + delay) / loop_delay;
2)分别计算在4个轮的位置
first = (uint8_t)(timeout_stamp_ / FIRST_ROUND);
second = (uint8_t)((timeout_stamp_ % FIRST_ROUND) / SECOND_ROUND);
third = (uint8_t)((timeout_stamp_ % SECOND_ROUND) / THIRD_ROUND);
fourth = (uint8_t) (timeout_stamp_ % THIRD_ROUND);
3)根据位置坐标找到对应的轮位置删除轮上的定时事件
second = (uint8_t)((timeout_stamp_ % FIRST_ROUND) / SECOND_ROUND);
third = (uint8_t)((timeout_stamp_ % SECOND_ROUND) / THIRD_ROUND);
fourth = (uint8_t) (timeout_stamp_ % THIRD_ROUND);
代码：
template<class HANDLER, class FUNCTOR, class LOCK>
void CTimerQueue_T<HANDLER, FUNCTOR, LOCK>::cancel_timer(uint32_t timer_id, const void **act)
{
BASE_GUARD(LOCK, cf_mon, mutex_);
if(timer_id < heap_size_ && heap_[timer_id] != NULL)
{
//查找对应的定时事件内容
BaseTimerNode_T<HANDLER>* timer_obj = heap_[timer_id];
//删除轮上的定时事件
delete_node(timer_obj);
heap_[timer_id] = NULL;
if(used_num_ > 0)
used_num_ --;
freeTimers_.push_back(timer_id);
*act = timer_obj->get_act();
upcall_functor().cancel_timer(timer_obj->get_handler(), timer_id);
node_pool_.push_obj(timer_obj);
}
}
template<class HANDLER, class FUNCTOR, class LOCK>
void CTimerQueue_T<HANDLER, FUNCTOR, LOCK>::delete_node(BaseTimerNode_T<HANDLER>* node)
{
uint32_t timer_id = node->get_timer_id();
uint8_t poss[RINGS_SIZE] = {0};
node->get_revolver_pos(poss[RINGS_SIZE - 1], poss[RINGS_SIZE - 2], poss[RINGS_SIZE - 3], poss[RINGS_SIZE - 4]);
//删除掉对应的定时事件
for(uint8_t index = 0; index < RINGS_SIZE; index ++) //在每个轮上进行删除
{
rings_[index].delete_element(poss[index], timer_id);
}
}
4.测试
在revovler的test工程中的main()函数中将test_timer_queue注释去掉，就可以进行测试。以下是test_timer_queue的代码:
void test_timer_queue()
{
srand(time(NULL));
CTimerFunctor functor;
TIMEQUEUE timer_queue(&functor);
CTest_Event_Handler handler;
handler.tq_ = &timer_queue;
CBaseTimeValue begin_timer = CBaseTimeValue::get_time_value();
for(int i = 0; i < 1000000; i ++)
{
insert_timer(&handler, (rand() % 240) * 1000, timer_queue);
}
CBaseTimeValue stop_timer = CBaseTimeValue::get_time_value();
stop_timer = stop_timer - begin_timer;
std::cout << "insert 1000000 timer, delay = " << stop_timer.msec() << " MS" << std::endl;
g_ts = stop_timer.get_time_value().msec();
#if _DEBUG
//timer_queue.set_ring_id();
#endif
std::cout << "exprie ......" << std::endl;
while(1)
{
uint32_t ms = timer_queue.expire();
usleep((1000));
}
}这个函数可以测试插入100万个定时事件的耗时多少，在100个定时事件在定时器管理的时候，CPU和内存都可以进行相对应的监控和查看。我在window 7下面的release版本的信息如下:
以下是100万个定时事件在处理过程中的CPU和内存占用图。
5.总结
定时器的实现从效率和功能上实现都达到了最初设想的效率，但是内存使用上稍微过高（120M），如果用C的代码实现，对内存做严格的控制和内存管理，应该会好很多，以后优化的工作应该重点在这。如果需要完整的代码,请到： https://github.com/yuanrongxi/revolver 下载。