Linux下进程级令牌桶的一种实现方法

1、前言

最近看了一些QoS的实现原理，大致分为漏桶模型、令牌桶模型。特别是令牌桶，什么双桶双速三色概念都比较复杂还得慢慢理解，但对于单桶单速，就开始怎么进行原型实现了。
需求上看的话，多线程版本可以直接拿一个变量实现，但是多进程版本的话，就得考虑进程间问题了。

2、原理

Linux进程间共享的变量，于是想到了POSIX信号量semaphore，提供了整型计数器，调用者可进行获取、释放的PV操作，常用于标识资源是否可用，若信号量变为0，进行P接口调用将进入阻塞。

A  semaphore is an integer whose value is never allowed to fall below zero.  Two operations can be performed on semaphores: increment the
semaphore value by one (sem_post(3)); and decrement the semaphore value by one (sem_wait(3)).  If the value of a semaphore  is  currently
zero, then a sem_wait(3) operation will block until the value becomes greater than zero.

2.1 有名信号量

有名信号量支持进程间使用，在 /dev/shm 下创建信号量文件，

Named semaphores
	A named semaphore is identified by a name of the form /somename; that is, a null-terminated string of up to NAME_MAX-4 (i.e., 251)
	characters consisting of an initial slash, followed by one or more characters, none of which are slashes.  Two processes can oper‐
	ate on the same named semaphore by passing the same name to sem_open(3).
	
	The sem_open(3) function creates a new named semaphore or opens an existing named semaphore.  After the semaphore has been opened,
	it can be operated on using sem_post(3) and sem_wait(3).  When a process has finished using the semaphore, it can use sem_close(3)
	to  close  the  semaphore.   When  all  processes  have  finished  using  the  semaphore,  it can be removed from the system using
	sem_unlink(3).

2.2 匿名信号量

匿名信号量是基于内存的信号量，常用于多线程间、或亲缘关系的多进程场景（如fork出来的父子进程）

Unnamed semaphores (memory-based semaphores)
	An unnamed semaphore does not have a name.  Instead the semaphore is placed in a region of memory that is shared between  multiple
	threads  (a thread-shared semaphore) or processes (a process-shared semaphore).  A thread-shared semaphore is placed in an area of
	memory shared between the threads of a process, for example, a global variable.  A process-shared semaphore must be  placed  in  a
	shared  memory  region (e.g., a System V shared memory segment created using shmget(2), or a POSIX shared memory object built cre‐
	ated using shm_open(3)).
	
	Before being used, an unnamed semaphore must be initialized using sem_init(3).  It can then be operated on using  sem_post(3)  and
	sem_wait(3).  When the semaphore is no longer required, and before the memory in which it is located is deallocated, the semaphore
	should be destroyed using sem_destroy(3).

3. 实践

3.1 编程

主要思路是实现一个令牌生产者，每秒以恒定的速率生产令牌；
然后消费者每次就必须获取到令牌了才能do_something，令牌不足时就进行阻塞等待；
生产者代码，使用sem_open 以读写形式打开信号量，
循环中使用 sem_getvalue sem_post 进行生产：

void token_generate()
{
    int nums = 0;
    struct timeb tb = {0};

    sem_t *shm = sem_open(_NAME, O_RDWR | O_CREAT, 0755, 1);
    assert(shm);

    while (1) {
#if 0
        usleep(1000 * 1000 / _NUMS_PER_SECOND);
#else
        struct timeval tv = {0};
        tv.tv_usec = 1000 * 1000 / _NUMS_PER_SECOND;
        select(0, NULL, NULL, NULL, &tv);
#endif
        sem_getvalue(shm, &nums);

        if (nums < _NUMS_PER_SECOND) {
            sem_post(shm);
        }

        ftime(&tb);
        printf("[%ld.%03d] Tokens: %d\n", tb.time, tb.millitm, nums);
    }

    sem_close(shm);
}

消费者处理：sem_open以只读方式打开信号量，然后每次循环干活之前，都先sem_wait获取令牌，简单粗暴；

void token_consumer()
{
    sem_t *shm = sem_open(_NAME, O_RDONLY, 0755, 1);
    assert(shm);

    int cnt = 0;
    time_t tm = 0;
    struct timeb tb = {0};

    while (1) {
        sem_wait(shm);
        
        ftime(&tb);
        printf("[%ld.%03d] Getting: %d\n", tb.time, tb.millitm, cnt);

        // TODO something

        /* Logger every second */
        cnt++;
        if (tm != time(NULL)) {
            printf("[%ld.%03d] Getting: %d, per-second\n", tb.time, tb.millitm, cnt);
            cnt = 0;
            tm = time(NULL);
        }
    }

    sem_close(shm);
}

最后设置一下头文件，main入口

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#define _NAME "token.shm"
#define _NUMS_PER_SECOND 10

void token_generate();
void token_consumer();

int main(int argc, char *argv[])
{
    if (argv[1]) {
        token_generate();
    }
    else {
        token_consumer();
    }
    exit(EXIT_SUCCESS);
}

3.2 运行

编译过程需要加入 -lrt -lpthread

生产者运行，打印如下：

[1594478212.692] Tokens: 0
[1594478212.792] Tokens: 1
[1594478212.893] Tokens: 2
[1594478212.993] Tokens: 3
[1594478213.093] Tokens: 4
[1594478213.193] Tokens: 5
[1594478213.293] Tokens: 6
[1594478213.393] Tokens: 7
[1594478213.494] Tokens: 8
[1594478213.594] Tokens: 9
[1594478213.694] Tokens: 10
[1594478213.794] Tokens: 10
[1594478213.894] Tokens: 10
[1594478213.995] Tokens: 10
[1594478214.095] Tokens: 10
[1594478214.195] Tokens: 10
[1594478214.295] Tokens: 10

通过lsof -p ${pid}，能够明显看见打开了文件：/dev/shm/sem.token.shm

调试信息上看，每秒生成10个令牌，一秒内令牌生成完毕，进入满状态

.ps: 之前代码特意打印了时间戳，代码中特意也实验了一下usleep和select来控制休眠时间，结果是select更准确一些

开启一个消费者，结果为正常获取，每秒10个

[1594478933.171] Getting: 1
[1594478933.272] Getting: 2
[1594478933.372] Getting: 3
[1594478933.472] Getting: 4
[1594478933.572] Getting: 5
[1594478933.672] Getting: 6
[1594478933.772] Getting: 7
[1594478933.873] Getting: 8
[1594478933.973] Getting: 9
[1594478934.073] Getting: 10
[1594478934.073] Getting: 10, per-second
[1594478934.173] Getting: 1
[1594478934.273] Getting: 2
[1594478934.373] Getting: 3
[1594478934.474] Getting: 4
[1594478934.574] Getting: 5
[1594478934.674] Getting: 6
[1594478934.774] Getting: 7
[1594478934.874] Getting: 8
[1594478934.974] Getting: 9
[1594478935.075] Getting: 10
[1594478935.075] Getting: 10, per-second

开启两个消费者，可见速度变为 5个每秒

[1594478979.360] Getting: 1
[1594478979.560] Getting: 2
[1594478979.760] Getting: 3
[1594478979.961] Getting: 4
[1594478980.161] Getting: 5
[1594478980.161] Getting: 5, per-second
[1594478980.361] Getting: 1
[1594478980.562] Getting: 2
[1594478980.762] Getting: 3
[1594478980.962] Getting: 4
[1594478981.163] Getting: 5
[1594478981.163] Getting: 5, per-second
[1594478981.363] Getting: 1
[1594478981.564] Getting: 2
[1594478981.764] Getting: 3
[1594478981.964] Getting: 4
[1594478982.164] Getting: 5
[1594478982.164] Getting: 5, per-second

4.总结

通过上述手册的翻阅，原型编码以及运行实践，Linux下进程级令牌桶，可以非常方便地通过POSIX共享内存、信号量来进行实现，是可以满足基础需求的。

期间还正好顺手测试了延迟函数，发现select确实比usleep更精准一些。

最后压测了一下，发现还得注意一个生产频率的问题，比较合适的是在每秒1000次以下，CPU不会有明显的开销。

参考文章：
[1] https://linux.die.net/man/7/sem_overview