内存池 c语言实现

目录

  • 前言
  • 为什么会用到内存池?
  • 设计内存池
    • 小块内存
    • 大块内存
  • 分配与管理
  • 代码实现
    • 宏定义
    • 结构体定义
    • 线程池的创建
    • 内存池的销毁
    • 内存申请
      • block(页)申请,扩容
      • 大块内存申请
    • 内存释放
    • 内存池重置
  • 代码测试

前言

本文旨在纯c实现内存池。根据服务器一个客户端对应一个连接,一个连接对应一个内存池的业务场景实现内存池。

为什么会用到内存池?

  首先我们来理解下内存碎片,内存碎片分为:内部碎片和外部碎片。
  内部碎片就是已经被分配出去(能明确指出属于哪个进程)却不能被利用的内存空间;进程占有这块存储块时,系统无法利用它。直到进程释放它,或进程结束时,系统才有可能利用这个存储块。
  外部碎片指的是还没有被分配出去(不属于任何进程),但由于太小了无法分配给申请内存空间的新进程的内存空闲区域。外部碎片是出于任何已分配区域或页面外部的空闲存储块,这些存储块的总和可以满足当前申请的长度要求,但是由于它们的地址不连续或其他原因,使得系统无法满足当前申请。

  比如我们申请了512B 64B 128B 64B和128B堆空间。
内存池 c语言实现_第1张图片

内存池 c语言实现_第2张图片
  我们释放了中间的两个64B,再申请128B。会发现这空闲的总和是有128B的,但是没有办法用上。就形成了外部碎片。
  在不停运转的服务器中,会出现大量的内存碎片,到时候明明有空间却申请内存失败。
  内存池可以协调之间的关系,资源利用最大化,并且可以很好的避免内存泄漏的问题。内存池预先分配一大块内存来做一个内存池,业务中的内存分配和释放都由这个内存池来管理,内存池内的内存不足时其内部会自己申请。所以内存碎片的问题就交由内存池的算法来优化,而内存泄漏的问题只需要遵守内存池提供的api,就非常容易避免内存泄漏了。
  即使出现了内存泄漏问题,排查的思路也会变得清晰。1.先检查是不是内存池的问题;2.如果不是内存池的问题,就检查是不是第三方库的内存泄漏。

设计内存池

  因为本文主要业务场景一个客户端对应一个连接,一个连接对应一个内存池,每个连接都会申请相应的内存。根据场景后续介绍和代码都是以4k为分界线,大于4k的我们认为是大块内存;小于4k的我们认为是小块内存。并且注意这里的4k,并不是严格遵照4096,而是在描述上,用4k比较好描述。
  内存池避免频繁向内核申请/释放内存,在真正使用内存之前,内存池提前分配一定数量且大小相等的内存块以作备用,当真正被用户调用api分配内存的时候,直接从内存块中获取内存(指小块内存),当内存块不够用了,再由内存池去申请新的内存块。而如果是需要大块内存,则内存池直接申请大块内存再返回给用户。
  内存池主要实现就三个:分配,扩容,回收。

小块内存

  内存池预申请一块4k的内存块,这里称为block,即block=4k内存块。当用户向内存池申请内存size小于4k时,内存池从block的空间中划分出去size空间,当再有新申请时,再划分出去。直到block中的剩余空间不足以分配size大小,那么此时内存池会再次申请一块block,再从新的block中划分size空间给用户。每一次申请小内存,都会在对应的block中引用计数加1,每一次释放小内存时,都会在block中引用计数减1,只有当引用计数为的时候,才会回收block使他重新成为空闲空间,以便重复利用空间。这样,内存池避免频繁向内核申请/释放内存,从而提高系统性能。

大块内存

  内存池主要解决小块的外部碎片导致的问题,所以内存池不预先申请内存。当申请大块内存的时候,内存池再申请内存给用户,回收的时候也就直接free就行。

分配与管理

  线程池刚创建只申请一个内存块和两个结构体,一个是pool结构体负责管理每个块和大块内存,还有个node结构体相当于和其对应的内存块绑定,用来提供具体内存块的相关信息。head指向所有小块内存块的首个块,current后面有大用处,当前可以理解为当前的内存块。
(end指向内存块的最后位置,last用来指向内存已经使用到的位置。)
内存池 c语言实现_第3张图片
  当有小块内存来的时候就放到4k里面(前提是放的下),然后移动相应的last指针。
内存池 c语言实现_第4张图片
内存池 c语言实现_第5张图片
  当有大块内存来的时候,只将负责管理大块内存的结构体大小的内存放入到block里面。管理large的结构体内存有相关的信息,如申请的内存指向哪里,大小等等,释放的时候只释放指向的内存区域,本身这个结构体跟着block最后一起销毁。
内存池 c语言实现_第6张图片
  当内存不够的时候就需要申请内存块。
内存池 c语言实现_第7张图片
  有些细化的东西下面代码看着看着就懂了,一切尽在代码中。

代码实现

宏定义

#define PAGE_SIZE 4096
//用来内存对齐
#define MP_ALIGNMENT 16
#define mp_align(n, alignment) (((n)+(alignment-1)) & ~(alignment-1))
#define mp_align_ptr(p, alignment) (void *)((((size_t)p)+(alignment-1)) & ~(alignment-1))

  做内存对齐可以更好提高内存访问速度,并且某些平台(arm)不支持未内存对齐的访问。

结构体定义

  结构体也就起到个用来管理和提供相关信息的作用。

//每一页的结构体
struct mp_node_s {
    unsigned char *end;//块的结尾
    unsigned char *last;//使用到哪了
    struct mp_node_s *next;//链表
    int quote;//引用计数
    int failed;//失效次数
};

//大块内存结构体
struct mp_large_s {
    struct mp_large_s *next;//链表
    int size;//alloc的大小
    void *alloc;//大块内存的起始地址
};

struct mp_pool_s {
    struct mp_large_s *large;
    struct mp_node_s *head;
    struct mp_node_s *current;
};

线程池的创建

  不用malloc()而用内置的posix_memalign()可以更好的内存对齐。

struct mp_pool_s *mp_create_pool(size_t size) {
    struct mp_pool_s *pool;
    if (size < PAGE_SIZE || size % PAGE_SIZE != 0) {
        size = PAGE_SIZE;
    }

    //分配4k以上不用malloc,用posix_memalign
    /*
        int posix_memalign (void **memptr, size_t alignment, size_t size);
     */

    int ret = posix_memalign((void **) &pool, MP_ALIGNMENT, size + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s)); //4K + mp_pool_s
    if (ret) {
        return NULL;
    }

    pool->large = NULL;
    pool->current = pool->head = (unsigned char *) pool + sizeof(struct mp_pool_s);
    pool->head->last = (unsigned char *) pool + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s);
    pool->head->end = (unsigned char *) pool + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s)+PAGE_SIZE;
    pool->head->failed = 0;

    return pool;
}

内存池的销毁

void mp_destroy_pool(struct mp_pool_s *pool) {
    struct mp_large_s *large;
    //释放了指向的空间,结构体还在小块内存中,还可以继续使用
    for (large = pool->large; large; large = large->next) {
        if (large->alloc) {
            free(large->alloc);
        }
    }

    //从第二页开始释放,最后再释放初始的第一页和前面的mp_pool_s结构体
    struct mp_node_s *cur, *next;
    cur = pool->head->next;

    while (cur) {
        next = cur->next;
        free(cur);
        cur = next;
    }
    free(pool);
}

内存申请

void *mp_malloc(struct mp_pool_s *pool, size_t size) {
    if (size <= 0) {
        return NULL;
    }
    if (size > PAGE_SIZE ) {
        //large
        return mp_malloc_large(pool, size);
    }
    else {
        //small
        unsigned char *mem_addr = NULL;
        struct mp_node_s *cur = NULL;
        cur = pool->current;//为什么是current不是head重点在于下面的扩容处
        while (cur) {
            mem_addr = mp_align_ptr(cur->last, MP_ALIGNMENT);
            if (cur->end - mem_addr >= size) {
                cur->quote++;//引用+1
                cur->last = mem_addr + size;
                return mem_addr;
            }
            else {
                cur = cur->next;
            }
        }
        //没位置放就多申请出来一页
        return mp_malloc_block(pool, size);// open new space
    }
}

void *mp_calloc(struct mp_pool_s *pool, size_t size) {
    void *mem_addr = mp_malloc(pool, size);
    if (mem_addr) {
        memset(mem_addr, 0, size);
    }
    return mem_addr;
}

block(页)申请,扩容

  既然已经进入到block申请了,说明前面几个页的空间都不够放这次申请的空间,在循环的时候给其failed+1,大于4的时候就换current的位置,下次就不从前面几个块里面遍历了,前面几个块剩余的那点小空间就相当于彻底放弃并且浪费了,至于为什么是4,是一个经验值。

void *mp_malloc_block(struct mp_pool_s *pool, size_t size) {
    unsigned char *block;
    int ret = posix_memalign((void **) &block, MP_ALIGNMENT, sizeof(struct mp_node_s) +PAGE_SIZE);
    if (ret) {
        return NULL;
    }

    struct mp_node_s *new_node = (struct mp_node_s *) block;
    new_node->end = block + sizeof(struct mp_node_s) +PAGE_SIZE;
    new_node->next = NULL;

    unsigned char *ret_addr = mp_align_ptr(block + sizeof(struct mp_node_s), MP_ALIGNMENT);

    new_node->last = ret_addr + size;
    new_node->quote++;

    struct mp_node_s *current = pool->current;
    struct mp_node_s *cur = NULL;

    for (cur = current; cur->next; cur = cur->next) {
        if (cur->failed++ > 4) {
            current = cur->next;
        }
    }

    //now cur = last node
    cur->next = new_node;
    pool->current = current ? current : new_node;
    return ret_addr;
}

大块内存申请

void *mp_malloc_large(struct mp_pool_s *pool, size_t size) {
    unsigned char *big_addr;
    int ret = posix_memalign((void **) &big_addr, MP_ALIGNMENT, size); //size
    if (ret) {
        return NULL;
    }

    struct mp_large_s *large;

    int n = 0;
    for (large = pool->large; large; large = large->next) {
        if (large->alloc == NULL) {
            large->size = size;
            large->alloc = big_addr;
            return big_addr;
        }
        if (n++ > 3) {
            break;// 为了避免过多的遍历,限制次数
        }
    }

    //把大块内存的相关信息结构体放到小块内存里
    large = mp_malloc(pool, sizeof(struct mp_large_s));
    if (large == NULL) {
        free(big_addr);
        return NULL;
    }
    large->size = size;
    large->alloc = big_addr;
    //头插法
    large->next = pool->large;
    pool->large = large;
    return big_addr;
}

内存释放

void mp_free(struct mp_pool_s *pool, void *p) {
    struct mp_large_s *large;
    for (large = pool->large; large; large = large->next) {//大块
        if (p == large->alloc) {
            free(large->alloc);
            //结构体信息不释放,留着记录下一次大块内存。释放的时候也方便,释放页的时候顺便一起释放
            large->size = 0;
            large->alloc = NULL;
            return;
        }
    }
    //小块 引用-1
    struct mp_node_s *cur = NULL;
    for (cur = pool->head; cur; cur = cur->next) {
        if ((unsigned char *) cur <= (unsigned char *) p && (unsigned char *) p <= (unsigned char *) cur->end) {
            cur->quote--;
            if (cur->quote == 0) {
                if (cur == pool->head) {
                    pool->head->last = (unsigned char *) pool + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s);
                }
                else {
                    cur->last = (unsigned char *) cur + sizeof(struct mp_node_s);
                }
                cur->failed = 0;
                pool->current = pool->head;
            }
            return;
        }
    }
}

内存池重置

void mp_reset_pool(struct mp_pool_s *pool) {
    struct mp_node_s *cur;
    struct mp_large_s *large;

    for (large = pool->large; large; large = large->next) {
        if (large->alloc) {
            free(large->alloc);
        }
    }

    pool->large = NULL;
    pool->current = pool->head;
    for (cur = pool->head; cur; cur = cur->next) {
        cur->last = (unsigned char *) cur + sizeof(struct mp_node_s);
        cur->failed = 0;
        cur->quote = 0;
    }
}

代码测试

#include 
#include 
#include 

#define PAGE_SIZE 4096
#define MP_ALIGNMENT 16
#define mp_align(n, alignment) (((n)+(alignment-1)) & ~(alignment-1))
#define mp_align_ptr(p, alignment) (void *)((((size_t)p)+(alignment-1)) & ~(alignment-1))

struct mp_pool_s *mp_create_pool(size_t size);

void mp_destroy_pool(struct mp_pool_s *pool);

void *mp_malloc(struct mp_pool_s *pool, size_t size);

void *mp_calloc(struct mp_pool_s *pool, size_t size);

void mp_free(struct mp_pool_s *pool, void *p);

void mp_reset_pool(struct mp_pool_s *pool);

void monitor_mp_poll(struct mp_pool_s *pool, char *tk);

void *mp_malloc_block(struct mp_pool_s *pool, size_t size);

void *mp_malloc_large(struct mp_pool_s *pool, size_t size);

struct mp_node_s {
    unsigned char *end;//块的结尾
    unsigned char *last;//使用到哪了
    struct mp_node_s *next;//链表
    int quote;//引用计数
    int failed;//失效次数
};

struct mp_large_s {
    struct mp_large_s *next;//链表
    int size;//alloc的大小
    void *alloc;//大块内存的起始地址
};

struct mp_pool_s {
    struct mp_large_s *large;
    struct mp_node_s *head;
    struct mp_node_s *current;
};

struct mp_pool_s *mp_create_pool(size_t size) {
    struct mp_pool_s *pool;
    if (size < PAGE_SIZE || size % PAGE_SIZE != 0) {
        size = PAGE_SIZE;
    }

    //分配4k以上不用malloc,用posix_memalign
    /*
        int posix_memalign (void **memptr, size_t alignment, size_t size);
     */

    int ret = posix_memalign((void **) &pool, MP_ALIGNMENT, size + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s)); //4K + mp_pool_s
    if (ret) {
        return NULL;
    }

    pool->large = NULL;
    pool->current = pool->head = (unsigned char *) pool + sizeof(struct mp_pool_s);
    pool->head->last = (unsigned char *) pool + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s);
    pool->head->end = (unsigned char *) pool + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s)+PAGE_SIZE;
    pool->head->failed = 0;

    return pool;
}

void mp_destroy_pool(struct mp_pool_s *pool) {
    struct mp_large_s *large;
    //释放了指向的空间,结构体还在小块内存中,还可以继续使用
    for (large = pool->large; large; large = large->next) {
        if (large->alloc) {
            free(large->alloc);
        }
    }

    //从第二页开始释放,最后再释放初始的第一页和前面的mp_pool_s结构体
    struct mp_node_s *cur, *next;
    cur = pool->head->next;

    while (cur) {
        next = cur->next;
        free(cur);
        cur = next;
    }
    free(pool);
}

void *mp_malloc(struct mp_pool_s *pool, size_t size) {
    if (size <= 0) {
        return NULL;
    }
    if (size > PAGE_SIZE ) {
        //large
        return mp_malloc_large(pool, size);
    }
    else {
        //small
        unsigned char *mem_addr = NULL;
        struct mp_node_s *cur = NULL;
        cur = pool->current;
        while (cur) {
            mem_addr = mp_align_ptr(cur->last, MP_ALIGNMENT);
            if (cur->end - mem_addr >= size) {
                cur->quote++;//引用+1
                cur->last = mem_addr + size;
                return mem_addr;
            }
            else {
                cur = cur->next;
            }
        }
        //没位置放就多申请出来一页
        return mp_malloc_block(pool, size);// open new space
    }
}

void *mp_calloc(struct mp_pool_s *pool, size_t size) {
    void *mem_addr = mp_malloc(pool, size);
    if (mem_addr) {
        memset(mem_addr, 0, size);
    }
    return mem_addr;
}

void *mp_malloc_block(struct mp_pool_s *pool, size_t size) {
    unsigned char *block;
    int ret = posix_memalign((void **) &block, MP_ALIGNMENT, sizeof(struct mp_node_s) +PAGE_SIZE);
    if (ret) {
        return NULL;
    }

    struct mp_node_s *new_node = (struct mp_node_s *) block;
    new_node->end = block + sizeof(struct mp_node_s) +PAGE_SIZE;
    new_node->next = NULL;

    unsigned char *ret_addr = mp_align_ptr(block + sizeof(struct mp_node_s), MP_ALIGNMENT);

    new_node->last = ret_addr + size;
    new_node->quote++;

    struct mp_node_s *current = pool->current;
    struct mp_node_s *cur = NULL;

    for (cur = current; cur->next; cur = cur->next) {
        if (cur->failed++ > 4) {
            current = cur->next;
        }
    }

    //now cur = last node
    cur->next = new_node;
    pool->current = current ? current : new_node;
    return ret_addr;
}

void *mp_malloc_large(struct mp_pool_s *pool, size_t size) {
    unsigned char *big_addr;
    int ret = posix_memalign((void **) &big_addr, MP_ALIGNMENT, size); //size
    if (ret) {
        return NULL;
    }

    struct mp_large_s *large;

    int n = 0;
    for (large = pool->large; large; large = large->next) {
        if (large->alloc == NULL) {
            large->size = size;
            large->alloc = big_addr;
            return big_addr;
        }
        if (n++ > 3) {
            break;// 为了避免过多的遍历,限制次数
        }
    }

    //把大块内存的相关信息结构体放到小块内存里
    large = mp_malloc(pool, sizeof(struct mp_large_s));
    if (large == NULL) {
        free(big_addr);
        return NULL;
    }
    large->size = size;
    large->alloc = big_addr;
    //头插法
    large->next = pool->large;
    pool->large = large;
    return big_addr;
}

void mp_free(struct mp_pool_s *pool, void *p) {
    struct mp_large_s *large;
    for (large = pool->large; large; large = large->next) {//大块
        if (p == large->alloc) {
            free(large->alloc);
            //结构体信息不释放,留着记录下一次大块内存。释放的时候也方便,释放页的时候顺便一起释放
            large->size = 0;
            large->alloc = NULL;
            return;
        }
    }
    //小块 引用-1
    struct mp_node_s *cur = NULL;
    for (cur = pool->head; cur; cur = cur->next) {
        if ((unsigned char *) cur <= (unsigned char *) p && (unsigned char *) p <= (unsigned char *) cur->end) {
            cur->quote--;
            if (cur->quote == 0) {
                if (cur == pool->head) {
                    pool->head->last = (unsigned char *) pool + sizeof(struct mp_pool_s) + sizeof(struct mp_node_s);
                }
                else {
                    cur->last = (unsigned char *) cur + sizeof(struct mp_node_s);
                }
                cur->failed = 0;
                pool->current = pool->head;
            }
            return;
        }
    }
}

void mp_reset_pool(struct mp_pool_s *pool) {
    struct mp_node_s *cur;
    struct mp_large_s *large;

    for (large = pool->large; large; large = large->next) {
        if (large->alloc) {
            free(large->alloc);
        }
    }

    pool->large = NULL;
    pool->current = pool->head;
    for (cur = pool->head; cur; cur = cur->next) {
        cur->last = (unsigned char *) cur + sizeof(struct mp_node_s);
        cur->failed = 0;
        cur->quote = 0;
    }
}

//测试代码:
void monitor_mp_poll(struct mp_pool_s *pool, char *tk) {
    printf("\r\n\r\n------start monitor poll------%s\r\n\r\n", tk);
    struct mp_node_s *head = NULL;
    int i = 0;
    for (head = pool->head; head; head = head->next) {
        i++;
        if (pool->current == head) {
            printf("current==>%d块\n", i);
        }
        if (i == 1) {
            printf("第%02d块small block  已使用:%4ld  剩余空间:%4ld  引用:%4d  failed:%4d\n",     \
                   i,                                                                                 \
                   (unsigned char *) head->last - ((unsigned char *) pool+sizeof(struct mp_node_s)),                             \
                   head->end - head->last, head->quote, head->failed);
        }
        else {
            printf("第%02d块small block  已使用:%4ld  剩余空间:%4ld  引用:%4d  failed:%4d\n",  \
                   i,                                                                               \
                   (unsigned char *) head->last - ((unsigned char *) head+sizeof(struct mp_node_s)),                           \
                   head->end - head->last, head->quote, head->failed);                              \
        }
    }

    struct mp_large_s *large;
    i = 0;
    for (large = pool->large; large; large = large->next) {
        i++;
        if (large->alloc != NULL) {
            printf("第%d块large block  size=%d\n", i, large->size);
        }
    }
    printf("\r\n\r\n------stop monitor poll------\r\n\r\n");
}

int main() {
    struct mp_pool_s *p = mp_create_pool(PAGE_SIZE);
    monitor_mp_poll(p, "create memory pool");
    void *mp[30];
    int i;
    for (i = 0; i < 30; i++) {
        mp[i] = mp_malloc(p, 512);
    }
    monitor_mp_poll(p, "申请512字节30个");

    for (i = 0; i < 30; i++) {
        mp_free(p, mp[i]);
    }
    monitor_mp_poll(p, "销毁512字节30个");

    int j;
    for (i = 0; i < 50; i++) {
        char *pp = mp_calloc(p, 32);
        for (j = 0; j < 32; j++) {
            if (pp[j]) {
                printf("calloc wrong\n");
                exit(-1);
            }
        }
    }
    monitor_mp_poll(p, "申请32字节50个");

    for (i = 0; i < 50; i++) {
        char *pp = mp_malloc(p, 3);
    }
    monitor_mp_poll(p, "申请3字节50个");


    void *pp[10];
    for (i = 0; i < 10; i++) {
        pp[i] = mp_malloc(p, 5120);
    }
    monitor_mp_poll(p, "申请大内存5120字节10个");

    for (i = 0; i < 10; i++) {
        mp_free(p, pp[i]);
    }
    monitor_mp_poll(p, "销毁大内存5120字节10个");

    mp_reset_pool(p);
    monitor_mp_poll(p, "reset pool");

    for (i = 0; i < 100; i++) {
        void *s = mp_malloc(p, 256);
    }
    monitor_mp_poll(p, "申请256字节100个");

    mp_destroy_pool(p);
    return 0;

}

内存池 c语言实现_第8张图片

你可能感兴趣的:(操作系统,c语言,服务器)