mysql内存管理

 1 内存管理结构

 mysql有自己的内存申请和释放机制

 mysql层有mem_root

 innodb层有mem_heap,mem_pool,buf_pool

 它们的结构图如下

 

 

 

2 mem_root

mem_root是mysql层的动态内存管理

 

typedef struct st_used_mem
{                   /* struct for once_alloc (block) */
  struct st_used_mem *next;       /* Next block in use */
  unsigned int    left;           /* memory left in block  */
  unsigned int    size;           /* size of block */
} USED_MEM;

typedef struct st_mem_root
{
  USED_MEM *free;                  /* blocks with free memory in it */
  USED_MEM *used;                  /* blocks almost without free memory */
  USED_MEM *pre_alloc;             /* preallocated block */
  /* if block have less memory it will be put in 'used' list */
  size_t min_malloc;
  size_t block_size;               /* initial block size */
  unsigned int block_num;          /* allocated blocks counter */
  /* 
     first free block in queue test counter (if it exceed 
     MAX_BLOCK_USAGE_BEFORE_DROP block will be dropped in 'used' list)
  */
  unsigned int first_block_usage;

  void (*error_handler)(void);
} MEM_ROOT;

mem_root主要由两个链表组成free和used。

free：存放有空闲的block，其中的block可能部分被使用。

used：存放已使用的block，其中block可能有少量空闲.

block_size：初始块大小,后面分配的块可能比block_size大。

pre_alloc：一直指向初始化时分配的block，改block开始在free链表中，随着内存的不断申请，改block可能会存在于used链表中

 

2.1 init_alloc_root

初始化一个block放入free链表，pre_alloc指向这个链表

2.2 alloc_root

首先从free链表中查找，如果有空间合适的block，则直接使用，若该block剩余空间小于min_malloc，则会放入used链表头部。如果没有空间合适的block，则新分配block放入free链表尾部。

2.3 reset_root_defaults

1没有初始化的mem_root不需要reset

满足if (!mem_root->pre_alloc || mem_root->pre_alloc->size != size)才reset

 

2 只释放完全没有使用的空闲块。

 

2.4 free_root

MY_MARK_BLOCKS_FREE:释放所有块

MY_KEEP_PREALLOC：只保留pre_alloc块

 

3 mem_pool

内存池。innodb早期自己实现的一套伙伴算法内存管理系统，由参数innodb_use_sys_malloc 控制开启和关闭，默认关闭，即直接从操作系统申请和释放内存.参见http://docs.oracle.com/cd/E17952_01/refman-5.5-en/innodb-performance-use_sys_malloc.html

mem_pool_t

 

/** Data structure for a memory pool. The space is allocated using the buddy

algorithm, where free list i contains areas of size 2 to power i. */

struct mem_pool_t{

    byte*       buf;        /*!< memory pool */

    ulint       size;       /*!< memory common pool size */

    ulint       reserved;   /*!< amount of currently allocated

                    memory */

    ib_mutex_t      mutex;      /*!< mutex protecting this struct */

    UT_LIST_BASE_NODE_T(mem_area_t)

            free_list[64];  /*!< lists of free memory areas: an

                    area is put to the list whose number

                    is the 2-logarithm of the area size */

};

/** Memory area header */

struct mem_area_t{

    ulint       size_and_free;  /*!< memory area size is obtained by

                    anding with ~MEM_AREA_FREE; area in

                    a free list if ANDing with

                    MEM_AREA_FREE results in nonzero */

    UT_LIST_NODE_T(mem_area_t)

            free_list;  /*!< free list node */

};

buf:一片连续的内存区域

mem_area_t：内存块, size_and_free标记内存块的大小和是否已使用。

free_list[64]： 64个桶，每个桶编号为0,1,2,3,…,i;每个桶依次存放2ⁱ大小的内存块。

 

3.1初始化

mem_pool_create

在服务启动时调用一次

如初始化一个19字节的内存池

19=100011=2⁴+2¹+2⁰

 3.2 mem_area_alloc

void*

mem_area_alloc(

/*===========*/

    ulint*      psize,  /*!< in: requested size in bytes; for optimum

                space usage, the size should be a power of 2

                minus MEM_AREA_EXTRA_SIZE;

                out: allocated size in bytes (greater than

                or equal to the requested size) */

    mem_pool_t* pool)

从pool中申请内存块，psize<=2ⁱ,i为满足此条件的最小值。当第i个中存在空闲时，直接取出。当第i个中不存在空闲块时，从i+1个桶中切割一半到第i个桶中，这是一个递归的过程。具体实现在mem_pool_fill_free_list；

以申请大小为3的内存为例，（为了说明方便，这里不考虑块中MEM_AREA_EXTRA_SIZE的空间）

申请的得到大小为4的块，这是内存池中的结构如下

 

3.3 mem_pool_free

释放一个块到pool中，将块放入对应大小的桶i中，如果桶i中存在自己的伙伴（和自己相邻的内存块，可能在左边也可能在右边），则合并放入i+1的桶中，此过程递归，直到没有可以合并的伙伴。

 

找自己伙伴的函数如下

mem_area_t*

mem_area_get_buddy(

/*===============*/

    mem_area_t* area,   /*!< in: memory area */

    ulint       size,   /*!< in: memory area size */

    mem_pool_t* pool)   /*!< in: memory pool */

{

    mem_area_t* buddy;

 

    ut_ad(size != 0);

 

    if (((((byte*) area) - pool->buf) % (2 * size)) == 0) {

 

        /* The buddy is in a higher address */

 

        buddy = (mem_area_t*)(((byte*) area) + size);

 

        if ((((byte*) buddy) - pool->buf) + size > pool->size) {

 

            /* The buddy is not wholly contained in the pool:

            there is no buddy */

 

            buddy = NULL;

        }

    } else {

        /* The buddy is in a lower address; NOTE that area cannot

        be at the pool lower end, because then we would end up to

        the upper branch in this if-clause: the remainder would be

        0 */

 

        buddy = (mem_area_t*)(((byte*) area) - size);

    }

 

    return(buddy);

}

 

找x的伙伴：

1 首先确定x的伙伴在高位还是在低位，x左边的块大小都>=x的大小，且x左边的块大小=2ⁱ*sizeof(x) (i>0);。

如果x的伙伴是y,则(c-a)%(2*sizeof(x))=sizeof(x) != 0

如果x的伙伴是z,则(c-a)%(2*sizeof(x))= 0

  

需要说明的是：在桶中的块都是空闲块。

 

以释放刚申请得的大小为4的块为例，释放后，内存结构如下，回到了初始状态。

 

3.4 mem_pool_free

在服务关闭时调用一次

释放buf和mem_pool结构即可

mem_pool_free(

/*==========*/

    mem_pool_t* pool)   /*!< in, own: memory pool */

{

    ut_free(pool->buf);

    ut_free(pool);

}

 

 

4 buf_pool

 

块/页缓存池。

4.1buf_pool_init

Innodb_buffer_pool_instances指定了bp的个数，innodb_buffer_pool_size指定所有bp的总大小。

buf_pool_init在服务启动时调用一次

 bp初始化主要是申请内存，并初始化free链表。每个buf_page_t的大小为innodb_page_size.

UT_LIST_BASE_NODE_T(buf_page_t) free;

4.2 buf_block_alloc

从free链表头部摘出

不放入LRU链表

4.3 buf_block_free

放回free链表头部

 

4.4 buffer_pool_free

在服务关闭时调用一次

5 mem_heap

/** The info structure stored at the beginning of a heap block */

struct mem_block_info_t {

    ulint   magic_n;/* magic number for debugging */

    char    file_name[8];/* file name where the mem heap was created */

    ulint   line;   /*!< line number where the mem heap was created */

    UT_LIST_BASE_NODE_T(mem_block_t) base; /* In the first block in the

            the list this is the base node of the list of blocks;

            in subsequent blocks this is undefined */

    UT_LIST_NODE_T(mem_block_t) list; /* This contains pointers to next

            and prev in the list. The first block allocated

            to the heap is also the first block in this list,

            though it also contains the base node of the list. */

    ulint   len;    /*!< physical length of this block in bytes */

    ulint   total_size; /*!< physical length in bytes of all blocks

            in the heap. This is defined only in the base

            node and is set to ULINT_UNDEFINED in others. */

    ulint   type;   /*!< type of heap: MEM_HEAP_DYNAMIC, or

            MEM_HEAP_BUF possibly ORed to MEM_HEAP_BTR_SEARCH */

    ulint   free;   /*!< offset in bytes of the first free position for

            user data in the block */

    ulint   start;  /*!< the value of the struct field 'free' at the

            creation of the block */

#ifndef UNIV_HOTBACKUP

    void*   free_block;

            /* if the MEM_HEAP_BTR_SEARCH bit is set in type,

            and this is the heap root, this can contain an

            allocated buffer frame, which can be appended as a

            free block to the heap, if we need more space;

            otherwise, this is NULL */

    void*   buf_block;

            /* if this block has been allocated from the buffer

            pool, this contains the buf_block_t handle;

            otherwise, this is NULL */

#endif /* !UNIV_HOTBACKUP */

#ifdef MEM_PERIODIC_CHECK

    UT_LIST_NODE_T(mem_block_t) mem_block_list;

            /* List of all mem blocks allocated; protected

            by the mem_comm_pool mutex */

#endif

}; 

 innodb绝大多数内存申请和释放都是在mem_heap上进行的

 mem_heap实际上是一个内存块链表，内存块大小依次增长，至少是两倍的增长。

 

5.1 mem_heap_create

初始内存块链表，根据参数初始化内存块，此时base链表只有一个内存块

   

 if (heap && heap->magic_n != MEM_BLOCK_MAGIC_N) {

        mem_analyze_corruption(heap);

    }

 

    /* In dynamic allocation, calculate the size: block header + data. */

    len = MEM_BLOCK_HEADER_SIZE + MEM_SPACE_NEEDED(n);

 

#ifndef UNIV_HOTBACKUP

    if (type == MEM_HEAP_DYNAMIC || len < UNIV_PAGE_SIZE / 2) {

 

        ut_ad(type == MEM_HEAP_DYNAMIC || n <= MEM_MAX_ALLOC_IN_BUF);

 

        block = static_cast<mem_block_t*>(

            mem_area_alloc(&len, mem_comm_pool));

    } else {

        len = UNIV_PAGE_SIZE;

 

        if ((type & MEM_HEAP_BTR_SEARCH) && heap) {

            /* We cannot allocate the block from the

            buffer pool, but must get the free block from

            the heap header free block field */

 

            buf_block = static_cast<buf_block_t*>(heap->free_block);

            heap->free_block = NULL;

 

            if (UNIV_UNLIKELY(!buf_block)) {

 

                return(NULL);

            }

        } else {

            buf_block = buf_block_alloc(NULL);

        }

 

        block = (mem_block_t*) buf_block->frame;

    }

 

    ut_ad(block);

    block->buf_block = buf_block;

    block->free_block = NULL;

#else /* !UNIV_HOTBACKUP */

    len = MEM_BLOCK_HEADER_SIZE + MEM_SPACE_NEEDED(n);

    block = ut_malloc(len);

    ut_ad(block);

#endif /* !UNIV_HOTBACKUP */

 

 以上代码可以看出

Mem_heap的内存来源可以有三种

1 mem_pool 的mem_area_alloc

2 buffer_pool的buf_block_alloc

3 ut_malloc ->malloc

 

5.2 mem_heap_alloc

从base链表最后一个内存块分配内存，不够则新分配一个较大内存块，放入base链表的最后。

5.3 mem_heap_free_heap_top

void

mem_heap_free_heap_top(

/*===================*/

    mem_heap_t* heap,   /*!< in: heap from which to free */

    byte*       old_top)/*!< in: pointer to old top of heap */

 

 释放base链表从old_top到最后的内存块，old_top不释放

5.4 mem_heap_free_heap_top

释放base链表最后一个内存块

5.5 mem_heap_free

释放base链表所有内存块