Redis4.0新特性-主动内存碎片整理

What is active defragmentation?

什么是主动内存碎片整理？

Active (online) defragmentation allows a Redis server to compact the spaces left between small allocations and deallocations of data in memory, thus allowing to reclaim back memory.

主动内存碎片整理允许一个Redis服务器在内存中的数据分配和释放过程中对产生的空间进行压缩，以便释放内存。

Fragmentation is a natural process that happens with every allocator (but less so with Jemalloc, fortunately) and certain workloads. Normally a server restart is needed in order to lower the fragmentation, or at least to flush away all the data and create it again. However thanks to this feature implemented by Oran Agra for Redis 4.0 this process can happen at runtime in a “hot” way, while the server is running.

所有分配器在一定负载下都会自然的发生碎片整理（Jemalloc 很少）。 通常，需要重启服务器以降低碎片，或至少清除所有数据然后重新创建它们。但是，由于Oran Agra在Redis 4.0中实现了这个特性，碎片整理过程可以在服务器运行时进行。

Basically when the fragmentation is over a certain level (see the
configuration options below) Redis will start to create new copies of the values in contiguous memory regions by exploiting certain specific Jemalloc features (in order to understand if an allocation is causing fragmentation and to allocate it in a better place), and at the same time, will release the old copies of the data. This process, repeated incrementally for all the keys
will cause the fragmentation to drop back to normal values.

基本上，当碎片达到一定水平（参见下面的配置选项），Redis将通过Jemalloc的功能（以了解分配是否导致碎片，并为它们分配更好的位置）并开始在临近的内存中创建新的这些值的副本，同时也将释放旧的值的副本。这个过程将重复地递增地对所有的键进行，直到碎片整理回到正常值。

Important things to understand:

1. This feature is disabled by default, and only works if you compiled Redis to use the copy of Jemalloc we ship with the source code of Redis. This is the default with Linux builds.

   这个功能默认是关闭的，而且只有在你用我们发布的带Jemalloc代码的Redis源码来编译Redis时才能使用这个功能，Linux发布版默认是带这个功能的。

2. You never need to enable this feature if you don’t have fragmentation issues.

   如果没有碎片整理问题，你不需要启用这个功能。

3. Once you experience fragmentation, you can enable this feature when
   needed with the command “CONFIG SET activedefrag yes”.

   你可以通过"CONFIG SET activedefrag yes"启用这个功能。

The configuration parameters are able to fine tune the behavior of the defragmentation process. If you are not sure about what they mean it is a good idea to leave the defaults untouched.

配置参数可以调整碎片整理的效果，但如果你不确定这些配置参数的含义，那么你可以让它保持默认值。

# Active defragmentation is disabled by default
# 在默认情况下，主动内存碎片整理是关闭的
# activedefrag no

# Minimum amount of fragmentation waste to start active defrag
# 开始主动内存碎片整理的最小碎片损耗量
# active-defrag-ignore-bytes 100mb

# Minimum percentage of fragmentation to start active defrag
# 开始主动内存碎片整理的最小碎片百分比
# active-defrag-threshold-lower 10

# Maximum percentage of fragmentation at which we use maximum effort
# 最大力度进行碎片整理的最大碎片率百分比
# active-defrag-threshold-upper 100

# Minimal effort for defrag in CPU percentage, to be used when the lower
# threshold is reached
# 碎片整理占用CPU下限阈值
# active-defrag-cycle-min 1

# Maximal effort for defrag in CPU percentage, to be used when the upper
# threshold is reached 
# 碎片整理占用CPU上限阈值
# active-defrag-cycle-max 25

# Maximum number of set/hash/zset/list fields that will be processed from
# the main dictionary scan
# set/hash/zset/list 类型的扫描最大元素数量
# active-defrag-max-scan-fields 1000

测试碎片整理

使用配置创建redis实例

redis-standalone.conf

port 7007

bind 0.0.0.0
daemonize yes
logfile "/opt/cachecloud/logs/redis-standalone-7007.log"
dir "/opt/cachecloud/data"
dbfilename "dump_7007.rdb"
# 配置开启主动碎片整理
activedefrag yes
active-defrag-ignore-bytes 10mb
active-defrag-threshold-lower 2
active-defrag-threshold-upper 150
active-defrag-cycle-min 1
active-defrag-cycle-max 25
active-defrag-max-scan-fields 1000

启动redis实例

redis-server /opt/cachecloud/conf/redis-standalone.conf

插入大量数据

==========插入大量数据以后内存状况==========
[root@reids-7-104 ~]# redis-cli -p 7007 info memory
# Memory
used_memory:545047960
used_memory_human:519.80M
used_memory_rss:563924992
used_memory_rss_human:537.80M
allocator_frag_ratio:1.00
allocator_frag_bytes:161464
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:18938936
active_defrag_running:0

[root@reids-7-104 ~]# redis-cli -p 7007 memory stats
37) "allocator-fragmentation.ratio"
38) "1.0003455877304077"
39) "allocator-fragmentation.bytes"
40) (integer) 188344
49) "fragmentation"
50) "1.0352247953414917"
51) "fragmentation.bytes"
52) (integer) 19196984

开始执行python脚本批量删除数据

import redis
import traceback

re = redis.StrictRedis(host = "10.4.7.104",port = 7007,password = "")

key = "jwjrand"
cur = '0'
cou = 1000
while cur != 0:
    # cur,data = re.scan(cursor=cur, match=key, count = cou)
    cur,data = re.execute_command('scan', cur, 'match', 'dmntest1*', "count", 1000)
    try:
        p = re.pipeline()
        for item in data:
            print("==>",item.decode())
            p.delete(item.decode())
        p.execute()
        print("==>",cur)
    except Exception as e:
        print(traceback.format_exc())

===============数据清理过程中内存状况==============
[root@reids-7-104 ~]# redis-cli -p 7007 info memory
# Memory
used_memory:478123936
used_memory_human:455.97M
used_memory_rss:565260288
used_memory_rss_human:539.07M
allocator_frag_ratio:1.14
allocator_frag_bytes:65979840
active_defrag_running:2
[root@reids-7-104 ~]# redis-cli -p 7007 memory stats
37) "allocator-fragmentation.ratio"
38) "1.213597297668457"
39) "allocator-fragmentation.bytes"
40) (integer) 95918528
49) "fragmentation"
50) "1.2589750289916992"
51) "fragmentation.bytes"
52) (integer) 116249680

[root@reids-7-104 ~]# redis-cli -p 7007 info memory
# Memory
used_memory:375470976
used_memory_human:358.08M
used_memory_rss:564625408
used_memory_rss_human:538.47M
mem_fragmentation_ratio:1.50
mem_fragmentation_bytes:187056320
active_defrag_running:7

[root@reids-7-104 ~]# redis-cli -p 7007 memory stats
37) "allocator-fragmentation.ratio"
38) "1.5605411529541016"
39) "allocator-fragmentation.bytes"
40) (integer) 191159088
49) "fragmentation"
50) "1.6560860872268677"
51) "fragmentation.bytes"
52) (integer) 223619224

[root@reids-7-104 ~]# redis-cli -p 7007 info memory
# Memory
used_memory:265272096
used_memory_human:252.98M
used_memory_rss:557862912
used_memory_rss_human:532.02M
allocator_frag_ratio:1.76
allocator_frag_bytes:204627600
mem_fragmentation_ratio:2.09
mem_fragmentation_bytes:290492704
active_defrag_running:12

[root@reids-7-104 ~]# redis-cli -p 7007 memory stats
37) "allocator-fragmentation.ratio"
38) "1.5875883102416992"
39) "allocator-fragmentation.bytes"
40) (integer) 113410912
49) "fragmentation"
50) "2.6743261814117432"
51) "fragmentation.bytes"
52) (integer) 322839920

[root@reids-7-104 ~]# redis-cli -p 7007 info memory
# Memory
used_memory:915584
used_memory_human:894.12K
used_memory_rss:145321984
used_memory_rss_human:138.59M
mem_fragmentation_ratio:170.23
mem_fragmentation_bytes:144468304
mem_allocator:jemalloc-5.1.0
active_defrag_running:17

[root@reids-7-104 ~]# redis-cli -p 7007 memory stats
35) "allocator-fragmentation.ratio"
36) "1.2425054311752319"
37) "allocator-fragmentation.bytes"
38) (integer) 215048
47) "fragmentation"
48) "170.23004150390625"
49) "fragmentation.bytes"
50) (integer) 144468304

===============数据清理完以后内存状况==============
[root@reids-7-104 ~]# redis-cli -p 7007 info memory
# Memory
used_memory:853688
used_memory_human:833.68K
used_memory_rss:151199744
used_memory_rss_human:144.20M
used_memory_peak:545047968
allocator_frag_ratio:1.23
allocator_frag_bytes:190568
mem_fragmentation_ratio:190.96
mem_fragmentation_bytes:150407960
active_defrag_running:0
[root@reids-7-104 ~]# redis-cli -p 7007 memory stats
35) "allocator-fragmentation.ratio"
36) "1.2568612098693848"
37) "allocator-fragmentation.bytes"
38) (integer) 218480
47) "fragmentation"
48) "190.96084594726562"
49) "fragmentation.bytes"
50) (integer) 150407960

从清理数据过程中info memory和memory stats命令查看到的内存使用情况和内存详情可以看到，清理过程中有触发主动碎片整理，active_defrag_running:17， 并且碎片整理占用的CPU为17%。还可以从mem_fragmentation_bytes的变化过程中看到，在大量删除数据的过程中，mem_fragmentation_bytes因为碎片产生而逐渐变大，而在执行主动碎片整理后,mem_fragmentation_bytes的值逐渐变小。

另外info命令几个相关的返回值需要解释下:

• mem_fragmentation_ratio: used_memory_rssRedis从系统申请的内存和used_memoryRedis给数据分配的内存的比率。不仅包括碎片，还包括其他进程及代码、共享库、堆栈的内存开销等等(参见allocator_* 指标)。
• mem_fragmentation_bytes: used_memory_rss 和 used_memory的差值. 要注意如果的碎片量很低(几兆)但是碎片率很高，没啥事。
• allocator_allocated: Redis分配器分配的总内存的字节量，包含内部碎片。通常与used_memory量一样。
• allocator_active: 分配器活动页中的字节量，包含外部碎片。
• allocator_resident: 分配器包含的总字节量，包含可以发布到系统中的页的总字节量（通过MEMORY PURGE 命令或者等待）。
• allocator_frag_ratio: allocator_active和allocator_allocated的比率。这是真实的外部碎片指标（不是mem_fragmentation_ratio）。
• active_defrag_running: 开启activedefrag以后，这个指标可以展示是否在执行主动碎片整理，以及执行回收将使用的CPU率.