Redis进阶-如何发现和优雅的处理BigKey一二事

文章目录

  • PreView
  • 模拟写入一个BigKey
  • 如何发现BigKey
    • redis-cli --bigkeys
    • debug object
  • 如何优雅的删除BigKey (lazy delete)
  • 关于lazyfree的几个配置项
  • 如何优化bigKey

在这里插入图片描述


PreView

Redis进阶-Redis键值设计及BigKey问题粗略的讨论了BigKey,这里我们继续再来研究下BigKey

Redis Version : 5.0.3


模拟写入一个BigKey

Redis进阶-如何发现和优雅的处理BigKey一二事_第1张图片

在这里插入图片描述


如何发现BigKey

总结下无非几个点

  • 应用异常日志 (没法直接从日志关键字中看出来,因为这个bigkey阻塞以后,会引发一些奇怪的问题,比如从连接池获取不到连接等等)
  • redis-cli --bigkeys
  • 监控系统上报
  • 其他…

redis-cli --bigkeys

先看下 redis-cli 都支持什么参数

[redis@artisan bin]$ ./redis-cli --help
redis-cli 5.0.3

Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]
  -h <hostname>      Server hostname (default: 127.0.0.1).
  -p <port>          Server port (default: 6379).
  -s <socket>        Server socket (overrides hostname and port).
  -a <password>      Password to use when connecting to the server.
                     You can also use the REDISCLI_AUTH environment
                     variable to pass this password more safely
                     (if both are used, this argument takes predecence).
  -u <uri>           Server URI.
  -r <repeat>        Execute specified command N times.
  -i <interval>      When -r is used, waits <interval> seconds per command.
                     It is possible to specify sub-second times like -i 0.1.
  -n <db>            Database number.
  -x                 Read last argument from STDIN.
  -d <delimiter>     Multi-bulk delimiter in for raw formatting (default: \n).
  -c                 Enable cluster mode (follow -ASK and -MOVED redirections).
  --raw              Use raw formatting for replies (default when STDOUT is
                     not a tty).
  --no-raw           Force formatted output even when STDOUT is not a tty.
  --csv              Output in CSV format.
  --stat             Print rolling stats about server: mem, clients, ...
  --latency          Enter a special mode continuously sampling latency.
                     If you use this mode in an interactive session it runs
                     forever displaying real-time stats. Otherwise if --raw or
                     --csv is specified, or if you redirect the output to a non
                     TTY, it samples the latency for 1 second (you can use
                     -i to change the interval), then produces a single output
                     and exits.
  --latency-history  Like --latency but tracking latency changes over time.
                     Default time interval is 15 sec. Change it using -i.
  --latency-dist     Shows latency as a spectrum, requires xterm 256 colors.
                     Default time interval is 1 sec. Change it using -i.
  --lru-test <keys>  Simulate a cache workload with an 80-20 distribution.
  --replica          Simulate a replica showing commands received from the master.
  --rdb <filename>   Transfer an RDB dump from remote server to local file.
  --pipe             Transfer raw Redis protocol from stdin to server.
  --pipe-timeout <n> In --pipe mode, abort with error if after sending all data.
                     no reply is received within <n> seconds.
                     Default timeout: 30. Use 0 to wait forever.
  --bigkeys          Sample Redis keys looking for big keys.
  --hotkeys          Sample Redis keys looking for hot keys.
                     only works when maxmemory-policy is *lfu.
  --scan             List all keys using the SCAN command.
  --pattern <pat>    Useful with --scan to specify a SCAN pattern.
  --intrinsic-latency <sec> Run a test to measure intrinsic system latency.
                     The test will run for the specified amount of seconds.
  --eval <file>      Send an EVAL command using the Lua script at <file>.
  --ldb              Used with --eval enable the Redis Lua debugger.
  --ldb-sync-mode    Like --ldb but uses the synchronous Lua debugger, in
                     this mode the server is blocked and script changes are
                     not rolled back from the server memory.
  --cluster <command> [args...] [opts...]
                     Cluster Manager command and arguments (see below).
  --verbose          Verbose mode.
  --no-auth-warning  Don't show warning message when using password on command
                     line interface.
  --help             Output this help and exit.
  --version          Output version and exit.

Cluster Manager Commands:
  Use --cluster help to list all available cluster manager commands.

Examples:
  cat /etc/passwd | redis-cli -x set mypasswd
  redis-cli get mypasswd
  redis-cli -r 100 lpush mylist x
  redis-cli -r 100 -i 1 info | grep used_memory_human:
  redis-cli --eval myscript.lua key1 key2 , arg1 arg2 arg3
  redis-cli --scan --pattern '*:12345*'

  (Note: when using --eval the comma separates KEYS[] from ARGV[] items)

When no command is given, redis-cli starts in interactive mode.
Type "help" in interactive mode for information on available commands
and settings.

[redis@artisan bin]$ 



官方提供的工具

Redis进阶-如何发现和优雅的处理BigKey一二事_第2张图片

执行检测

[redis@artisan bin]$ ./redis-cli --bigkeys

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far 'artisan43206' with 12 bytes
[37.40%] Biggest string found so far 'artisanBIG' with 26890 bytes

-------- summary -------

Sampled 100004 keys in the keyspace!
Total key length in bytes is 1188921 (avg len 11.89)

Biggest string found 'artisanBIG' has 26890 bytes

100004 strings with 1215805 bytes (100.00% of keys, avg size 12.16)
0 lists with 0 items (00.00% of keys, avg size 0.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
0 hashs with 0 fields (00.00% of keys, avg size 0.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
[redis@artisan bin]$ 

redis 也是通过scan方式对key进行统计,无需担心对redis造成阻塞 ,从结果上看 发现了 artisanBIG 这个bigkey


debug object

这个命令本身会阻塞Redis,如果 O(N) ,需要关注N的大小。

# 查看artisanBIG序列化后的长度
[redis@artisan bin]$ ./redis-cli debug object artisanBIG
Value at:0x7fa527edd0b0 refcount:1 encoding:raw serializedlength:5044 lru:7863297 lru_seconds_idle:112

输出参数说明:

  • Value at:key的内存地址
  • refcount:引用次数
  • encoding:编码类型
  • serializedlength:序列化长度 5044 —> 5KB ,并不代表真实的大小,仅供参考
  • lru_seconds_idle:空闲时间

Redis进阶-如何发现和优雅的处理BigKey一二事_第3张图片

object命令: https://redis.io/commands/object


如何优雅的删除BigKey (lazy delete)

  • 传统的方式: DEL 呗

Redis进阶-如何发现和优雅的处理BigKey一二事_第4张图片

DEL 毫无疑问,阻塞命令,如果 O(N) N 很大,需要考虑。

  • 另外一点需要考虑的: 有些key 设置了过期时间、rename操作等,这些key redis 自己维护 ,那处理这些BigKey , 同样的会阻塞。 并且这行操作不会记录到慢查询中,你都发现不了。 (慢查询只记录客户端的行为,Redis server本身的行为并不会被记录

  • 推荐: Redis 4.0 提供了 lazy delete (unlink命令)

那4.0 一下的版本,怎么处理这些bigKey呢? -----》 scan , 分批次删除

Redis进阶-如何发现和优雅的处理BigKey一二事_第5张图片

我们重点来看 lazy delete -------------> UNLINK 命令

Lazy Redis is better Redis

UNLINK 官方文档: https://redis.io/commands/unlink
Redis进阶-如何发现和优雅的处理BigKey一二事_第6张图片

127.0.0.1:6379> UNLINK artisanBIG
(integer) 1
127.0.0.1:6379> 


关于lazyfree的几个配置项

Redis进阶-如何发现和优雅的处理BigKey一二事_第7张图片

  • lazyfree-lazy-eviction:是否异步驱逐key,当内存达到上限,分配失败后
  • lazyfree-lazy-expire:是否异步进行key过期事件的处理
  • lazyfree-lazy-server-del:del命令是否异步执行删除操作,类似unlink
  • replica-lazy-flush:replica client做全同步的时候,是否异步flush本地db

默认都是no,按需开启


如何优化bigKey

  • 优化数据结构 : 字符串? hash? 。。。。
  • 合理拆分key : 1个大key 拆分成N个小key
  • 命令优化 : getall–> mget
  • 物理隔离或者升级网卡(治标不治本)

Redis进阶-Redis键值设计及BigKey问题

Redis进阶-如何发现和优雅的处理BigKey一二事_第8张图片

你可能感兴趣的:(【Redis-进阶实战】)