Redis进阶-如何发现和优雅的处理BigKey一二事
文章目录
- PreView
- 模拟写入一个BigKey
- 如何发现BigKey
- redis-cli --bigkeys
- debug object
- 如何优雅的删除BigKey （lazy delete）
- 关于lazyfree的几个配置项
- 如何优化bigKey
PreView
Redis进阶-Redis键值设计及BigKey问题粗略的讨论了BigKey，这里我们继续再来研究下BigKey
Redis Version : 5.0.3
模拟写入一个BigKey
如何发现BigKey
总结下无非几个点
- 应用异常日志 （没法直接从日志关键字中看出来，因为这个bigkey阻塞以后，会引发一些奇怪的问题，比如从连接池获取不到连接等等）
- redis-cli --bigkeys
- 监控系统上报
- 其他…
redis-cli --bigkeys
先看下 redis-cli 都支持什么参数
[redis@artisan bin]$ ./redis-cli --help
redis-cli 5.0.3
Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]
-h <hostname> Server hostname (default: 127.0.0.1).
-p <port> Server port (default: 6379).
-s <socket> Server socket (overrides hostname and port).
-a <password> Password to use when connecting to the server.
You can also use the REDISCLI_AUTH environment
variable to pass this password more safely
(if both are used, this argument takes predecence).
-u <uri> Server URI.
-r <repeat> Execute specified command N times.
-i <interval> When -r is used, waits <interval> seconds per command.
It is possible to specify sub-second times like -i 0.1.
-n <db> Database number.
-x Read last argument from STDIN.
-d <delimiter> Multi-bulk delimiter in for raw formatting (default: \n).
-c Enable cluster mode (follow -ASK and -MOVED redirections).
--raw Use raw formatting for replies (default when STDOUT is
not a tty).
--no-raw Force formatted output even when STDOUT is not a tty.
--csv Output in CSV format.
--stat Print rolling stats about server: mem, clients, ...
--latency Enter a special mode continuously sampling latency.
If you use this mode in an interactive session it runs
forever displaying real-time stats. Otherwise if --raw or
--csv is specified, or if you redirect the output to a non
TTY, it samples the latency for 1 second (you can use
-i to change the interval), then produces a single output
and exits.
--latency-history Like --latency but tracking latency changes over time.
Default time interval is 15 sec. Change it using -i.
--latency-dist Shows latency as a spectrum, requires xterm 256 colors.
Default time interval is 1 sec. Change it using -i.
--lru-test <keys> Simulate a cache workload with an 80-20 distribution.
--replica Simulate a replica showing commands received from the master.
--rdb <filename> Transfer an RDB dump from remote server to local file.
--pipe Transfer raw Redis protocol from stdin to server.
--pipe-timeout <n> In --pipe mode, abort with error if after sending all data.
no reply is received within <n> seconds.
Default timeout: 30. Use 0 to wait forever.
--bigkeys Sample Redis keys looking for big keys.
--hotkeys Sample Redis keys looking for hot keys.
only works when maxmemory-policy is *lfu.
--scan List all keys using the SCAN command.
--pattern <pat> Useful with --scan to specify a SCAN pattern.
--intrinsic-latency <sec> Run a test to measure intrinsic system latency.
The test will run for the specified amount of seconds.
--eval <file> Send an EVAL command using the Lua script at <file>.
--ldb Used with --eval enable the Redis Lua debugger.
--ldb-sync-mode Like --ldb but uses the synchronous Lua debugger, in
this mode the server is blocked and script changes are
not rolled back from the server memory.
--cluster <command> [args...] [opts...]
Cluster Manager command and arguments (see below).
--verbose Verbose mode.
--no-auth-warning Don't show warning message when using password on command
line interface.
--help Output this help and exit.
--version Output version and exit.
Cluster Manager Commands:
Use --cluster help to list all available cluster manager commands.
Examples:
cat /etc/passwd | redis-cli -x set mypasswd
redis-cli get mypasswd
redis-cli -r 100 lpush mylist x
redis-cli -r 100 -i 1 info | grep used_memory_human:
redis-cli --eval myscript.lua key1 key2 , arg1 arg2 arg3
redis-cli --scan --pattern '*:12345*'
(Note: when using --eval the comma separates KEYS[] from ARGV[] items)
When no command is given, redis-cli starts in interactive mode.
Type "help" in interactive mode for information on available commands
and settings.
[redis@artisan bin]$
官方提供的工具
执行检测
[redis@artisan bin]$ ./redis-cli --bigkeys
# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type. You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).
[00.00%] Biggest string found so far 'artisan43206' with 12 bytes
[37.40%] Biggest string found so far 'artisanBIG' with 26890 bytes
-------- summary -------
Sampled 100004 keys in the keyspace!
Total key length in bytes is 1188921 (avg len 11.89)
Biggest string found 'artisanBIG' has 26890 bytes
100004 strings with 1215805 bytes (100.00% of keys, avg size 12.16)
0 lists with 0 items (00.00% of keys, avg size 0.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
0 hashs with 0 fields (00.00% of keys, avg size 0.00)
0 zsets with 0 members (00.00% of keys, avg size 0.00)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
[redis@artisan bin]$
redis 也是通过scan方式对key进行统计，无需担心对redis造成阻塞 ，从结果上看 发现了 artisanBIG 这个bigkey
debug object
这个命令本身会阻塞Redis，如果 O（N） ，需要关注N的大小。
# 查看artisanBIG序列化后的长度
[redis@artisan bin]$ ./redis-cli debug object artisanBIG
Value at:0x7fa527edd0b0 refcount:1 encoding:raw serializedlength:5044 lru:7863297 lru_seconds_idle:112
输出参数说明：
- Value at：key的内存地址
- refcount：引用次数
- encoding：编码类型
- serializedlength：序列化长度 5044 —> 5KB ,并不代表真实的大小，仅供参考
- lru_seconds_idle：空闲时间
- …
object命令： https://redis.io/commands/object
如何优雅的删除BigKey （lazy delete）
- 传统的方式: DEL 呗
DEL 毫无疑问，阻塞命令，如果 O(N) N 很大，需要考虑。
- 另外一点需要考虑的： 有些key 设置了过期时间、rename操作等，这些key redis 自己维护 ，那处理这些BigKey , 同样的会阻塞。 并且这行操作不会记录到慢查询中，你都发现不了。 （慢查询只记录客户端的行为，Redis server本身的行为并不会被记录
- 推荐： Redis 4.0 提供了 lazy delete (unlink命令)
那4.0 一下的版本，怎么处理这些bigKey呢？ -----》 scan , 分批次删除
我们重点来看 lazy delete -------------> UNLINK 命令
Lazy Redis is better Redis
UNLINK 官方文档： https://redis.io/commands/unlink
127.0.0.1:6379> UNLINK artisanBIG
(integer) 1
127.0.0.1:6379>
关于lazyfree的几个配置项
- lazyfree-lazy-eviction：是否异步驱逐key，当内存达到上限，分配失败后
- lazyfree-lazy-expire：是否异步进行key过期事件的处理
- lazyfree-lazy-server-del：del命令是否异步执行删除操作，类似unlink
- replica-lazy-flush：replica client做全同步的时候，是否异步flush本地db
默认都是no，按需开启
如何优化bigKey
- 优化数据结构 : 字符串？ hash? 。。。。
- 合理拆分key ： 1个大key 拆分成N个小key
- 命令优化 ： getall–> mget
- 物理隔离或者升级网卡（治标不治本）
Redis进阶-Redis键值设计及BigKey问题
