通过xargs实现redis cluster批量keys操作

通过xargs实现redis cluster批量keys操作
为什么会出现批量删除keys的脚本，那是因为redis不支持删除（DEL）带有通配符“*”的key，而查询支持带通配符“*”。

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD KEYS hello*
1) "hello8"
2) "hello5"
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD DEL hello*
(integer) 0

工具简介

xargs

This manual page documents the GNU version of xargs.  xargs reads items from the standard input, delimited by blanks (which can be  protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by items read from standard input.  Blank lines on the standard input are ignored.

The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used).  The specified com‐mand  will be invoked as many times as necessary to use up the list of input items.  In general, there will be many fewer invocations of com‐mand than there were items in the input.  This will normally have significant performance benefits.  Some commands can usefully  be  executed in parallel too; see the -P option.

Because  Unix filenames can contain blanks and newlines, this default behaviour is often problematic; filenames containing blanks and/or new‐lines are incorrectly processed by xargs.  In these situations it is better to use the -0 option, which prevents such problems.   When  using this  option  you will need to ensure that the program which produces the input for xargs also uses a null character as a separator.  If that program is GNU find for example, the -print0 option does this for you.

If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input.  An error message is issued on stderr when this happens.

redis-cli

Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]
      -h       Server hostname (default: 127.0.0.1).
      -p           Server port (default: 6379).
      -s         Server socket (overrides hostname and port).
      -a       Password to use when connecting to the server.
      -r         Execute specified command N times.
      -i       When -r is used, waits  seconds per command.
                         It is possible to specify sub-second times like -i 0.1.
      -n             Database number.
      -x                 Read last argument from STDIN.
      -d      Multi-bulk delimiter in for raw formatting (default: \n).
      -c                 Enable cluster mode (follow -ASK and -MOVED redirections).
      --scan             List all keys using the SCAN command.
      --pattern     Useful with --scan to specify a SCAN pattern.

key-value操作

查询

KEYS查询

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD KEYS hello*
1) "hello8"
2) "hello5"
3) "hello1"

GET值查询

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD GET hello8
"955"
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD GET hello5
"555"
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD GET hello1
"999"

SET重置

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD SET hello1 666
OK
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD GET hello1
"666"

DEL删除

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD DEL hello1
(integer) 1
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD GET hello1
(nil)
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD KEYS hello1
(empty list or set)

通过xargs实现批量删除

第一种方式：

redis-cli -a "password" -n 0 -p 6379 keys "*:login" | xargs -i redis-cli -a "password" -n 0 -p 6379 del {}

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 KEYS hello*
1) "hello8"
2) "hello5"
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 KEYS "hello*" | xargs -i redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 DEL {}
(integer) 1
(integer) 1
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 KEYS hello*
(empty list or set)

这样一个坏处每次都要建立一个连接，量小的话还可以接受，量大的话，效率不行。

第二种方式：

自从redis2.8以后就开始支持scan命令，模式匹配可以采取下面的形式来批删除大量的key

redis-cli -a "password" -n 0 -p 6379 --scan --pattern "*:login" | xargs -i redis-cli -a "password" -n 0 -p 6379 DEL {}

[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 --scan --pattern "hello*"
hello1
hello4
hello5
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 --scan --pattern "hello*" | xargs -i redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 DEL {}
(integer) 1
(integer) 1
(integer) 1
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 --scan --pattern "hello*"
[root@localhost ~]# redis-cli -c -h 10.1.37.113 -p 7002 -a PASSWORD -n 0 KEYS "hello*"
(empty list or set)

速度处理也是非常快的。

第三种方式：

redis-cli -c -h $ip -p $port keys $pkey | xargs -r -t -n1 redis-cli -c -h $ip -p $port del

批量删除kyes脚本

脚本造key测试

#!/bin/bash

redis_list=("10.1.37.113:7002")

pkey_list=("ValuationRuleSummary" "ValuationRuleDetail" "MerchantValuation" "QueryValuationRules" "GetMerchantByUserId")

for info in ${redis_list[@]}
do
    echo "开始执行:$info"  
    ip=`echo $info | cut -d \: -f 1`
    port=`echo $info | cut -d \: -f 2`

    for pkey in ${pkey_list[@]}
    do
        for i in `seq 0 10000`
        do
            echo $pkey"_"$i     $i
            redis-cli -c -h $ip -p $port -a PASSWORD SET $pkey"_"$i $i
        done
    done
done
echo "完成"

第一种批量删除方式

#!/bin/bash

redis_list=("10.1.37.113:7000" "10.1.37.113:7001" "10.1.37.113:7002" "10.1.37.113:7003" "10.1.37.113:7004" "10.1.37.113:7005")

pkey_list=("ValuationRuleSummary*" "ValuationRuleDetail*" "MerchantValuation*" "QueryValuationRules*" "GetMerchantByUserId*")

for info in ${redis_list[@]}
    do
        echo "开始执行:$info"  
        ip=`echo $info | cut -d \: -f 1`
        port=`echo $info | cut -d \: -f 2`

        for pkey in ${pkey_list[@]}
        do
            redis-cli -c -h $ip -p $port -a PASSWORD KEYS $pkey | xargs -n 1 -t -i redis-cli -c -h $ip -p $port -a PASSWORD DEL {}
        done
    done
echo "完成"

通过keys文件列表操作

key.txt –待删除的key

key1
key2
key3

redis_delete_key.sh
redis_list 填入集群节点

redis_list=("127.0.0.1:6379" "127.0.0.1:6380")

for info in ${redis_list[@]}
    do  
        echo "开始执行:$info"  
        ip=`echo $info | cut -d \: -f 1`
        port=`echo $info | cut -d \: -f 2`
        cat key.txt |xargs -t -n1 redis-cli -h $ip -p $port -c del  
    done 
    echo "完成"

总结

这种方式怎么删除还是要看xargs怎么使用了，xargs用什么参数看自己需求，详细请man查看。
但是这种方式还是比较慢，当keys非常巨大时，就能体现出来有多慢了，有可能你删一天都删不完，详细可以查看redis  DEL  命令的时间复杂度。
要想加快速度怎么办？还得使用程序进行，或者使用 Lua 脚本等方法，毕竟shell脚本还是一条一条去删除的。

请大佬看到多多批评指正，献丑了！