redis-migrate-tool 是一个很方便并且很有用的工具,它用于在redis之间进行数据迁移。
https://github.com/vipshop/redis-migrate-tool
$ git clone https://github.com/vipshop/redis-migrate-tool.git
$ cd redis-migrate-tool
$ yum -y install automake libtool autoconf bzip2
$ autoreconf -fvi
$ ./configure
$ make
$ ./src/redis-migrate-tool
如果能正常输出内容,则说明安装成功
假设现在又如下两个集群:
# 拥有一个key。 700key1: 700value1
127.0.0.1:7001 master
127.0.0.1:7002 master
127.0.0.1:7003 master
127.0.0.1:7004 slave
127.0.0.1:7005 slave
127.0.0.1:7006 slave
# 拥有一个key。 800key1: 800value1
127.0.0.1:8001 master
127.0.0.1:8002 master
127.0.0.1:8003 master
127.0.0.1:8004 slave
127.0.0.1:8005 slave
127.0.0.1:8006 slave
可以使用 create-local-redis-cluster-easily 该项目在本地虚拟机快速搭建一个redis集群。
现在要将7001-7006集群的数据迁移到8001-8006集群。
首先,修改 rmt.conf 文件为:
[source]
type: redis cluster # 源redis类型
servers :
- 127.0.0.1:7001 # 源redis地址,可以只填一个,保证cluster nodes能获取到正常信息即可
[target]
type: redis cluster # 目标redis类型
servers:
- 127.0.0.1:8001 # 目标redis地址
[common]
listen: 0.0.0.0:8888 # 服务启动后,监听的地址和端口
启动迁移工具:
./src/redis-migrate-tool -c rmt.conf
输出这些表示迁移成功:
[root@localhost redis-migrate-tool]# ./src/redis-migrate-tool -c rmt.conf
[2019-10-18 05:05:32.523] rmt_core.c:525 Nodes count of source group : 3
[2019-10-18 05:05:32.523] rmt_core.c:526 Total threads count : 2
[2019-10-18 05:05:32.523] rmt_core.c:527 Read threads count assigned: 1
[2019-10-18 05:05:32.523] rmt_core.c:528 Write threads count assigned: 1
[2019-10-18 05:05:32.524] rmt_core.c:836 instances_by_host:
[2019-10-18 05:05:32.524] rmt_core.c:840 127.0.0.1:7002@17002
[2019-10-18 05:05:32.524] rmt_core.c:840 127.0.0.1:7001@17001
[2019-10-18 05:05:32.524] rmt_core.c:840 127.0.0.1:7003@17003
[2019-10-18 05:05:32.524] rmt_core.c:842
[2019-10-18 05:05:32.524] rmt_core.c:2444 Total threads count in fact: 2
[2019-10-18 05:05:32.524] rmt_core.c:2445 Read threads count in fact: 1
[2019-10-18 05:05:32.524] rmt_core.c:2446 Write threads count in fact: 1
[2019-10-18 05:05:32.524] rmt_core.c:2455 read thread(0):
[2019-10-18 05:05:32.524] rmt_core.c:2461 127.0.0.1:7002@17002
[2019-10-18 05:05:32.524] rmt_core.c:2461 127.0.0.1:7001@17001
[2019-10-18 05:05:32.524] rmt_core.c:2461 127.0.0.1:7003@17003
[2019-10-18 05:05:32.524] rmt_core.c:2488 write thread(0):
[2019-10-18 05:05:32.524] rmt_core.c:2494 127.0.0.1:7002@17002
[2019-10-18 05:05:32.524] rmt_core.c:2494 127.0.0.1:7001@17001
[2019-10-18 05:05:32.524] rmt_core.c:2494 127.0.0.1:7003@17003
[2019-10-18 05:05:32.524] rmt_core.c:2551 migrate job is running...
[2019-10-18 05:05:32.525] rmt_redis.c:1706 Start connecting to MASTER[127.0.0.1:7002@17002].
[2019-10-18 05:05:32.525] rmt_redis.c:1740 Master[127.0.0.1:7002@17002] replied to PING, replication can continue...
[2019-10-18 05:05:32.525] rmt_redis.c:1051 Partial resynchronization for MASTER[127.0.0.1:7002@17002] not possible (no cached master).
[2019-10-18 05:05:32.526] rmt_redis.c:1110 Full resync from MASTER[127.0.0.1:7002@17002]: f00d5c7260ae3652fe82b8c76b57b25e2ada8ed5:11300
[2019-10-18 05:05:32.593] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 200 bytes from master[127.0.0.1:7002@17002]
[2019-10-18 05:05:32.593] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[127.0.0.1:7002@17002] is received, used: 0 s
[2019-10-18 05:05:32.593] rmt_redis.c:1643 rdb file node127.0.0.1:[email protected] write complete
[2019-10-18 05:05:32.594] rmt_redis.c:6601 Rdb file for node[127.0.0.1:7002@17002] parsed finished, use: 0 s.
[2019-10-18 05:05:32.594] rmt_redis.c:1706 Start connecting to MASTER[127.0.0.1:7001@17001].
[2019-10-18 05:05:32.595] rmt_redis.c:1740 Master[127.0.0.1:7001@17001] replied to PING, replication can continue...
[2019-10-18 05:05:32.595] rmt_redis.c:1051 Partial resynchronization for MASTER[127.0.0.1:7001@17001] not possible (no cached master).
[2019-10-18 05:05:32.596] rmt_redis.c:1110 Full resync from MASTER[127.0.0.1:7001@17001]: 56b1e3002b0382acfb526ad6bab81576526b2d62:11523
[2019-10-18 05:05:32.693] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 238 bytes from master[127.0.0.1:7001@17001]
[2019-10-18 05:05:32.693] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[127.0.0.1:7001@17001] is received, used: 0 s
[2019-10-18 05:05:32.693] rmt_redis.c:1643 rdb file node127.0.0.1:[email protected] write complete
[2019-10-18 05:05:32.694] rmt_redis.c:6601 Rdb file for node[127.0.0.1:7001@17001] parsed finished, use: 0 s.
[2019-10-18 05:05:32.695] rmt_redis.c:1706 Start connecting to MASTER[127.0.0.1:7003@17003].
[2019-10-18 05:05:32.696] rmt_redis.c:1740 Master[127.0.0.1:7003@17003] replied to PING, replication can continue...
[2019-10-18 05:05:32.696] rmt_redis.c:1051 Partial resynchronization for MASTER[127.0.0.1:7003@17003] not possible (no cached master).
[2019-10-18 05:05:32.698] rmt_redis.c:1110 Full resync from MASTER[127.0.0.1:7003@17003]: 5ab5f6c86143bcca51d11d55158398e52e6ff8aa:11318
[2019-10-18 05:05:32.794] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 219 bytes from master[127.0.0.1:7003@17003]
[2019-10-18 05:05:32.794] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[127.0.0.1:7003@17003] is received, used: 0 s
[2019-10-18 05:05:32.794] rmt_redis.c:1643 rdb file node127.0.0.1:[email protected] write complete
[2019-10-18 05:05:32.795] rmt_redis.c:6601 Rdb file for node[127.0.0.1:7003@17003] parsed finished, use: 0 s.
[2019-10-18 05:05:32.795] rmt_redis.c:6709 All nodes' rdb file parsed finished for this write thread(0).
此时对7001集群的操作会实时同步到8001集群,实现集群之间的数据同步
参考这篇文章:
https://blog.csdn.net/zhaohongfei_358/article/details/102665418
[root@localhost redis-migrate-tool]# ./src/redis-migrate-tool -h
This is redis-migrate-tool-0.1.0
Usage: redis-migrate-tool [-?hVdIn] [-v verbosity level] [-o output file]
[-c conf file] [-C command]
[-f source address] [-t target address]
[-p pid file] [-m mbuf size] [-r target role]
[-T thread number] [-b buffer size]
Options:
-h, --help : this help 帮助
-V, --version : show version and exit 打印版本
-d, --daemonize : run as a daemon 以守护进程的方式运行
-I, --information : print some useful information 打印一些使用信息
-n, --noreply : don't receive the target redis reply 不接收目标集群回复
-v, --verbosity=N : set logging level (default: 5, min: 0, max: 11) 设置日志级别(默认:5,最小:0,最大11)
-o, --output=S : set logging file (default: stderr) 设置日志文件(默认:stderr)
-c, --conf-file=S : set configuration file (default: rmt.conf) 设置配置文件(默认:rmt.conf)
-p, --pid-file=S : set pid file (default: off) 设置pid文件(默认:off)
-m, --mbuf-size=N : set mbuf size (default: 512) 设置mbuf的大小(默认:512)
-C, --command=S : set command to execute (default: redis_migrate) 设置要执行的命令(默认:redis_migrate)
-r, --source-role=S : set the source role (default: single, you can input: single, twemproxy or redis_cluster) 设置源redis的类型(默认:single,你可以输入:single, twemproxy 或 redis_cluster)
-R, --target-role=S : set the target role (default: single, you can input: single, twemproxy or redis_cluster) 设置目标redis的类型(默认:single,你可以输入:single, twemproxy 或 redis_cluster)
-T, --thread=N : set how many threads to run the job(default: 1) 设置运行该程序的线程数量(默认:1)
-b, --buffer=S : set buffer size to run the job (default: 1048576 byte, unit:G/M/K) 设置运行该程序的缓冲区大小(默认:1048576,单位:G/M/K)
-f, --from=S : set source redis address (default: 127.0.0.1:6379) 设置源redis地址(默认:127.0.0.1:6379)
-t, --to=S : set target redis group address (default: 127.0.0.1:6380) 设置目标redis地址(默认:127.0.0.1:6380)
-s, --step=N : set step (default: 1) 设置step级别(默认:1)
Commands:
redis_migrate : Migrate data from source group to target group.
将数据从源redis迁移到目标redis
redis_check : Compare data between source group and target group. Default compare 1000 keys. You can set a key count behind.
比较源redis和目标redis的数据。默认比较1000个key。你可以在后面设置key的数量
redis_testinsert : Just for test! Insert some string, list, set, zset and hash keys into the source redis group. Default 1000 keys. You can set key type and key count behind.
只是测试用。插入一些string, list, set, zset 和 hash 类型的key到源redis中。默认1000个key,你也可以在后面指定key类型和key的数量。
选项配置的具体含义下面的rmt.conf配置文件中有详细说明
配置文件分为三个部分:source、target和common。
filter支持通配符表达式:
特殊字符使用“\”进行转义。
[source]
type: single
servers:
- 127.0.0.1:6379
- 127.0.0.1:6380
- 127.0.0.1:6381
- 127.0.0.1:6382
[target]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
[common]
listen: 0.0.0.0:8888
threads: 2
step: 1
mbuf_size: 1024
source_safe: true
[source]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6379
- 127.0.0.1:6380
- 127.0.0.1:6381
- 127.0.0.1:6382
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 1
mbuf_size: 512
[source]
type: redis cluster
servers:
- 127.0.0.1:8379
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
filter: abc*
[source]
type: rdb file
servers:
- /data/redis/dump1.rdb
- /data/redis/dump2.rdb
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 2
mbuf_size: 512
source_safe: false
[source]
type: aof file
servers:
- /data/redis/appendonly1.aof
- /data/redis/appendonly2.aof
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 2
你可以使用redis-cli连到redis-migrate-tool上。地址和端口就是common配置中设置的。
例如,你可以使用 info 命令:
$redis-cli -h 127.0.0.1 -p 8888
127.0.0.1:8888> info
# Server
version:0.1.0
os:Linux 2.6.32-573.12.1.el6.x86_64 x86_64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:9199
tcp_port:8888
uptime_in_seconds:1662
uptime_in_days:0
config_file:/ect/rmt.conf
# Clients
connected_clients:1
max_clients_limit:100
total_connections_received:3
# Memory
mem_allocator:jemalloc-4.0.4
# Group
source_nodes_count:32
target_nodes_count:48
# Stats
all_rdb_received:1
all_rdb_parsed:1
all_aof_loaded:0
rdb_received_count:32
rdb_parsed_count:32
aof_loaded_count:0
total_msgs_recv:7753587
total_msgs_sent:7753587
total_net_input_bytes:234636318
total_net_output_bytes:255384129
total_net_input_bytes_human:223.77M
total_net_output_bytes_human:243.55M
total_mbufs_inqueue:0
total_msgs_outqueue:0
127.0.0.1:8888>
该命令会干下面几件事:
参数:
Parameter:
例如,你使用shutdown命令:
$redis-cli -h 127.0.0.1 -p 8888
127.0.0.1:8888> shutdown
OK
在迁移完数据之后,你可以使用 redis_check 命令检查原集群和目标集群的数据。
使用redis_check命令:
$src/redis-migrate-tool -c rmt.conf -o log -C redis_check
Check job is running...
Checked keys: 1000
Inconsistent value keys: 0
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 1000
All keys checked OK!
Check job finished, used 1.041s
如果你想检查更多的key,可以使用下面命令:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_check 200000"
Check job is running...
Checked keys: 200000
Inconsistent value keys: 0
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 200000
All keys checked OK!
Check job finished, used 11.962s
使用 redis_testinsert 命令:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert"
Test insert job is running...
Insert string keys: 200
Insert list keys : 200
Insert set keys : 200
Insert zset keys : 200
Insert hash keys : 200
Insert total keys : 1000
Correct inserted keys: 1000
Test insert job finished, used 0.525s
如果你想插入更多的key:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert 30000"
Test insert job is running...
Insert string keys: 6000
Insert list keys : 6000
Insert set keys : 6000
Insert zset keys : 6000
Insert hash keys : 6000
Insert total keys : 30000
Correct inserted keys: 30000
Test insert job finished, used 15.486s
如果你只想插入string类型的key:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert string"
Test insert job is running...
Insert string keys: 1000
Insert list keys : 0
Insert set keys : 0
Insert zset keys : 0
Insert hash keys : 0
Insert total keys : 1000
Correct inserted keys: 1000
Test insert job finished, used 0.024s
如果你插入一些特定类型的key:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert string|set|list 10000"
Test insert job is running...
Insert string keys: 3336
Insert list keys : 3336
Insert set keys : 3328
Insert zset keys : 0
Insert hash keys : 0
Insert total keys : 10000
Correct inserted keys: 10000
Test insert job finished, used 5.539s