Python脚本之操作Redis Cluster【一】
本文为博主原创，未经授权，严禁转载及使用。
本文链接：https://blog.csdn.net/zyooooxie/article/details/123760358
之前曾经分享过2篇 关于Redis命令：https://blog.csdn.net/zyooooxie/article/details/120334491 、 https://blog.csdn.net/zyooooxie/article/details/95352530，这一篇博客是 讲Redis cluster对key相关的操作。
【实际这篇博客推迟发布N个月】
个人博客：https://blog.csdn.net/zyooooxie
Redis集群 命令
CLUSTER INFO
CLUSTER KEYSLOT key
CLUSTER NODES
CLUSTER SLOTS
CLUSTER GETKEYSINSLOT slot count
CLUSTER COUNTKEYSINSLOT slot
想要更加详细学习 可以参考： https://redis.io/commands/?group=cluster
redis-py
https://pypi.org/project/redis/
redis-py now supports cluster mode and provides a client for Redis Cluster.
代码
最初搜 使用Python操作 Redis cluster，所有答案都是redis-py 不支持。但我在看到 https://redis-py.readthedocs.io/en/stable/clustering.html， 我恍惚了，又确认好几遍 是支持的。
我在使用 redis 4.4.1
"""
@blog: https://blog.csdn.net/zyooooxie
@qq: 153132336
@email: [email protected]
"""
import random
from redis.cluster import RedisCluster
from redis.cluster import ClusterNode
from user_log import Log
from XXX_use.common_redis import get_cluster_info, redis_connect
# Connecting redis-py to a Redis Cluster instance(s) requires at a minimum a single node for cluster discovery.
# cluster-enabled : If yes, enables Redis Cluster support in a specific Redis instance. Otherwise the instance starts as a standalone instance as usual.
host2 = ''
p1wd = ''
port = 1234
gl_key_name = 'TEST_xie*'
Log.info('------')
gl_real_string = ''
gl_real_hash = ''
gl_real_list = ''
gl_real_set = ''
gl_no_exist = 'TEST_zyooooxie'
gl_test_str = 'test_str'
gl_test_hash = 'test_hash'
gl_test_list = 'test_list'
gl_test_set = 'test_set'
Log.info('------')
# Redis Cluster provides a way to run a Redis installation where data is automatically sharded across multiple Redis nodes.
# Redis集群提供了一种运行Redis安装的方式，其中数据在多个Redis节点上自动分片。
Log.info('------')
# Redis Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call a hash slot.
# There are 16384 hash slots in Redis Cluster, and to compute the hash slot for a given key, we simply take the CRC16 of the key modulo 16384.
# Redis集群不使用一致哈希，而是使用一种不同形式的分片，其中每个键在概念上都是我们称之为哈希槽的一部分。
# Redis集群中有16384个哈希槽，为了计算给定键的哈希槽，我们只需将键的CRC16模取16384。
# Every node in a Redis Cluster is responsible for a subset of the hash slots, so, for example, you may have a cluster with 3 nodes, where:
#
# Node A contains hash slots from 0 to 5500.
# Node B contains hash slots from 5501 to 11000.
# Node C contains hash slots from 11001 to 16383.
Log.info('------')
# Redis is an in-memory but persistent on disk database,
# so it represents a different trade off where very high write and read speed is achieved with the limitation of data sets that can't be larger than memory.
# Redis是一个内存中的数据库，但是持久化在磁盘上，所以它代表了一种不同的权衡，在数据集不能大于内存的限制下，实现了非常高的写入和读取速度。
Log.info('------')
# When a RedisCluster instance is being created it first attempts to establish a connection to one of the provided startup nodes. If none of the startup nodes are reachable, a ‘RedisClusterException’ will be thrown.
# After a connection to the one of the cluster’s nodes is established, the RedisCluster instance will be initialized with 3 caches:
# a slots cache which maps each of the 16384 slots to the node/s handling them,
# a nodes cache that contains ClusterNode objects (name, host, port, redis connection) for all of the cluster’s nodes,
# and a commands cache contains all the server supported commands that were retrieved using the Redis ‘COMMAND’ output.
# 在创建 RedisCluster 实例时，它首先尝试建立与提供的启动节点之一的连接。如果无法访问任何启动节点，则会抛出“RedisClusterException”。
# 在连接到集群的一个节点后，RedisCluster实例将用3个缓存初始化:
# 一个槽缓存将16384个槽中的每一个映射到处理它们的节点，
# 一个节点缓存包含集群所有节点的ClusterNode对象(名称，主机，端口，redis连接)，
# 一个命令缓存包含所有使用redis“命令”输出检索的服务器支持的命令。
Log.info('------')
# RedisCluster instance can be directly used to execute Redis commands. When a command is being executed through the cluster instance, the target node(s) will be internally determined.
# When using a key-based command, the target node will be the node that holds the key’s slot.
# Cluster management commands and other commands that are not key-based have a parameter called ‘target_nodes’ where you can specify which nodes to execute the command on.
# In the absence of target_nodes, the command will be executed on the default cluster node.
# As part of cluster instance initialization, the cluster’s default node is randomly selected from the cluster’s primaries, and will be updated upon reinitialization.
# Using r.get_default_node(), you can get the cluster’s default node, or you can change it using the ‘set_default_node’ method.
# RedisCluster实例可以直接用于执行 Redis 命令。当通过集群实例执行命令时，目标节点将在内部确定。
# 当使用基于键的命令时，目标节点将是持有键槽的节点。
# 集群管理命令和其他不基于键的命令 有一个名为“target_nodes”的参数，您可以在其中指定要在哪些节点上执行命令。
# 如果没有target_nodes，该命令将在默认集群节点上执行。
# 作为集群实例初始化的一部分，集群的默认节点是从集群的主节点中随机选择的，并将在重新初始化时更新。
# 使用r.get_default_node()，您可以获得集群的默认节点，或者您可以使用' set_default_node '方法更改它。
Log.info('------')
# all non key-based RedisCluster commands accept the kwarg parameter ‘target_nodes’ that specifies the node/nodes that the command should be executed on.
# The best practice is to specify target nodes using RedisCluster class’s node flags: PRIMARIES, REPLICAS, ALL_NODES, RANDOM.
# When a nodes flag is passed along with a command, it will be internally resolved to the relevant node/s.
# If the nodes topology of the cluster changes during the execution of a command, the client will be able to resolve the nodes flag again with the new topology and attempt to retry executing the command.
# 所有非基于键的RedisCluster命令都接受kwarg参数 ' target_nodes '，该参数指定应该在哪个节点上执行命令。
# 最佳实践是使用RedisCluster类的节点标志:primary、REPLICAS、ALL_NODES、RANDOM来指定目标节点。
# 当节点标志与命令一起传递时，它将在内部解析到相关节点。
# 如果集群的节点拓扑在执行命令期间发生变化，客户机将能够使用新的拓扑再次解析节点标志，并尝试重试执行命令。
Log.info('------')
# You could also pass ClusterNodes directly if you want to execute a command on a specific node / node group that isn’t addressed by the nodes flag.
# However, if the command execution fails due to cluster topology changes, a retry attempt will not be made, since the passed target node/s may no longer be valid, and the relevant cluster or connection error will be returned.
# 如果您想在某个特定节点/节点组上执行命令，而该命令不是由nodes标志寻址的，您也可以直接传递ClusterNodes。
# 但是，如果由于集群拓扑更改而导致命令执行失败，则不会进行重试尝试，因为传递的目标节点可能不再有效，并且将返回相关的集群或连接错误。
Log.info('------')
# Redis supports multi-key commands in Cluster Mode, such as Set type unions or intersections, mset and mget, as long as the keys all hash to the same slot.
# By using RedisCluster client, you can use the known functions (e.g. mget, mset) to perform an atomic multi-key operation.
# However, you must ensure all keys are mapped to the same slot, otherwise a RedisClusterException will be thrown.
# Redis支持集群模式下的多键命令，如Set类型 联合或交集，mset和mget，只要键都散列到同一个槽。
# 通过使用RedisCluster客户端，您可以使用已知的函数(例如mget, mset)来执行原子多键操作。
# 但是，您必须确保所有键都映射到相同的槽，否则将抛出RedisClusterException。
Log.info('------')
# To remain available when a subset of master nodes are failing or are not able to communicate with the majority of nodes, Redis Cluster uses a master-replica model where every hash slot has from 1 (the master itself) to N replicas (N-1 additional replica nodes).
# 为了在主节点子集出现故障或无法与大多数节点通信时保持可用，Redis Cluster使用主-复制模型，其中每个哈希槽有从1(主节点本身)到N个副本(N-1个额外的副本节点)。
# Redis 集群对节点使用了主从复制功能： 集群中的每个节点都有 1 个至 N 个复制品（replica）， 其中一个复制品为主节点（master）， 而其余的 N-1 个复制品为从节点（slave）。
Log.info('------')
# Redis Cluster does not guarantee strong consistency. In practical terms this means that under certain conditions it is possible that Redis Cluster will lose writes that were acknowledged by the system to the client.
# Redis集群不保证强一致性。实际上，这意味着在某些情况下，Redis集群可能会丢失系统向客户端确认的写操作。
# The first reason why Redis Cluster can lose writes is because it uses asynchronous replication. This means that during writes the following happens:
#
# Your client writes to the master B.
# The master B replies OK to your client.
# The master B propagates the write to its replicas B1, B2 and B3.
# As you can see, B does not wait for an acknowledgement from B1, B2, B3 before replying to the client, since this would be a prohibitive latency penalty for Redis, so if your client writes something, B acknowledges the write, but crashes before being able to send the write to its replicas, one of the replicas (that did not receive the write) can be promoted to master, losing the write forever.
Log.info('------')
# https://redis.io/docs/connect/clients/python/
# https://redis-py.readthedocs.io/en/v4.4.1/clustering.html
def redis_cluster_connect_1():
"""
:return:
"""
# via the ClusterNode class
nodes = [ClusterNode(host2, port)]
rc = RedisCluster(startup_nodes=nodes, password=pwd, decode_responses=True)
Log.info(rc)
Log.info(type(rc))
Log.info(rc.get_nodes())
Log.info(rc.get_random_node())
Log.info(rc.get_default_node())
Log.info(rc.get_primaries())
Log.info(rc.get_replicas())
Log.error('已连接')
return rc
def redis_cluster_connect_2():
"""
:return:
"""
# Using ‘host’ and ‘port’ arguments
rc = RedisCluster(host=host2, port=port, password=pwd, decode_responses=True)
Log.info(rc)
Log.info(type(rc))
Log.info(rc.info())
Log.info(rc.get_nodes())
Log.info(rc.get_random_node())
Log.info(rc.get_default_node())
Log.info(rc.get_primaries())
Log.info(rc.get_replicas())
Log.error('已连接')
return rc
"""
@blog: https://blog.csdn.net/zyooooxie
@qq: 153132336
@email: [email protected]
"""
def cluster_commands(rc: RedisCluster):
# CLUSTER INFO
Log.info(rc.cluster_info()) # 集群的信息
# CLUSTER SLOTS
Log.info(rc.cluster_slots()) # 查看slot和节点的对应关系
# CLUSTER NODES
Log.info(rc.cluster_nodes()) # 集群当前已知的所有节点（node），以及这些节点的相关信息
Log.info('------')
# CLUSTER KEYSLOT key
# Return the hash slot for .
# Returns the number of keys in the specified Redis Cluster hash slot.
Log.info(rc.cluster_countkeysinslot(exist_key_slot)) # 返回 slot 目前包含的键值对数量
# CLUSTER GETKEYSINSLOT slot count
# Return key names stored by current node in a slot
# The command returns an array of keys names stored in the contacted node and hashing to the specified hash slot.
# The maximum number of keys to return is specified via the count argument,
Log.info(rc.cluster_get_keys_in_slot(exist_key_slot, 0)) # 返回 n 个 slot的键
Log.info(rc.cluster_get_keys_in_slot(exist_key_slot, 1))
Log.info(rc.cluster_get_keys_in_slot(exist_key_slot, 2))
Log.info(rc.cluster_get_keys_in_slot(exist_key_slot, 3))
Log.info('------')
Log.info(rc.cluster_keyslot(gl_real_hash))
Log.info(rc.cluster_keyslot(gl_real_list))
Log.info(rc.cluster_keyslot(gl_real_set))
Log.info(rc.cluster_keyslot(gl_no_exist))
"""
@blog: https://blog.csdn.net/zyooooxie
@qq: 153132336
@email: [email protected]
"""
def cluster_str(rc: RedisCluster):
"""
:param rc:
:return:
"""
# https://redis.io/docs/data-types/strings/
Log.info(rc.delete(gl_test_str))
Log.info(rc.set(gl_test_str, 'https://blog.csdn.net/zyooooxie', ex=1000))
Log.info(rc.get(gl_test_str))
# 类似mset, mget这样的多个key的原生批量操作命令, redis集群只支持所有key落在同一slot的情况
key1 = 'external:XXX1'
key2 = 'external:XXX2'
key3 = 'external:XXX3'
# MSET - all keys must map to the same key slot
Log.info(rc.mset({key1: 'value 1', key2: 'value 2', key3: '3个确定都是相同slot'}))
Log.info(rc.mget(key1, key3, key2))
key4 = 'external:TEST'
# Log.info(rc.mget(key1, key4)) # RedisClusterException: MGET - all keys must map to the same key slot
Log.info('------')
Log.info(rc.unlink(key1, key2, key3, gl_no_exist))
Log.info(rc.unlink(key1, key4, gl_no_exist))
Log.info(rc.exists(gl_test_str))
Log.info(rc.type(gl_test_str))
Log.info(rc.ttl(gl_test_str))
Log.info(rc.expire(gl_test_str, 2 * 60 * 60))
def cluster_hash(rc: RedisCluster):
"""
:param rc:
:return:
"""
# https://redis.io/docs/data-types/hashes/
Log.info(rc.delete(gl_test_hash))
Log.info(rc.hset(gl_test_hash, mapping={'hash_key0': 'hash_value0', 'hash_key1': 'hash_value1',
'hash_key2': 'hash_value2', 'hash_key3': 'hash_value3',
'hk4': 'hv4', 'hk5': 'hv5',
'hk6': 'hv6'
}))
Log.info(rc.hget(gl_test_hash, 'hash_key0'))
Log.info(rc.hgetall(gl_test_hash))
Log.info(rc.hlen(gl_test_hash))
Log.info(rc.hexists(gl_test_hash, 'hash_key2222'))
Log.info(rc.hkeys(gl_test_hash))
Log.info(rc.hvals(gl_test_hash))
Log.info(rc.hdel(gl_test_hash, 'hash_key2222', 'hash_key0', 'hk6'))
Log.info(rc.hmget(gl_test_hash, 'hash_key2222', 'hash_key2'))
Log.info(rc.hmget(gl_test_hash, ['hash_key2222', 'hash_key2']))
# DeprecationWarning: RedisCluster.hmset() is deprecated. Use RedisCluster.hset() instead.
Log.info(rc.hmset(gl_test_hash, {'test': 'test_value', 'test2': 'test_value2'}))
Log.info('------')
Log.info(rc.hset(gl_no_exist, mapping={'test': 'test_value', 'test2': 'test_value2'}))
Log.info(rc.unlink(gl_no_exist))
Log.info(rc.exists(gl_test_hash))
Log.info(rc.type(gl_test_hash))
Log.info(rc.ttl(gl_test_hash))
Log.info(rc.expire(gl_test_hash, 2 * 60 * 60))
def cluster_list(rc: RedisCluster):
"""
:param rc:
:return:
"""
# https://redis.io/docs/data-types/lists/
Log.info(rc.delete(gl_test_list))
Log.info(rc.rpush(gl_test_list, 'list1', 'list2', 'list3'))
Log.info(rc.lindex(gl_test_list, 1))
Log.info(rc.llen(gl_test_list))
Log.info(rc.lpush(gl_test_list, 'list0', 'list0'))
Log.info(rc.linsert(gl_test_list, 'BEFORE', 'list0', 'BEFORE__'))
Log.info(rc.linsert(gl_test_list, 'AFTER', 'list0', 'AFTER__')) # 放在第一个list0 之后
Log.info(rc.lrange(gl_test_list, 0, -1))
Log.info(rc.lpop(gl_test_list))
Log.info(rc.rpop(gl_test_list))
Log.info(rc.lrem(gl_test_list, 1, 'list0'))
Log.info(rc.lset(gl_test_list, 0, '新的_0'))
Log.info(rc.lrange(gl_test_list, 0, -1))
Log.info('------')
Log.info(rc.lpush(gl_no_exist, 0, 'list_0', 1, 'list_1'))
Log.info(rc.unlink(gl_no_exist))
Log.info(rc.type(gl_test_list))
Log.info(rc.exists(gl_test_list))
Log.info(rc.ttl(gl_test_list))
Log.info(rc.expire(gl_test_list, 2 * 60 * 60))
def cluster_set(rc: RedisCluster):
"""
:param rc:
:return:
"""
# https://redis.io/docs/data-types/sets/
Log.info(rc.delete(gl_test_set))
Log.info(rc.sadd(gl_test_set, 'set1', 'set2', 'set3', 'set3', 'set3', 'set3', 'set4'))
Log.info(rc.sismember(gl_test_set, 'set1111'))
Log.info(rc.srem(gl_test_set, 'set1'))
Log.info(rc.scard(gl_test_set))
Log.info(rc.smembers(gl_test_set))
Log.info('------')
Log.info(rc.sadd(gl_no_exist, 'set3', 'set3', 'set3', 'set3'))
Log.info(rc.unlink(gl_no_exist))
Log.info(rc.type(gl_test_set))
Log.info(rc.exists(gl_test_set))
Log.info(rc.ttl(gl_test_set))
Log.info(rc.expire(gl_test_set, 2 * 60 * 60))
"""
@blog: https://blog.csdn.net/zyooooxie
@qq: 153132336
@email: [email protected]
"""
def keys_commands(rc: RedisCluster):
# Another way to iterate over the keyspace is to use the KEYS command, but this approach should be used with care,
# since KEYS will block the Redis server until all keys are returned.
# 另一种遍历键空间的方法是使用KEYS命令，但这种方法应该小心使用，因为KEYS会阻塞Redis服务器，直到所有键都返回。
# target-node: default-node
data_list = rc.keys(gl_key_name)
Log.info(len(data_list))
Log.info(type(data_list))
Log.error('------')
data_list = rc.keys(gl_key_name, target_nodes=RedisCluster.DEFAULT_NODE)
Log.info(len(data_list))
Log.error('------')
data_list = rc.keys(gl_key_name, target_nodes=RedisCluster.RANDOM)
Log.info(len(data_list))
Log.error('------')
# get the keys from all cluster nodes
data_list = rc.keys(gl_key_name, target_nodes=RedisCluster.ALL_NODES)
Log.info(len(data_list))
Log.error('------')
# all primaries
data_list = rc.keys(gl_key_name, target_nodes=RedisCluster.PRIMARIES)
Log.info(len(data_list))
Log.error('------')
# all replicas
data_list = rc.keys(gl_key_name, target_nodes=RedisCluster.REPLICAS)
Log.info(len(data_list))
Log.error('------')
master_list, *args = get_cluster_info(host=host2, port=port, pwd=pwd)
node = master_list[random.randint(0, len(master_list) - 1)]
Log.info(node)
node_ = rc.get_node(*node.split(":"))
# Get the keys only for that specific node
data_list = rc.keys(gl_key_name, target_nodes=node_)
Log.info(len(data_list))
Log.error('------')
data_list = rc.keys(gl_key_name, target_nodes=rc.get_random_node())
Log.info(len(data_list))
Log.error('------')
def scan_iter_method(rc: RedisCluster):
primary_list = rc.get_primaries()
for pl in primary_list:
r = pl.redis_connection
Log.info(r)
data_ = r.scan_iter(match=gl_key_name, count=5000)
Log.info(len(list(data_)))
Log.error('------')
replica_list = rc.get_replicas()
for rl in replica_list:
r = rl.redis_connection
Log.info(r)
data_ = r.scan_iter(match=gl_key_name, count=5000)
Log.info(len(list(data_)))
Log.error('------')
data = rc.scan_iter(match=gl_key_name, count=5000)
Log.info(data)
data_ = list(data)
Log.info(len(data_))
Log.error('------')
def scan_commands(rc: RedisCluster):
# To incrementally iterate over the keys in a Redis database in an efficient manner, you can use the SCAN command.
# 要以有效的方式增量迭代Redis数据库中的键，可以使用SCAN命令。
# Since SCAN allows for incremental iteration, returning only a small number of elements per call,
# it can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements.
# 由于SCAN允许增量迭代，每次调用只返回少量元素，
# 因此可以在生产环境中使用它，而不会像KEYS或SMEMBERS这样的命令那样，在针对大的键或元素集合调用时可能会阻塞服务器很长时间(甚至几秒钟)。
node_ = rc.get_default_node()
nodes_scan(node_.redis_connection)
Log.error('------')
node_ = rc.get_random_node()
nodes_scan(node_.redis_connection)
Log.error('------')
primary_list = rc.get_primaries()
for node in primary_list:
nodes_scan(node.redis_connection)
Log.error('------')
replica_list = rc.get_replicas()
for node in replica_list:
nodes_scan(node.redis_connection)
Log.error('------')
master_list, *args = get_cluster_info(host=host2, port=port, pwd=pwd)
for ml in master_list:
data_list = list()
host_m, port_m = ml.split(":")
r_ = redis_connect(host_m, port_m, pwd)
cursor = 0
while True:
cursor, data = r_.scan(cursor, gl_key_name, count=5000)
data_list.extend(data)
if not bool(cursor):
break
r_.close()
Log.error(len(data_list))
Log.error('------')
node = master_list[random.randint(0, len(master_list) - 1)]
Log.info(node)
node_ = rc.get_node(*node.split(":"))
nodes_scan(node_.redis_connection)
Log.error('------')
def nodes_scan(rc: RedisCluster):
Log.info(f'{rc}')
data_list = list()
cur = 0
while True:
cur, data = rc.scan(cursor=cur, match=gl_key_name, count=5000)
# Log.debug(f'{len(data)}')
# 我试了 几个测试环境的集群，都在报 scan：Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.
# 排查后，发现问题是 第一次使用的scan命令，不清楚 cursor 为啥返回的是 一个dict ？
# 导致后续scan命令 cursor传的值不对啊。
Log.debug(f'{cur}') # 集群的'redis_version': '5.0.5'；使用的redis-py: 4.4.2；
# TODO redis-py 高于4.1.0 已支持 Cluster Mode；
# 稍微低的版本 pip install redis==4.1.0 pip install redis==4.1.2，查的数据 不准确（对比：使用key命令 + 直接在Windows客户端 直接查），但没报错；
# 其他版本 pip install redis==4.2.1 pip install redis==4.3.0 pip install redis==4.3.6 pip install redis==4.5.1 pip install redis==5.0.0 在报这个错；
data_list.extend(data)
if not cur:
break
Log.error(len(data_list))
return data_list
def scan_commands_TEST(rc: RedisCluster):
Log.error('------')
try:
nodes_scan(rc)
except Exception as e:
Log.info(e.args)
if __name__ == '__main__':
pass
Log.error('------')
# rc_m = redis_cluster_connect_1()
rc_m = redis_cluster_connect_2()
# cluster_str(rc_m)
# cluster_hash(rc_m)
# cluster_list(rc_m)
# cluster_set(rc_m)
#
# cluster_commands(rc_m)
#
Log.error(gl_key_name)
# keys_commands(rc_m)
scan_iter_method(rc=rc_m)
# scan_commands(rc=rc_m)
# scan_commands_TEST(rc_m)
Log.error('------')
rc_m.close()
本文链接：https://blog.csdn.net/zyooooxie/article/details/123760358
交流技术 欢迎+QQ 153132336 zy
个人博客 https://blog.csdn.net/zyooooxie