Gossip 协议介绍和Redis集群过大问题

Simply put

The Gossip protocol is a distributed communication protocol used to disseminate information across a network of nodes in a decentralized and fault-tolerant manner. It is commonly used in distributed systems such as peer-to-peer networks, cluster management systems, and in-memory databases.

The Gossip protocol works by having each node periodically exchange information with a random subset of other nodes in the network. Each node maintains a local view of the network state, which is gradually updated as it receives and processes information from other nodes. Nodes may also proactively disseminate information to other nodes if they detect a change in the local state.

The Gossip protocol has several desirable properties, including scalability, fault-tolerance, and eventual consistency. Because each node only communicates with a small subset of other nodes, the protocol can scale to large networks without overwhelming any individual node. Additionally, because the protocol is decentralized, it can tolerate failures or partitions of individual nodes without affecting the overall network.

One of the main challenges of the Gossip protocol is ensuring that information is disseminated efficiently and accurately across the network. To address this challenge, various optimizations and extensions to the basic protocol have been proposed, such as using anti-entropy mechanisms, probabilistic message dissemination, and intelligent message routing algorithms.

redis集群节点过大会发生Gossip 协议广播压力过大

Yes, that is correct. In Redis Cluster, the Gossip protocol is used to communicate cluster state information between nodes. As the number of nodes in the cluster grows, the amount of Gossip communication required to maintain cluster state can become a bottleneck and lead to increased network traffic and latency.
To address this issue, Redis Cluster provides several mechanisms to optimize Gossip communication, including:

1. Cluster Node Hashing: Redis Cluster uses consistent hashing to distribute keys across the cluster. By partitioning the keyspace into smaller subsets, the amount of Gossip communication required to maintain cluster state can be reduced.
2. Node Sampling: Rather than communicating with all nodes in the cluster, Redis Cluster uses a sampling technique to select a subset of nodes for Gossip communication. This reduces the overall communication overhead while still maintaining cluster state consistency.
3. Adaptive Gossip: Redis Cluster uses an adaptive Gossip mechanism that adjusts the frequency and scope of Gossip communication based on the current state of the cluster. This helps to reduce unnecessary communication and improve overall cluster performance.
   Despite these optimizations, Redis Cluster can still experience Gossip-related performance issues when the number of nodes in the cluster is very large. In such cases, it may be necessary to consider alternative cluster architectures or partitioning strategies to better manage network traffic and latency.

代码实例

First, Redis Cluster maintains a data structure called clusterNode to represent each node in the cluster. The clusterNode structure includes information such as the node ID, IP address, port number, and other metadata.

When a new node joins the cluster, it sends a MEET command to one or more existing nodes to announce its presence. Each receiving node then adds the new node to its own clusterNode table and sends a PING command to the new node to confirm its existence.

Every node in the cluster regularly sends PING commands to a random subset of other nodes in the cluster to check their health and update their own clusterNode table. If a node fails to respond to several PING commands, it is marked as failed and removed from the clusterNode table.

Nodes also exchange information about their clusterNode tables using the GOSSIP command. A node randomly selects one or more nodes to gossip with and sends them a message containing its own clusterNode table. The receiving nodes merge the received clusterNode table with their own and propagate the updated table to other nodes in the cluster.

Here is an example of how a PING command is sent to a random node:

void clusterSendPing(void *arg) {
    clusterNode *node = arg;
    redisAssert(node->flags & (REDIS_NODE_MYSELF|REDIS_NODE_HANDSHAKE));
    if (node->flags & REDIS_NODE_MYSELF) {
        redisLog(REDIS_DEBUG,"Skipping self PING %s:%d myself:%d",
            node->ip, node->port, node->flags & REDIS_NODE_MYSELF);
        return;
    }
    redisLog(REDIS_DEBUG,"Sending PING to node %s:%d",
        node->ip, node->port);
    clusterSendMessage(node, CLUSTERMSG_TYPE_PING, NULL, 0);
}

And here is an example of how a GOSSIP command is sent to a random subset of nodes:

void clusterSendGossip(void *arg) {
    clusterNode *node = arg;
    int j, max, count = 0, sent = 0;
    clusterMsg buf[1];
    unsigned char gossip_sent_table[CLUSTER_SLOTS/8+1];

    /* Don't attempt gossip if too many cluster messages are queued in
     * the output buffers. */
    if (listLength(server.cluster->msg_queue) > CLUSTERMSG_MAXLEN*2) return;

    /* Populate our gossip message with random entries selected from our
     * own state table. */
    memset(gossip_sent_table,0,sizeof(gossip_sent_table));
    buf->type = htons(CLUSTERMSG_TYPE_GOSSIP);
    buf->count = htons(clusterCountNonFailingNodes());
    if (nodeIsSlave(node)) {
        clusterNode *master = node->slaveof;
        redisAssert(master != NULL);
        clusterMsgDataGossip(buf) = master->name;
        clusterMsgDataGossipLen(buf) = REDIS_CLUSTER_NAMELEN;
    } else {
        clusterMsgDataGossip(buf) = server.cluster->myself->name;
        clusterMsgDataGossipLen(buf) = REDIS_CLUSTER_NAMELEN;
    }
    if (node->flags & REDIS_NODE_PROTECTED) buf->flags |= htons(CLUSTERMSG_FLAG0_PROTECTED);

    if (server.cluster->myself->numslots > 0)
        clusterMsgDataGossipSlot(buf) = htons(server.cluster->myself->slots[0]);
    else
        clusterMsgDataGossipSlot(buf) = htons(0);

    max = ceil(server.cluster->size / 10.0);
    if (max < 1) max = 1;
    for (j = 0; j < server.cluster->size && sent < max; j++) {
        clusterNode *this = server.cluster->nodes+j;

        if (this == node || clusterNodeIsFailing(this)) continue;
        if (this->flags & (REDIS_NODE_MYSELF|REDIS_NODE_HANDSHAKE)) continue;
        if (gossip_sent_table[this->slots[0]/8] & (1<<(this->slots[0]&7))) continue;

        if (++count > 3) break; /* We need to gossip with at least three nodes. */
        gossip_sent_table[this->slots[0]/8] |= 1<<(this->slots[0]&7);
        memcpy(buf->targets[sent].name,this->name,REDIS_CLUSTER_NAMELEN);
        sent++;
    }
    buf->size = htons(sizeof(clusterMsg)-sizeof(union clusterMsgData));

    clusterSendMessage(node, CLUSTERMSG_TYPE_GOSSIP,
        buf, sizeof(clusterMsg)-sizeof(union clusterMsgData)+(sent*sizeof(clusterMsgTarget)));
}

These are just snippets of code from the larger Redis source codebase, but they demonstrate how the Gossip protocol is implemented in Redis Cluster.

参考

https://blog.csdn.net/DiDi_Tech/article/details/130896033?spm=1000.2115.3001.5927