解决SpringBoot连接 Kafka失败 [Producer clientId=producer-1][Consumer clientId=producer-1]

小白一枚，在用SpringBoot创建produce连接kafka伪集群的时候，老是启动失败

为什么说是伪集群呢，因为是虚拟机啊  vmware中开了三个虚拟机。

循环打印的错误日志如下

15:05:48.995 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updated cluster metadata updateVersion 10 to MetadataCache{cluster=Cluster(id = ckRm4MrXT-W9HzytSM8Fig, nodes = [slave2:9092 (id: 2 rack: null), slave1:9092 (id: 1 rack: null), masterLinux:9092 (id: 0 rack: null)], partitions = [Partition(topic = firstTopic, partition = 0, leader = 2, replicas = [2,0,1], isr = [2,0,1], offlineReplicas = []), Partition(topic = firstTopic, partition = 1, leader = 0, replicas = [0,1,2], isr = [0,1,2], offlineReplicas = [])], controller = slave1:9092 (id: 1 rack: null))}
15:05:49.053 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1] Got error produce response with correlation id 21 on topic-partition firstTopic-0, retrying (2147483638 attempts left). Error: NOT_LEADER_FOR_PARTITION
15:05:49.053 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1] Received invalid metadata error in produce request on partition firstTopic-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now
15:05:49.095 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Sending metadata request MetadataRequestData(topics=[MetadataRequestTopic(name='firstTopic')], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to node slave2:9092 (id: 2 rack: null)
15:05:49.095 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updating last seen epoch from 0 to 0 for partition firstTopic-0
15:05:49.096 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updating last seen epoch from 0 to 0 for partition firstTopic-1
15:05:49.096 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updated cluster metadata updateVersion 11 to MetadataCache{cluster=Cluster(id = ckRm4MrXT-W9HzytSM8Fig, nodes = [slave1:9092 (id: 1 rack: null), slave2:9092 (id: 2 rack: null), masterLinux:9092 (id: 0 rack: null)], partitions = [Partition(topic = firstTopic, partition = 0, leader = 2, replicas = [2,0,1], isr = [2,0,1], offlineReplicas = []), Partition(topic = firstTopic, partition = 1, leader = 0, replicas = [0,1,2], isr = [0,1,2], offlineReplicas = [])], controller = slave1:9092 (id: 1 rack: null))}
15:05:49.155 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1] Got error produce response with correlation id 23 on topic-partition firstTopic-0, retrying (2147483637 attempts left). Error: NOT_LEADER_FOR_PARTITION
15:05:49.155 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1] Received invalid metadata error in produce request on partition firstTopic-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now
15:05:49.196 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Sending metadata request MetadataRequestData(topics=[MetadataRequestTopic(name='firstTopic')], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to node slave2:9092 (id: 2 rack: null)
15:05:49.200 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updating last seen epoch from 0 to 0 for partition firstTopic-0
15:05:49.201 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updating last seen epoch from 0 to 0 for partition firstTopic-1
15:05:49.202 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updated cluster metadata updateVersion 12 to MetadataCache{cluster=Cluster(id = ckRm4MrXT-W9HzytSM8Fig, nodes = [masterLinux:9092 (id: 0 rack: null), slave2:9092 (id: 2 rack: null), slave1:9092 (id: 1 rack: null)], partitions = [Partition(topic = firstTopic, partition = 0, leader = 2, replicas = [2,0,1], isr = [2,0,1], offlineReplicas = []), Partition(topic = firstTopic, partition = 1, leader = 0, replicas = [0,1,2], isr = [0,1,2], offlineReplicas = [])], controller = slave1:9092 (id: 1 rack: null))}
15:05:49.259 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1] Got error produce response with correlation id 25 on topic-partition firstTopic-0, retrying (2147483636 attempts left). Error: NOT_LEADER_FOR_PARTITION
15:05:49.259 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.producer.internals.Sender - [Producer clientId=producer-1] Received invalid metadata error in produce request on partition firstTopic-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now
15:05:49.299 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Sending metadata request MetadataRequestData(topics=[MetadataRequestTopic(name='firstTopic')], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to node masterLinux:9092 (id: 0 rack: null)
15:05:49.300 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updating last seen epoch from 0 to 0 for partition firstTopic-0
15:05:49.300 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Updating last seen epoch from 0 to 0 for partition firstTopic-1

 

先来说怎么解决的吧！如果kafka集群上的hosts文件修改过，直接看第二步

 一、kafka集群

各个linux主机上也修改host文件  vim /etc/hosts   （这一步不设置的话，kafka集群应该是启动失败，找不到彼此）   

我的集群是这样设置的，在hosts文件最后面 追加如下配置：

最好是把IP设置为static的，  默认bootproto=dhcp重启机器IP地址会变

           如何更改linux操作系统IP地址为静态   网上一搜一大把！

#根据自己linux网络去修改hosts配置文件，只要能ping 域名   ping通就OK   
192.168.200.135 masterLinux
192.168.200.138 slave1
192.168.200.137 slave2

如果设置为静态获取IP地址，一定要去物理机修改虚拟网卡的TCP/IPv4这个属性！！！

（血淋淋的教训！ 一天就这样过去了。。。。）

二、修改本地操作系统的hosts文件

 windows是C:\Windows\System32\drivers\etc\hosts       

 MacOS  网上搜下把，修改hosts文件即可。

追加一下kafka集群的各个ip地址
192.168.200.135 masterLinux
192.168.200.138 slave1
192.168.200.137 slave2

三、修改kafka集群中server.properties,在kafka根目录的config文件夹中

这句代码是注解的：advertised.listeners=PLAINTEXT://your.host.name:9092

根据当前机器的域名修改一下即可，每台都需要改！

代码也贴出来吧！

第一步肯定是在pom.xml中先导入kafka引用

        
            org.apache.kafka
            kafka-clients
            2.3.0
        
        
            org.apache.kafka
            kafka_2.12
            2.3.0
        

 

然后编写一个测试类，直接运行即可。

package com.test.kafka.Producer;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.util.Properties;
public class CustomerProducer {
    
    public static void main(String[] args) {
         //配置信息
        Properties props = new Properties();
        //kafka集群的信息
        props.put("bootstrap.servers","masterLinux:9092,slave1:9092,slave2:9092");
        //ack应答级别
        props.put("acks", "all");
        
        //KV的序列化类
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        //
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

//        ProducerConfig;  kafka的produce配置类

        //创建生产者对象
        KafkaProducer producer = new KafkaProducer<>(props);
        //循环发送数据，“testTopic”是已经创建好的topic
        for (int i = 0; i < 10; i++)
            producer.send(new ProducerRecord<>("testTopic", Integer.toString(i)+"message"));

        producer.close();


    }
}

想看到效果，那必须得有一个topic  and  consumer，我用的是xshell操作的

• 创建一个topic  打开kafka任意一台机器，使用如下命令创建一个topic

在kafka的根目录下 执行如下指令
bin/kafka-topics.sh --zookeeper masterLinux:2181 --create --replication-factor 3 --partitions 1 --topic testTopic

• 可以先创建一个produce去用这个topic发送一下消息

[root@slave2 kafka]# bin/kafka-console-producer.sh --broker-list masterLinux:9092 --topic testTopic
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>hi kafka

• 创建一个consumer去消费这个topic

bin/kafka-console-consumer.sh --bootstrap-server masterLinux:9092 --from-beginning --topic testTopic

代码执行后

[root@masterLinux kafka]# bin/kafka-console-consumer.sh --bootstrap-server masterLinux:9092 --from-beginning --topic testTopic
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hi kafka
0
1
2
3
4
5
6
7
8
9

如果还循环报错

        ctrl+c 把kafka集群停掉，把kafka根目录下的logs文件夹删除：rm -rf logs （数据会丢失）

        重新创建这个logs目录

        重启kafka集群,在kafka根目录下：[root@masterLinux kafka]# bin/kafka-server-start.sh config/server.properties

 我目前是这样解决掉问题的，如果有更好的解决方法   欢迎评论！

 

 

ps：在发布这篇博文之后的隔天，物理机连接虚拟机kafka集群，又出现了连接不上，找不到Leader的情况，懵逼！

在阿里云服务器上搭建了个单机的kafka之后~  完美运行，不知道是vmware的锅，还是因为linux是copy了两份的锅！