Kafka的Producer报错：Fetching topic metadata with correlation id 0 for topics

问题

在向Kafka生产消息时，报错：Fetching topic metadata with correlation id 0 for topics [Set(game-etl-finished-topic)] from broker [id:0,host:192.168.90.83,port:9093] failed

排查过程

出现问题之前做了一个操作：中断了一次消费程序后再生产数据时报错
具体报错内容如下(IP端口屏蔽了，IP不是localhost)：

 [Fetching topic metadata with correlation id 0 for topics [Set(game-etl-finished-topic)] from broker [id:0,host:*.*.*.*,port:*] failed]
   java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
    at kafka.utils.Utils$.swallow(Utils.scala:172)
    at kafka.utils.Logging$class.swallowError(Logging.scala:106)
    at kafka.utils.Utils$.swallowError(Utils.scala:45)
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
    at kafka.producer.Producer.send(Producer.scala:77)
    at kafka.javaapi.producer.Producer.send(Producer.scala:33)
    at com.hoolai.bi.etlengine.kafka.producer.ETLEngineFinishedProducer.send(ETLEngineFinishedProducer.java:36)
    at com.hoolai.bi.etlengine.engine2.ETLExecuterGameThread.sendToKafka(ETLExecuterGameThread.java:164)
    at com.hoolai.bi.etlengine.engine2.ETLExecuterGameThread.executer(ETLExecuterGameThread.java:125)
    at com.hoolai.bi.etlengine.engine2.ETLExecuterGameThread.run(ETLExecuterGameThread.java:81)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

然后尝试直接从kafka服务器上直接进行生产、消费，但依然报错，内容如下：

//producer的警告，报错信息与上面一致
WARN Property topic is not valid (kafka.utils.VerifiableProperties)
//consumer的警告
WARN [console-consumer-42610_etlengine-1533716213122-4d089dec], no brokers found when trying to rebalance. (kafka.consumer.ZookeeperConsumerConnector)

既然是kafka报的错，那么我们直接去到kafka的logs目录下查看server.log的内容，发现报错内容：

[2018-08-08 15:57:44,241] FATAL [KafkaApi-0] Halting due to unrecoverable I/O error while handling produce request:  (kafka.server.KafkaApis)
kafka.common.KafkaStorageException: I/O exception in append to log 'game-etl-finished-topic-0'
        at kafka.log.Log.append(Log.scala:323)
        at kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379)
        at kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.utils.Utils$.inReadLock(Utils.scala:541)
        at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
        at kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291)
        at kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:93)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:93)
        at scala.collection.Iterator$class.foreach(Iterator.scala:660)
        at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:43)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:93)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:194)
        at scala.collection.mutable.HashMap.map(HashMap.scala:43)
        at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:282)
        at kafka.server.KafkaApis.handleProducerOrOffsetCommitRequest(KafkaApis.scala:204)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /tmp/kafka-logs/game-etl-finished-topic-0/00000000000000000152.index (No such file or directory)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.(RandomAccessFile.java:243)
        at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:277)

根据报错，找到这个目录，看了一下里面的文件，果然没有kafka的相关文件。。。不知道是什么时候被删除的

解决方案

重启Kafka，问题解决