kafka consumer不能消费消息及其处理办法

我这里的Kafka Consumer代码是拷贝网上的，就是开一个线程监听kafka topic，一有消息就处理。开始的代码是这样的：

public void kafkaStart() {
        final String topic = hipchatAction.properties.getProperty("kafka.hipchat.topic");
        final int partitionNum = Integer.valueOf(hipchatAction.properties.getProperty("kafka.hipchat.topic.partitionNum"));

        log.debug("Comes to kafkaStart() with topic : " + topic + ", partitionNum : " + partitionNum);

        Map topicCountMap = new HashMap<>();

        topicCountMap.put(topic, partitionNum);
        Map>> streams = connector.createMessageStreams(topicCountMap);

        List> partitions = streams.get(topic);

        ExecutorService executor = Executors.newFixedThreadPool(partitionNum * 2);

        for (KafkaStream partition : partitions) {
            /**
             * Here also it is an unknown issue, if use anonymous inner class, then the thread seems died, must have
             * a named inner class!
             */
            executor.execute(new MessageRunner(partition));
        }
    }

    class MessageRunner implements Runnable {
        private KafkaStream partition;

        MessageRunner(KafkaStream partition) {
            this.partition = partition;
        }

        public void run() {
            ConsumerIterator it = partition.iterator();
            while (it.hasNext()) {
                MessageAndMetadata msgMeta = it.next();
                /**
                 * In order to consume Chinese message, here should use charset.
                 */
                String jsonStr = new String(msgMeta.message(), StandardCharsets.UTF_8);
                log.debug("********* Message to be consumed in UTF-8 is :: " + jsonStr);

                KafkaMsgVO msg = new Gson().fromJson(jsonStr, KafkaMsgVO.class);
                hipchatAction.sendMessageToRoom(msg.getRoomName(), msg.getToken(), msg.getMsgText());
            }
        }
    }

方法kafkaStart()会在Spring容器启动的时候执行。

对于 KafkaMsgVO msg = new Gson().fromJson(jsonStr, KafkaMsgVO.class); 这行，因为我的程序对消息要求是一个对象名为 KafkaMsgVO 的JSON格式数据，但是测试的时候测试人员就随便发了一条消息，没有进行对象的属性赋值并组装成 JSON 数据，所以抛出 JSONSyntaxException。问题就出在这，因为一旦抛出异常，这个线程就被破坏了，没办法进行后续消息的消费，尽管 kafka consumer 仍然能感知到 topic 里面有新的消息到来。

解决办法：

为了不让异常破坏 consumer 线程，我进行了出错代码的位置转移，将得到的消息直接通过 Apache AKKA 进行转发，然后由 AKKA 的 onReceive() 对消息进行处理，也就是出错代码移到 onReceive() 里面，这样就保证了 consumer 线程的健壮性。代码如下：

class MessageRunner implements Runnable {
        private KafkaStream partition;

        MessageRunner(KafkaStream partition) {
            this.partition = partition;
        }

        public void run() {
            ConsumerIterator it = partition.iterator();
            while (it.hasNext()) {
                MessageAndMetadata msgMeta = it.next();
                /**
                 * In order to consume Chinese message, here should use charset.
                 */
                String jsonStr = new String(msgMeta.message(), StandardCharsets.UTF_8);
                log.debug("********* Message to be consumed in UTF-8 is :: " + jsonStr);

                ActorRef sender = akkaSystem.getMsgConductor();
                sender.tell(new AkkaAdaptor(jsonStr, hipchatAction), sender);
            }
        }
    }

一旦消息被 AKKA 转发后，对消息的处理代码就放在了AKKA 的 Actor 里面了：

} else if (message instanceof AkkaAdaptor) {
		AkkaAdaptor akkaAdaptor = (AkkaAdaptor)message;

		String textMessage = akkaAdaptor.getTextMessage();
		KafkaMsgVO msg = null;
		try {
				msg = new Gson().fromJson(textMessage, KafkaMsgVO.class);
		} catch (Exception e) {
			log.debug(textMessage + " is malformed, it may miss some important property (value).");
			return;
		}

		HipchatAction hipchatAction = akkaAdaptor.getHipchatAction();

		log.debug("Kafka message sent by AKKA is :: " + msg.getMsgText());
		hipchatAction.sendMessageToRoom(msg.getRoomName(), msg.getToken(), msg.getMsgText());

	}