分布式消息中间件（七）——Kafka安装及配置详解（Linux）

一、Zookeeper集群准备

Kafka服务有赖于Zookeeper来管理，故在安装kakfa前先安装zk集群环境。具体安装步骤，详见 Linux 系列（七）——Zookeeper集群搭建

二、Kafka核心配置解析 

1、上传压缩包，解压

2、解压后，cd /lamp/kafka/config文件下，查看kafka所有相关配置

-rw-r--r--. 1 root root  906 May 17 21:26 connect-console-sink.properties
-rw-r--r--. 1 root root  909 May 17 21:26 connect-console-source.properties
-rw-r--r--. 1 root root 2760 May 17 21:26 connect-distributed.properties
-rw-r--r--. 1 root root  883 May 17 21:26 connect-file-sink.properties
-rw-r--r--. 1 root root  881 May 17 21:26 connect-file-source.properties
-rw-r--r--. 1 root root 1074 May 17 21:26 connect-log4j.properties
-rw-r--r--. 1 root root 2061 May 17 21:26 connect-standalone.properties
-rw-r--r--. 1 root root 1199 May 17 21:26 consumer.properties
-rw-r--r--. 1 root root 4369 May 17 21:26 log4j.properties
-rw-r--r--. 1 root root 1900 May 17 21:26 producer.properties
-rw-r--r--. 1 root root 5243 May 17 21:26 server.properties
-rw-r--r--. 1 root root 1032 May 17 21:26 tools-log4j.properties
-rw-r--r--. 1 root root 1041 Jun 18 11:19 zookeeper.properties

zookeeper.properties

# the directory where the snapshot is stored.
dataDir=/usr/local/cloud/zookeeper1/data
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0

server.properties

############################# Server Basics 指定broker ID，每个kafka实例拥有一个唯一brokerID作为标识#############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Log Basics  kafka日志持久化日志文件路径，生产环境指定到非tmp临时文件夹下#############################

# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1  # 配置本broker topic 分区个数

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.220.128:2181 #zk服务地址

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000  #zk响应过期时间

     server.properties是kafka broker的配置文件，主要用于配置broker的日志、分区、zk连接信息。除了上述提供的主要配置之外，还有Log Retention Policy日志保留策略（以总时间、segment大小、间隔时间配置）、Log Flush Policy 日志flush策略 ，日志中存有10000条记录开始持久化操作、每隔1000ms执行持久化，通过配置可调可配，非常灵活。

producer.properties

############################# Producer Basics #############################

# list of brokers used for bootstrapping knowledge about the rest of the cluster
# format: host1:port1,host2:port2 ...
bootstrap.servers=192.168.220.128:9092

# specify the compression codec for all data generated: none, gzip, snappy, lz4
compression.type=none

consumer.properties

# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
zookeeper.connect=127.0.0.1:2181

# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

#consumer group id
group.id=test-consumer-group

#consumer timeout
#consumer.timeout.ms=5000

三、Kafka常用操作

1、kafka启动

    启动kafka前先启动zk服务：  bin/zookeeper-server-start.sh  /usr/local/cloud/zookeeper1/conf/zoo.cfg --指定zk的配置文件

    启动kafka： bin/kafka-server-start.sh /lamp/kafka/config/server.properties >/dev/null 2>&1 &  --后台启动kafka

    执行jsp查看启动进程如下

2、创建topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic max

在创建完topic后，可查看kafka日志持久化路径，已经创建好刚才的log日志文件用于记录topic中parttion信息。（log ：partition=1：1）

3、查看topic

查看topic中list：bin/kafka-topics.sh --list --zookeeper localhost:2181  --返回以创建的topic名称，如max，linxi 

查看某topic 详情信息：bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic max

4、删除topic

bin/kafka-run-class.sh kafka.admin.TopicCommand –delete --topic max --zookeeper 192.168.220.128:2181

5、创建生产者

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic max  向名为max的topic中生产消息

6、创建消费者

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic max --from-beginning

生产者和消费间消息是实时的，这边生产，另一端立马开始消费。

另外在zk中记录了partion消息的消费记录信息，进入zk客户端连接服务，进行查看

三、Kafka集群

     Kafka的集群与zk集群类似，灰常简单，因为已经搭建好zk集群，kafka只需创建多个实例，修改上述server.properties中broker.id 不重复即可，and 在zk connection处添加上三个zk实例即可。

四、客户端实现

/*
*消息生产者
*/
public class Producer extends Thread{
	private String topic;
	public Producer(String topic)
	{
		super();
		this.topic=topic;
	}
	
	@Override
	public void run() {
		Producer producer=createProducer();
		producer.send(new KeyedMessage<>(topic, "xiaowei"));
	}
	
	private Producer createProducer()
	{
		Properties props=new Properties();
		props.setProperty("zookeeper.connect", "192.168.220.128:2181");
		props.setProperty("serializer.class", StringEncoder.class.getName());
		props.setProperty("metadata.broker.list", "192.168.220.128:9092");//p.properties的配置中的端口号
		return new Producer(new ProducerConfig(props));
	}
	
	public static void main(String[] args) {
		new Producer("max").start();
	}
}

/*
*消息消费者
*/
public class Consumer  extends Thread{
	private String topic;	
	public Consumer(String topic)
	{
		super();
		this.topic=topic;
	}
	@Override
	public void run() {
		//通过props创建consumer连接		
		ConsumerConnector consumer=createConsumer();
		//创建topicMap存放指定消费的topic名，分区号
		Map<String,Integer> topicMap=new HashMap<String,Integer>();
		topicMap.put(topic, 1);
		
		//msgstreams-topic中的所有消息，kafkaStream-某一条消息
		Map<String,List<KafkaStream<byte[],byte[]>>> msgStreams=consumer.createMessageStreams(topicMap);
		KafkaStream<byte[],byte[]> kafkaStream=msgStreams.get(topic).get(0);
		
		//数据处理过程
		ConsumerIterator<byte[],byte[]> iterator=kafkaStream.iterator();
		while(iterator.hasNext())
		{
			byte[] msg=iterator.next.message();
			System.out.println("message is" +new String(msg));
		}
	}
	private ConsumerConnector createConsumer()
	{
		Properties props=new Properties();
		props.setProperty("zookeeper.connect", "192.168.220.128:2181");
		props.setProperty("group.id", "max");
		return Consumer.createJavaConsumerConnector(new Consumer());
		
	}
	public static void main(String[] args)
	{
		new Consumer("max").start();
	}
}

    Kafka实际应用的消费者和生产者都属于某些项目或系统，例如在大数据应用中，使用flume收集信息，flume的sink配置kafka接收，kafka高速接收缓冲这些数据，storm从kafka获取，进行数据处理；实际flume就充当kafka的消息生产者，storm为消息消费者。