kafka 单实例安装

环境

• centos 6.5
• jdk 1.8
• zookeeper 3.5 伪集群
• kafka 2.12

kafka需要zookeeper集群,否则报

1 partitions have leader brokers without a matching listener

下载解压

$ cd /var/tmp
$ wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz
$ tar -xzf kafka_2.12-2.3.0.tgz
$ mv kafka_2.12-2.3.0 /usr/local/kafka

环境变量

$ vi /etc/profile.d/kafka.sh
export KAFKA_HOME=/usr/local/kafka
export PATH=$PATH:$KAFKA_HOME/bin

$ source /etc/profile

配置

$ vi /usr/local/kafka/config/server.properties
broker.id=0
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

配置需要listeners,否则报

Error while fetching metadata with correlation id

启动报错解决

启动命令会报错

sed: invalid option -- 'E'

解决

$ vi /usr/local/kafka/bin/kafka-run-class.sh
JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
改为
JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -n '1p'|awk -F '.' '{print $2}')

自动搜索server.properties

$ vi /usr/local/kafka/bin/kafka-server-start.sh
# 在EXTRA_ARGS后添加
CONFDIR="${base_dir}/../config"
COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    if [ -f $2 ];then
        CONFPATH=$2
    else
        CONFPATH=$CONFDIR"/"$2
    fi
    shift
    shift
    ;;
  *)
    if [ -f $1 ];then
        CONFPATH=$1
    else
        CONFPATH=$CONFDIR"/"$1
    fi
    shift	  
    ;;
esac
exec  $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka $CONFPATH  "$@"

在CDH环境stop不起作用

$ vi /usr/local/kafka/bin/kafka-server-stop.sh
PIDS=$(ps ax | grep -i ' kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}')
修改为
PIDS=$(ps ax | grep -i 'kafkaServer' | grep java | grep -v grep | awk '{print $1}')

启动kafka服务

//# 要先启动zookeeper
$ zkServer.sh start

//# debug方式启动kakfa
$ kafka-server-start.sh server.properties

//# 守护进程的方式启动
$ kafka-server-start.sh -daemon  server.properties

//# 查看端口
$ netstat -tnlp|grep java
:::9092
:::28119

停止

$ kafka-server-stop.sh

kafka会在zookeeper里保存集群信息

$ $ zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTING) 0] ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, isr_change_notification, latest_producer_id_block, log_dir_event_notification]