Linux 安装 kafka(单机集群)

前言

因为个人比较穷,所以只用一台服务器作为示例,请大家见谅,只要大家会举一反三,这些都不是问题,写得不好的地方,请大家多多包涵,阿导不胜感激。

准备工作

  • centos 7
  • zookeeper
  • kafka
    因为花名阿导,所以我会在根目录创建一个文件夹 dao,然后创建 zookeeper 和 kafka 两个文件夹,具体命令如下所示:
cd /
mkdir dao
cd dao
mkdir zookeeper kafka

zookeeper

好了,话不多说,先来一波 zookeeper 的配置,因为过去搭建过主从结构的 zookeeper 集群,我这边文件夹命名没有去改动,按照道理来说,这三个 zookeeper 节点地位相当,没有主次之分,我这边创建了8个文件夹,首先在 zookeeper 目录下创建 data 和 logs,然后分别在data 和 logs 下面创建 master 、slave_1 、slave_2,命令如下所示:

cd /dao/zookeeper
mkdir data logs
cd data
mkdir master slave_1 slave_2
cd …/logs
mkdir master slave_1 slave_2

这个时候需要下载一个 zookeeper,然后解压,拷贝三份,具体操作如下:

cd /dao/zookeeper
wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
tar -zxvf zookeeper-3.4.10.tar.gz
cp -r zookeeper-3.4.10 zookeeper_master
cp -r zookeeper-3.4.10 zookeeper_slave_1
mv zookeeper-3.4.10 zookeeper_slave_2

目录如下图
Linux 安装 kafka(单机集群)_第1张图片
Linux 安装 kafka(单机集群)_第2张图片
Linux 安装 kafka(单机集群)_第3张图片

嗯,接下来看看 zookeeper 重要的配置了,首先在 data 下面三个文件夹内添加 myid 文件,文件内容分别为 1,2,3,命令如下:

vi /dao/zookeeper/data/master/myid

输入 i ,然后 1,最后保存退出(wq)

vi /dao/zookeeper/data/slave_1/myid

输入 i ,然后 2,最后保存退出(wq)

vi /dao/zookeeper/data/slave_2/myid

输入 i ,然后 3,最后保存退出(wq)

然后我们看看 zoo.cfg 的配置

vi /dao/zookeeper/zookeeper_master/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/dao/zookeeper/data/master
dataLogDir=/dao/zookeeper/logs/master
# the port at which the clients will connect
clientPort=2191
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

vi /dao/zookeeper/zookeeper_slave_1/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/dao/zookeeper/data/slave_1
dataLogDir=/dao/zookeeper/logs/slave_1
# the port at which the clients will connect
clientPort=2192
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

vi /dao/zookeeper/zookeeper_slave_2/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/dao/zookeeper/data/slave_2
dataLogDir=/dao/zookeeper/logs/slave_2
# the port at which the clients will connect
clientPort=2193
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

配置(主要就是 clientPort 配置不同 )已经搞定,是时候来一波启动了,当然你可以写脚本,启动方式都差不多,到 zookeeper 文件夹下,然后启动启动就行了,具体命令如下:

cd /dao/zookeeper/zookeeper_master/bin/
./zkServer.sh start
cd /dao/zookeeper/zookeeper_slave_1/bin/
./zkServer.sh start
cd /dao/zookeeper/zookeeper_slave_2/bin/
./zkServer.sh start

其他就不说了哈,反正我不会(嘿嘿),直接看看 kafka

kafka

一样先创建一个 logs,然后创建三个文件夹 kafka_1 、kafka_2 、kafka_3,具体命令如下:

cd /dao/kafka
mkdir logs
cd logs
mkdir kafka_1 kafka_2 kafka_3

然后下载 kafka ,解压并拷贝三份

wget http://mirrors.hust.edu.cn/apache/kafka/2.0.1/kafka_2.11-2.0.1.tgz
tar -zxvf kafka_2.11-2.0.1.tgz
cp -r kafka_2.11-2.0.1 kafka_1
cp -r kafka_2.11-2.0.1 kafka_2
mv kafka_2.11-2.0.1 kafka_3

又到了看看主要配置的时候了,来来跟踪阿导的脚本,如下操作

vi /dao/kafka/kafka_1/config/server.properties

broker.id=0
port=9092
listeners=PLAINTEXT://{内网 IP}:9092
advertised.listeners=PLAINTEXT://{外网 IP 或者 域名}:9092
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/dao/kafka/logs/kafka_1/
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2191,localhost:2192,localhost:2193
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0


vi /dao/kafka/kafka_1/config/consumer.properties

bootstrap.servers={外网 IP}:9092 
group.id=logGroup
zookeeper.connect=localhost:2191,localhost:2192,localhost:2193

vi /dao/kafka/kafka_2/config/server.properties

broker.id=1
port=9093
listeners=PLAINTEXT://{内网 IP}:9093
advertised.listeners=PLAINTEXT://{外网 IP 或者 域名}:9093
message.max.byte=5242880 
default.replication.factor=2 
replica.fetch.max.bytes=5242880 
num.network.threads=3 
num.io.threads=8 
socket.send.buffer.bytes=102400 
socket.receive.buffer.bytes=102400 
socket.request.max.bytes=104857600
log.dirs=/dao/kafka/logs/kafka_2/ 
num.partitions=1 
num.recovery.threads.per.data.dir=1 
offsets.topic.replication.factor=1 
transaction.state.log.replication.factor=1
 transaction.state.log.min.isr=1 
 log.retention.hours=168 
 log.segment.bytes=1073741824
  log.retention.check.interval.ms=300000 
  zookeeper.connect=localhost:2191,localhost:2192,localhost:2193 
  zookeeper.connection.timeout.ms=6000 
  group.initial.rebalance.delay.ms=0 

vi /dao/kafka/kafka_2/config/consumer.properties

bootstrap.servers={外网 IP}:9093
group.id=logGroup
zookeeper.connect=localhost:2191,localhost:2192,localhost:2193

vi /dao/kafka/kafka_3/config/server.properties

broker.id=2
port=9094
listeners=PLAINTEXT://{内网 IP}:9094
advertised.listeners=PLAINTEXT://{外网 IP 或者 域名}:9094
message.max.byte=5242880 
default.replication.factor=2 
replica.fetch.max.bytes=5242880 
num.network.threads=3 
num.io.threads=8 
socket.send.buffer.bytes=102400 
socket.receive.buffer.bytes=102400 
socket.request.max.bytes=104857600
log.dirs=/dao/kafka/logs/kafka_2/ 
num.partitions=1 
num.recovery.threads.per.data.dir=1 
offsets.topic.replication.factor=1 
transaction.state.log.replication.factor=1
 transaction.state.log.min.isr=1 
 log.retention.hours=168 
 log.segment.bytes=1073741824
  log.retention.check.interval.ms=300000 
  zookeeper.connect=localhost:2191,localhost:2192,localhost:2193 
  zookeeper.connection.timeout.ms=6000 
  group.initial.rebalance.delay.ms=0 

vi /dao/kafka/kafka_3/config/consumer.properties

bootstrap.servers={外网 IP}:9094
group.id=logGroup
zookeeper.connect=localhost:2191,localhost:2192,localhost:2193

总结一下,主要以下配置不同

  • server.properties

broker.id
port
listeners
advertised.listeners
log.dirs

  • consumer.properties

bootstrap.servers={外网 IP}:9094

启动试试

sh /dao/kafka/kafka_1/bin/kafka-server-start.sh /dao/kafka/kafka_1/config/server.properties &
sh /dao/kafka/kafka_2/bin/kafka-server-start.sh /dao/kafka/kafka_2/config/server.properties &
sh /dao/kafka/kafka_3/bin/kafka-server-start.sh /dao/kafka/kafka_3/config/server.properties &

启动成功展示如下,一般启动出错都是你的 IP 没写对,我也是摸滚了好久。。。
Linux 安装 kafka(单机集群)_第4张图片

需要云服务器的不要错过优惠

阿里云低价购买云服务,值得一看

你可能感兴趣的:(kafka,运维,kafka,单机伪集群,hbase)