zookeeper是一个分布式(集群)应用程序协调系统,具有分布式以及开源的特性,也是大数据hadoop生态中的一个基础服务,但不去不止用于hadoop系统。本次实验记录zookeeper集群的部署,扩容,缩容,基本操作以及监控,本文档将实验结果记录下来,作为研究档案,供将来差缺补漏。
测试集群采用3个节点的虚拟机进行构建(虚拟机构建在openstack平台之上),节点信息如下:
主机名称 | 别名序号 | IP地址 | 系统 | 角色 |
---|---|---|---|---|
zookeeper-node-1 | server.1 | CentOS7 | 10.120.67.19 | zookeeper |
zookeeper-node-2 | server.2 | CentOS7 | 10.120.67.20 | zookeeper |
zookeeper-node-3 | server.3 | CentOS7 | 10.120.67.21 | zookeeper |
集群部署采用3个节点,之后会在创建2个节点用于扩容,缩容测试
这里便于测试将selinux与firewalld直接关闭服务,实际产线中一般会关闭selinux,firewalld根据实际需要配置(如果在内网里也可以直接关闭,实际情况很多组织都是这么做的)
~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
~]# setenforce 0
~]# systemctl stop firewalld.service
~]# systemctl disable firewalld.service
zookeeper集群的3台服务器标号从1到3,每台主机的名称命名规范如下
~]# hostnamectl set-hostname zookeeper-node-${编号}
3个节点的hosts文件配置如下(其实没有用到这个,配置文件直接配置的IP地址)
~]# cat /etc/hosts
...
10.120.67.19 zookeeper-node-1
10.120.67.20 zookeeper-node-2
10.120.67.21 zookeeper-node-3
没有去研究zookeeper系统的内核调优,这里将所在组织虚拟机模板中的参数直接使用,先用着,不合适再进行调整
~]# cat /etc/sysctl.conf
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmax = 1048560
kernel.msgmnb = 1073741824
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.core.somaxconn = 256
net.core.optmem_max = 20480
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_wmem = 8192 436600 16777216
net.ipv4.tcp_rmem = 32768 436600 16777216
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_retries2 = 5
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.ip_local_port_range = 1024 65000
vm.swappiness = 0
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
我的测试集群将10.120.67.9服务器作为时钟源,zookeeper集群配置如下(所有节点
)
~]# vi /etc/chrony.conf
server 10.120.67.9 iburst
~]# systemctl restart chronyd.service
~]# timedatectl
...
NTP enabled: yes
NTP synchronized: yes
~]# chronyc sources
210 Number of sources = 1
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 10.120.67.9 2 6 17 19 -25us[ -113us] +/- 104ms
这里采用官方网站下载JDK文件,解压安装的方式安装。文件:jdk-8u172-linux-x64.tar.gz,所有节点统一部署配置
,下载地址:JDK
~]# mkdir /usr/java
~]# tar zxvf jdk-8u172-linux-x64.tar.gz
~]# mv jdk1.8.0_172 /usr/java/
~]# vi /etc/profile #[最后追加java环境变量]
export JAVA_HOME=/usr/java/jdk1.8.0_172
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
~]# source /etc/profile
~]# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
下载zookeeper程序,下载地址:zookeeper
~]# tar zxvf zookeeper-3.4.12.tar.gz
~]# mv zookeeper-3.4.12 /opt/app/install/
~]# ln -s /opt/app/install/zookeeper-3.4.12 /opt/app/zookeeper
~]# ll /opt/app/
lrwxrwxrwx 1 root root 33 May 22 18:38 zookeeper -> /opt/app/install/zookeeper-3.4.12
# 配置(3个节点一样的配置)
~]# cd /opt/app/zookeeper/conf
conf]# mkdir -p /data/zookeeper
conf]# cp zoo_sample.cfg zoo.cfg
conf]# vi zoo.cfg #此处将注释行与空行省略掉了
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=10.120.67.19:2888:3888
server.2=10.120.67.20:2888:3888
server.3=10.120.67.21:2888:3888
# 配置数据目录以及myid文件 [各个节点分别对应执行]
zookeeper-node-1 ~]# echo 1 > /data/zookeeper/myid
zookeeper-node-2 ~]# echo 2 > /data/zookeeper/myid
zookeeper-node-3 ~]# echo 3 > /data/zookeeper/myid
在节点1(zookeeper-node-1)上执行服务启动
~]# /opt/app/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
~]# /opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
# 【查看日志 zookeeper.out文件可以了解到时因为其他2个节点没有起来导致的。依次启动剩余节点即可】
//zookeeper-node-2
~]# /opt/app/zookeeper/bin/zkServer.sh start
~]# /opt/app/zookeeper/bin/zkServer.sh status
...
Mode: leader
//zookeeper-node-3
~]# /opt/app/zookeeper/bin/zkServer.sh start
~]# /opt/app/zookeeper/bin/zkServer.sh status
...
Mode: follower
//zookeeper-node-1 #【回头再看下节点1 已经正常了】
~]# /opt/app/zookeeper/bin/zkServer.sh status
...
Mode: follower
~]# /opt/app/zookeeper/bin/zkCli.sh -server 10.120.67.19:2181
[zk: 10.120.67.19:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: 10.120.67.19:2181(CONNECTED) 1] create /demo 'test'
Created /demo
[zk: 10.120.67.19:2181(CONNECTED) 2] ls /
[zookeeper, demo]
~]# /opt/app/zookeeper/bin/zkCli.sh -server 10.120.67.20:2181
[zk: 10.120.67.20:2181(CONNECTED) 0] ls /
[zookeeper, demo]
如上,就完成了一个3节点的zookeeper集群的配置,可以提供服务了。