zookeeper集群管理(1) zookeeper集群部署

zookeeper是一个分布式(集群)应用程序协调系统，具有分布式以及开源的特性，也是大数据hadoop生态中的一个基础服务，但不去不止用于hadoop系统。本次实验记录zookeeper集群的部署，扩容，缩容，基本操作以及监控，本文档将实验结果记录下来，作为研究档案，供将来差缺补漏。

1. zookeeper集群结构与服务器信息

测试集群采用3个节点的虚拟机进行构建(虚拟机构建在openstack平台之上)，节点信息如下：

主机名称	别名序号	IP地址	系统	角色
zookeeper-node-1	server.1	CentOS7	10.120.67.19	zookeeper
zookeeper-node-2	server.2	CentOS7	10.120.67.20	zookeeper
zookeeper-node-3	server.3	CentOS7	10.120.67.21	zookeeper

集群部署采用3个节点，之后会在创建2个节点用于扩容，缩容测试

2. 服务器准备工作

(1) 关闭selinux 和 firewalld(iptables)

这里便于测试将selinux与firewalld直接关闭服务，实际产线中一般会关闭selinux，firewalld根据实际需要配置（如果在内网里也可以直接关闭，实际情况很多组织都是这么做的）

~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
~]# setenforce 0

~]# systemctl stop firewalld.service
~]# systemctl disable firewalld.service

(2) 配置主机名称与/etc/hosts文件

zookeeper集群的3台服务器标号从1到3，每台主机的名称命名规范如下

~]# hostnamectl set-hostname zookeeper-node-${编号}

3个节点的hosts文件配置如下(其实没有用到这个，配置文件直接配置的IP地址)

~]# cat /etc/hosts
...
10.120.67.19    zookeeper-node-1
10.120.67.20    zookeeper-node-2
10.120.67.21    zookeeper-node-3

(3) Linux系统参数调整

没有去研究zookeeper系统的内核调优，这里将所在组织虚拟机模板中的参数直接使用，先用着，不合适再进行调整

~]# cat /etc/sysctl.conf
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmax = 1048560
kernel.msgmnb = 1073741824
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.core.somaxconn = 256
net.core.optmem_max = 20480
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_wmem = 8192  436600  16777216
net.ipv4.tcp_rmem = 32768 436600  16777216
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_retries2 = 5
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.ip_local_port_range = 1024 65000
vm.swappiness = 0
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

(4) 节点NTP时钟同步

我的测试集群将10.120.67.9服务器作为时钟源，zookeeper集群配置如下（所有节点）

~]# vi /etc/chrony.conf
server 10.120.67.9 iburst

~]# systemctl restart chronyd.service
~]# timedatectl
             ...
     NTP enabled: yes
NTP synchronized: yes
~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 10.120.67.9                   2   6    17    19    -25us[ -113us] +/-  104ms

(5)所有节点安装JDK

这里采用官方网站下载JDK文件，解压安装的方式安装。文件：jdk-8u172-linux-x64.tar.gz，所有节点统一部署配置，下载地址:JDK

~]# mkdir /usr/java
~]# tar zxvf jdk-8u172-linux-x64.tar.gz
~]# mv jdk1.8.0_172 /usr/java/
~]# vi /etc/profile #[最后追加java环境变量]
    export JAVA_HOME=/usr/java/jdk1.8.0_172
    export JRE_HOME=$JAVA_HOME/jre
    export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
    export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
~]# source /etc/profile
~]# java -version
    java version "1.8.0_172"
    Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

3. 安装配置zookeeper集群

下载zookeeper程序，下载地址：zookeeper

~]# tar zxvf zookeeper-3.4.12.tar.gz
~]# mv zookeeper-3.4.12 /opt/app/install/
~]# ln -s /opt/app/install/zookeeper-3.4.12 /opt/app/zookeeper
~]# ll /opt/app/
lrwxrwxrwx 1 root root 33 May 22 18:38 zookeeper -> /opt/app/install/zookeeper-3.4.12

# 配置(3个节点一样的配置)
~]# cd /opt/app/zookeeper/conf
conf]# mkdir -p /data/zookeeper
conf]# cp zoo_sample.cfg zoo.cfg
conf]# vi zoo.cfg   #此处将注释行与空行省略掉了
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
server.1=10.120.67.19:2888:3888
server.2=10.120.67.20:2888:3888
server.3=10.120.67.21:2888:3888

# 配置数据目录以及myid文件 [各个节点分别对应执行]
zookeeper-node-1 ~]# echo 1 > /data/zookeeper/myid
zookeeper-node-2 ~]# echo 2 > /data/zookeeper/myid
zookeeper-node-3 ~]# echo 3 > /data/zookeeper/myid

4. 启动zookeeper集群

在节点1（zookeeper-node-1）上执行服务启动

~]# /opt/app/zookeeper/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

~]# /opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
# 【查看日志 zookeeper.out文件可以了解到时因为其他2个节点没有起来导致的。依次启动剩余节点即可】

//zookeeper-node-2
~]# /opt/app/zookeeper/bin/zkServer.sh start
~]# /opt/app/zookeeper/bin/zkServer.sh  status
    ...
    Mode: leader

//zookeeper-node-3
~]# /opt/app/zookeeper/bin/zkServer.sh start
~]# /opt/app/zookeeper/bin/zkServer.sh status
    ...
    Mode: follower

//zookeeper-node-1 #【回头再看下节点1 已经正常了】
 ~]# /opt/app/zookeeper/bin/zkServer.sh  status
    ...
    Mode: follower

5. 服务使用测试

~]# /opt/app/zookeeper/bin/zkCli.sh -server 10.120.67.19:2181
[zk: 10.120.67.19:2181(CONNECTED) 0] ls /
    [zookeeper]
[zk: 10.120.67.19:2181(CONNECTED) 1] create /demo 'test'
    Created /demo
[zk: 10.120.67.19:2181(CONNECTED) 2] ls /
    [zookeeper, demo]

~]# /opt/app/zookeeper/bin/zkCli.sh -server 10.120.67.20:2181
[zk: 10.120.67.20:2181(CONNECTED) 0] ls /
    [zookeeper, demo]

如上，就完成了一个3节点的zookeeper集群的配置，可以提供服务了。