Kubernetes部署大数据组件系列一：一键部署Zookeeper集群

目前的大数据平台存在的问题：

• 通过手工方式直接部署在物理机上，过程复杂、效率低，不适合大规模集群的部署和动态扩容。
• 由于平台组件繁多，对宿主机底层库依赖过高，一旦安装完成，很难进行迁移，也无法做成模版进行重复使用。
• 组件升级不够灵活，往往会对有相同底层依赖的其他组件带来影响。
• 采用虚机部署时，会对宿主机产生资源浪费、开销大。
• 物理机或虚机关启等管理操作速度较慢。
• 无法在同一套物理集群上部署多套大数据平台，使用虚机的方案太耗费资源。

针对上述问题，我们打算用时下流行的Docker容器来封装大数据各个组件，用Google的神器Kubernetes来编排这些Docker容器，最终达到：

• 平台能够自动化的快速部署（小规模集群能够达到秒级）、动态扩容。
• 平台构建一次，就能够形成模版重复使用、灵活迁移。
• 平台构建一次，就能够灵活部署在不同的环境中（比如公有云、私有云、裸机，虚拟机）。
• 组件能够灵活升级、灵活替代。
• 消除线上线下环境不一致导致的问题，再也不会有“在我的机器上运行的好好的”这种托词。
• 因为其多资源的低消耗，在物理集群上能同时部署多套大数据平台，供不同部门独 享。
• 很适合持续集成/持续部署(CI/CD)的环境，因为可以自动的下载、绿色的安装和彻底清除应用程序，不会影响宿主机的环境。

本篇是该系列的第一篇，手把手教你通过K8s部署zookeeper集群，涉及到的概念，各位看官自行脑补，我只讲一下部署过程，直接上干货了。

- 使用Kargo搭建Kubernetes 1.6集群:

地址	角色
10.0.7.14	Ansible Client （Kargo用到的工具）
10.0.8.182	Master, NODE 1
10.0.8.183	NODE 2

[下文提到的kargo.tg、kargo_images_v1.6.1.tar.gz压缩包]http://pan.baidu.com/s/1kUKyfBL

1，我用笔记本（macOS系统），作为Ansible client。先新建一个文件夹kargo，把kargo.tgz放进去并解压，目录结构如下：

2，需要安装python 2.7、pip 9.0.1、python-netaddr、ansible 2.3.1.0等工具 [ansible安装]http://valdhaus.co/writings/ansible-mac-osx/

3，修改 inventory/inventory.cfg

wangliangdeMacBook-Pro:kargo wangliang$ pwd
/Users/wangliang/kargo/kargo
wangliangdeMacBook-Pro:kargo wangliang$ vim inventory/inventory.cfg

[all]
node1    ansible_user=app ansible_host=10.0.8.182  ip=10.0.8.182 access_ip=10.0.8.182
node2    ansible_user=app ansible_host=10.0.8.183  ip=10.0.8.183 access_ip=10.0.8.183

[kube-master]
node1    ip=10.0.8.182 access_ip=10.0.8.182

[kube-node]
node1
node2

[etcd]
node1

[k8s-cluster:children]
kube-node
kube-master

[calico-rr]

4，分别在10.0.8.182(Node 1)、10.0.8.183(Node 2)上做如下操作：
首先安装Docker 1.13.1，并启动服务
[Ubuntu安装Docker]http://www.linuxidc.com/Linux/2017-01/139985.htm

其次把kargo_images_v1.6.1.tar.gz 拷贝到要安装k8s的节点上，放到k8s文件夹内，并解压，目录结构如下：

5，在kargo_images_v1.6.1目录下，进入bash，执行：

➜  k8s cd kargo_images_v1.6.1
➜  kargo_images_v1.6.1 pwd
/data/k8s/kargo_images_v1.6.1
➜  kargo_images_v1.6.1 bash
app@yuyan1:/data/k8s/kargo_images_v1.6.1$images=$(ls -l ../kargo_images_v1.6.1|awk -F' ' '{ print $9 }')

app@yuyan1:/data/k8s/kargo_images_v1.6.1$ for x in ${images[*]}; do sudo docker load -i $x; done

将目录下的镜像放入docker images。

6， Node1、Node2和Mac间都要SSH免密，Mac上还要打开共享里面的允许远端访问

Node1、Node2、Mac都在用户目录下执行

➜  ~ cd /home/app
ssh-keygen -t rsa  #一路回车就行
Generating public/private rsa key pair.
Enter file in which to save the key (/home/app/.ssh/id_rsa):
Created directory '/home/app/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/app/.ssh/id_rsa.
Your public key has been saved in /home/app/.ssh/id_rsa.pub.
The key fingerprint is:
7c:f4:2f:21:f2:85:7f:20:38:02:1d:ef:79:39:b6:be root@dcos1
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . o         |
|    . . . .      |
|     . o + +     |
|      . S X =    |
|       . B B +   |
|          o o o  |
|         .   o   |
|          E.     |
+-----------------+

➜  ~ cd /home/app/.ssh
➜  .ssh ssh-copy-id Node2
➜  .ssh ssh-copy-id Mac

每次执行，步骤都如下类似：
The authenticity of host 'Node1 (10.0.8.183)' can't be established.
ECDSA key fingerprint is 91:1a:13:a8:57:2b:a0:42:4d:aa:c9:83:c3:33:16:f9.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@dcos1's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'Node2'"
and check to make sure that only the key(s) you wanted were added.

所有虚机都执行完成后，可以通过ssh hostname来查看之间是否ssh连接已经免密了，如下测试

➜  ~ ssh 10.0.8.183
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-62-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

180 packages can be updated.
75 updates are security updates.


Last login: Wed Jun 28 09:16:54 2017 from 10.0.7.14
➜  ~

7，在Ansible client(Mac)上执行

$ ansible-playbook -i ~/kargo/inventory/inventory.cfg cluster.yml -b -v --user=username --ask-sudo-pass

安装成功，在Mac上可以看到如下输出：

在Node 1上可以看到：

遇到的错误：
1，有如下错误：

kubectl The connection to the server localhost:8080 was refused

修改了vim inventory/group_vars/k8s-cluster.ym 里面的8080为28080，避免和别人冲突，又有如下错误：

kubernetes/master : Master | wait for the apiserver to be running

原因是apiserver的镜像起不了，无法提供服务
怀疑可能是不支持docker 1.17版本，所以更换为docker 1.13.1版本子再次安装后没有上述错误了，安装成功。切记，要用docker 1.13.1版本。

2，如果按照上述操作始终没有成功，可以把节点上的

/run/kubernetes
/etc/kubernetes

都删除，然后把docker也都卸载重新装。然后在Ansible client节点上重新部署就可以啦。

• 使用Dockerfile制作包含zkui 2.0 的Zookeeper 3.4.10版本的镜像：
  1，建立你要做docker的目录，结构如下：
  
  
  

2，其中Dockerfile如下：

FROM openjdk:8-jre-alpine
MAINTAINER Wang Liang 

# Install required packages
RUN apk add --no-cache \
    bash \
    su-exec

ENV ZOO_USER=zookeeper \
    ZOO_CONF_DIR=/conf \
    ZOO_DATA_DIR=/data \
    ZOO_UI_DIR=/zkui \
    ZOO_DATA_LOG_DIR=/datalog \
    ZOO_PORT=2181 \
    ZOO_TICK_TIME=2000 \
    ZOO_INIT_LIMIT=5 \
    ZOO_SYNC_LIMIT=2

# Add a user and make dirs
RUN set -x \
    && adduser -D "$ZOO_USER" \
    && mkdir -p "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR" \
    && chown "$ZOO_USER:$ZOO_USER" "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR"

ARG GPG_KEY=C823E3E5B12AF29C67F81976F5CECB3CB5E9BD2D
ARG DISTRO_NAME=zookeeper-3.4.10

# Download Apache Zookeeper, verify its PGP signature, untar and clean up
RUN set -x \
    && apk add --no-cache --virtual .build-deps \
        gnupg \
    && wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz" \
    && wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" \
    && export GNUPGHOME="$(mktemp -d)" \
    && gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \
    && gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \
    && tar -xzf "$DISTRO_NAME.tar.gz" \
    && mv "$DISTRO_NAME/conf/"* "$ZOO_CONF_DIR" \
    && rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \
    && apk del .build-deps

ADD zkui-master $ZOO_UI_DIR

WORKDIR $DISTRO_NAME
VOLUME ["$ZOO_DATA_DIR", "$ZOO_DATA_LOG_DIR"]

EXPOSE $ZOO_PORT 2888 3888

ENV PATH=$PATH:/$DISTRO_NAME/bin:$ZOO_UI_DIR \
    ZOOCFGDIR=$ZOO_CONF_DIR

COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]

3，docker-entrypoint.sh如下：

#!/bin/bash

set -e

# Allow the container to be started with `--user`
if [ "$1" = 'zkServer.sh' -a "$(id -u)" = '0' ]; then
    chown -R "$ZOO_USER" "$ZOO_DATA_DIR" "$ZOO_DATA_LOG_DIR"
    exec su-exec "$ZOO_USER" "$0" "$@"
fi

# Generate the config only if it doesn't exist
if [ ! -f "$ZOO_CONF_DIR/zoo.cfg" ]; then
    CONFIG="$ZOO_CONF_DIR/zoo.cfg"

    echo "clientPort=$ZOO_PORT" >> "$CONFIG"
    echo "dataDir=$ZOO_DATA_DIR" >> "$CONFIG"
    echo "dataLogDir=$ZOO_DATA_LOG_DIR" >> "$CONFIG"

    echo "tickTime=$ZOO_TICK_TIME" >> "$CONFIG"
    echo "initLimit=$ZOO_INIT_LIMIT" >> "$CONFIG"
    echo "syncLimit=$ZOO_SYNC_LIMIT" >> "$CONFIG"

    for server in $ZOO_SERVERS; do
        echo "$server" >> "$CONFIG"
    done
fi

if [ -f "$ZOO_UI_DIR/config.cfg" ]; then
    CONFIG="$ZOO_UI_DIR/config.cfg"

    echo "serverPort=$ZOO_UI_PORT" >> "$CONFIG"
    echo "zkServer=$ZOO_UI_SERVER" >> "$CONFIG"
fi

# Write myid only if it doesn't exist
if [ ! -f "$ZOO_DATA_DIR/myid" ]; then
    echo "${ZOO_MY_ID:-1}" > "$ZOO_DATA_DIR/myid"
fi

cd $ZOO_UI_DIR
exec nohup java -jar zkui-2.0-SNAPSHOT-jar-with-dependencies.jar &
exec zkServer.sh start-foreground

[制作过程需要的文件]http://pan.baidu.com/s/1i5IG6sH

执行

➜  wldocker sudo docker build -t zookeeper_3.4.10_zkui_2.0:0.0.1 .

就可以制作出包含ZKUI的zookeeper 3.4.10的docker镜像了

把该镜像上传到HARBOR库，稍后k8s编排时会用到。

• 通过Kubernetes 编排启动Zookeeper集群：
  我们采取的方案是每个zkserver属于一个pod，每个pod绑定一个物理主机。
  只需要完成yaml文件即可:

zk-1.yaml:

---
apiVersion: v1
kind: Service
metadata:
  name: zk-1-svc
  labels:
    app: zk-1-svc
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  - port: 2181
    name: client
  - port: 9999
    name: zkui
  selector:
    app: zk-1
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zk-1-cm
data:
  jvm.heap: "1G"
  tick: "2000"
  init: "10"
  sync: "5"
  client.cnxns: "60"
  snap.retain: "3"
  purge.interval: "0"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: zk-1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zk-1
    spec:
      nodeSelector:
        zk: zk-1
      containers:
      - name: myzk
        imagePullPolicy: IfNotPresent
        image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1
        resources:
          requests:
            memory: "2Gi"
            cpu: "500m"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        - containerPort: 9999
          name: zkui
        env:
        - name : ZK_HEAP_SIZE
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: jvm.heap
        - name : ZK_TICK_TIME
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: tick
        - name : ZK_INIT_LIMIT
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: init
        - name : ZK_SYNC_LIMIT
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: tick
        - name : ZK_MAX_CLIENT_CNXNS
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: client.cnxns
        - name: ZK_SNAP_RETAIN_COUNT
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: snap.retain
        - name: ZK_PURGE_INTERVAL
          valueFrom:
            configMapKeyRef:
                name: zk-1-cm
                key: purge.interval
        - name: ZK_CLIENT_PORT
          value: "2181"
        - name: ZK_SERVER_PORT
          value: "2888"
        - name: ZK_ELECTION_PORT
          value: "3888"
        - name: ZOO_MY_ID
          value: "1"
        - name: ZOO_SERVERS
          value: "server.1=0.0.0.0:2888:3888 server.2=zk-2-svc:2888:3888"
        - name: ZOO_UI_PORT
          value: "9999"
        - name: ZOO_UI_SERVER
          value: "zk-1-svc:2181,zk-2-svc:2181"
#        readinessProbe:
#          exec:
#            command:
#            - "zkok.sh"
#          initialDelaySeconds: 10
#          timeoutSeconds: 5
#        livenessProbe:
#          exec:
#            command:
#            - "zkok.sh"
#          initialDelaySeconds: 10
#          timeoutSeconds: 5
        volumeMounts:
        - name: data
          mountPath: /data
        - name: datalog
          mountPath: /datalog
      volumes:
      - name: data
        hostPath:
          path: /data/k8s/zookeeper/data
      - name: datalog
        hostPath:
          path: /data/k8s/zookeeper/datalog

zk-2.yaml:

---
apiVersion: v1
kind: Service
metadata:
  name: zk-2-svc
  labels:
    app: zk-2-svc
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  - port: 2181
    name: client
  - port: 9999
    name: zkui
  selector:
    app: zk-2
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zk-2-cm
data:
  jvm.heap: "1G"
  tick: "2000"
  init: "10"
  sync: "5"
  client.cnxns: "60"
  snap.retain: "3"
  purge.interval: "0"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: zk-2
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zk-2
    spec:
      nodeSelector:
        zk: zk-2
      containers:
      - name: myzk
        imagePullPolicy: IfNotPresent
        image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1
        resources:
          requests:
            memory: "2Gi"
            cpu: "500m"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        - containerPort: 9999
          name: zkui
        env:
        - name : ZK_HEAP_SIZE
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: jvm.heap
        - name : ZK_TICK_TIME
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: tick
        - name : ZK_INIT_LIMIT
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: init
        - name : ZK_SYNC_LIMIT
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: tick
        - name : ZK_MAX_CLIENT_CNXNS
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: client.cnxns
        - name: ZK_SNAP_RETAIN_COUNT
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: snap.retain
        - name: ZK_PURGE_INTERVAL
          valueFrom:
            configMapKeyRef:
                name: zk-2-cm
                key: purge.interval
        - name: ZK_CLIENT_PORT
          value: "2181"
        - name: ZK_SERVER_PORT
          value: "2888"
        - name: ZK_ELECTION_PORT
          value: "3888"
        - name: ZOO_MY_ID
          value: "2"
        - name: ZOO_SERVERS
          value: "server.1=zk-1-svc:2888:3888 server.2=0.0.0.0:2888:3888 "
        - name: ZOO_UI_PORT
          value: "9999"
        - name: ZOO_UI_SERVER
          value: "zk-1-svc:2181,zk-2-svc:2181"
#        readinessProbe:
#          exec:
#            command:
#            - "zkok.sh"
#          initialDelaySeconds: 10
#          timeoutSeconds: 5
#        livenessProbe:
#          exec:
#            command:
#            - "zkok.sh"
#          initialDelaySeconds: 10
#          timeoutSeconds: 5
        volumeMounts:
        - name: data
          mountPath: /data
        - name: datalog
          mountPath: /datalog
      volumes:
      - name: data
        hostPath:
          path: /data/k8s/zookeeper/data
      - name: datalog
        hostPath:
          path: /data/k8s/zookeeper/datalog

[yaml共享]http://pan.baidu.com/s/1jIyvGPK

执行：

➜  zookeeper kubectl create -f zk-1.yaml
service "zk-1-svc" created
configmap "zk-1-cm" created
deployment "zk-1" created
➜  zookeeper kubectl create -f zk-2.yaml
service "zk-2-svc" created
configmap "zk-2-cm" created
deployment "zk-2" created

• 功能验证：

➜  zookeeper kubectl get po  #看po情况
NAME                    READY     STATUS    RESTARTS   AGE
zk-1-1238243890-phcm8   1/1       Running   0          1m
zk-2-4022999611-x27vq   1/1       Running   0          1m
➜  zookeeper kubectl get po -o wide #看绑定物理节点情况
NAME                    READY     STATUS    RESTARTS   AGE       IP               NODE
zk-1-1238243890-phcm8   1/1       Running   0          1m        11.233.112.152   yuyan1
zk-2-4022999611-x27vq   1/1       Running   0          1m        11.233.77.83     yuyan2
➜  zookeeper kubectl get service  #看端口映射情况
NAME         CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                       AGE
kubernetes   11.233.0.1              443/TCP                                                       4d
zk-1-svc     11.233.30.168          2888:31292/TCP,3888:30513/TCP,2181:31570/TCP,9999:30476/TCP   1m
zk-2-svc     11.233.27.68           2888:30906/TCP,3888:30842/TCP,2181:32473/TCP,9999:31490/TCP   1m
➜  zookeeper kubectl exec  zk-2-4022999611-x27vq -t -i -- bash #进入容器看服务情况
bash-4.3# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
Mode: leader  #选举正常

通过宿主机ip加映射后的端口访问zkui http://10.0.8.183:31490

提供对外访问正常

参考
[Kargo部署K8s]http://oo3p.com/2017/04/17/kargo容器化部署kubernetes高可用集群/
[zookeeper docker]https://github.com/31z4/zookeeper-docker/tree/f12428ab7c6ea263ef037cf258129b83276c009c
[k8s pod]https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
[k8s编排zookeeper集群]https://github.com/kubernetes/contrib/tree/master/statefulsets/zookeeper

作者：俺是亮哥
链接：http://www.jianshu.com/p/315e18c2f3a2
來源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。