Greenplum on Kubernetes

Greenplum 是一款基于postsqlgre实现的分布式关系型数据库，我们今天尝试下如何把它部署到kubernetes上。

docker-compose部署方式

在部署到kkubernetes之前，我们先来演示一个使用docker部署的例子，本例是基于这篇文章，感谢提供的基础镜像。
1. 下载greenplum
https://github.com/greenplum-db/gpdb/releases

2. 编译greenplum镜像
定义Dockerfile文件

#  lyasper/gphost 基础镜像提供了sshd服务，及一些初始化脚本
FROM lyasper/gphost
# open-source-greenplum-xx...rpm为上面下载的greenplum安装包
COPY open-source-greenplum-db-6.14.1-rhel7-x86_64.rpm /home/gpadmin/greenplum-db.rpm
RUN yum install -y /home/gpadmin/greenplum-db.rpm
RUN chown -R gpadmin /usr/local/greenplum-db*
RUN rm -f /home/gpadmin/greenplum-db.rpm

docker build -t greenplum:6 .

3. 编写一个docker-compose.yaml文件

version: '3'
services:
  mdw:
    hostname: mdw
    image: "greenplum:6"
    ports:
     - "6222:22"
     - "6432:5432"
  sdw1:
    hostname: sdw1
    image: "greenplum:6"
  sdw2:
    hostname: sdw2
    image: "greenplum:6"
  etl:
    hostname: etl
    image: "greenplum:6"

docker-compose up

4. 初始化greenplum

# 登陆master节点
ssh -p 6222 [email protected]
# 或者 ssh -p 6222 [email protected]
# 密码: changeme

# 初始化配置文件
source /usr/local/greenplum-db/greenplum_path.sh

# 配置greenplum文件
artifact/prepare.sh -s 2 -n 2
# -s: segment(容器)的个数
# -n: 每个segment(容器)上primary的个数

# 初始化集群，会生成env.sh 文件(greenplum所需的环境变量)
gpinitsystem -a -c gpinitsystem_config
source env.sh

# 开启远程无密码访问
artifact/postinstall.sh

# 查看安装结果
ps -ef | grep postgres

# 查看集群状态
gpstate -s

image.png

k8s 部署方式

制作k8s专用镜像（可选）
在部署到k8s之前，我们需要对现有的镜像做一些处理，生成一个k8s专用的greenplum镜像。稍后我会把制作好的镜像放到阿里云上，方便大家拉取。

先启动一个容器并进入

注意：docker run 后面一定不要加/bin/bash等参数，否则后面commit容器时会覆盖以前镜像的CMD参数，所以这里采用先启用再进入到容器的方式

$  docker run -it --rm greenplum:6
$  docker exec -it a2b9a823d845 /bin/bash

修改 artifact/prepare.sh 脚本，需修改3个地方

修改第7行master地址末尾加.gp：MASTERHOST=hostname.gp
修改第8行数据节点的前缀：SEG_PREFIX=gp-
修改87、90行，分别在地址后面加个 .gp

#!/bin/bash

set -x
if [ -z ${GPHOME+x} ]; then echo "GPHOME is unset";exit 1 ; fi

MASTERHOST=`hostname`.gp
SEG_PREFIX=gp-
SEG_HOSTNUM=0 # 0 means muster only
SEG_NUMPERHOST=1
VERBOSE=0
.....
if [ $SEG_HOSTNUM -eq 0 ];then
     echo $MASTERHOST.gp >  $HOSTFILE
else
   for i in $(seq 1 $SEG_HOSTNUM); do
      echo $SEG_PREFIX$i.gp >> $HOSTFILE
   done
 fi

修改/usr/local/greenplum-db-6.14.1/bin/gpinitsystem 脚本第1320行，在解析主机地址时，末尾添加.gp，这样就可以用gp-1.gp去连接segment服务器，而不是gp-1。

   1314 HOST_LOOKUP() {
   1315         res=`echo $1 | $GPHOSTCACHELOOKUP`
   1316         err_index=`echo $res | awk '{print index($0,"__lookup_of_hostname_failed__")}'`
   1317         if [ $err_index -ne 0 ]; then
   1318                 echo "__lookup_of_hostname_failed__"
   1319         else
   1320                 echo $res.gp  # 在主机
   1321         fi
   1322 }

创建一个初始化脚本
新建一个脚本 /home/gpadmin/init.sh，这个脚本仅在master上运行。

#!/bin/bash

# whoami: gpadmin

# only run on master
host=`hostname`
if [ $host = "gp-0" ];then
   source /usr/local/greenplum-db/greenplum_path.sh
   artifact/prepare.sh -s 2 -n 2
   gpinitsystem -a -c gpinitsystem_config
   source env.sh
   artifact/postinstall.sh
   gpstate -s
fi

生成一个新的镜像
我已经把镜像放到了阿里云上，所以后面直接拉取即可

$ docker commit -a "wangjc" -m "Greenplum on Kubernetes" 60875d7be058  greenplum-k8s:6.14.18
$ docker tag greenplum-k8s:6.14.18 registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18
$ docker push registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18

部署到k8s

我的 k8s版本为： 1.18.0，低版本是不支持PVC的，注意下

创建Local Volume
这里我们使用本地的存储卷来保存greenplum的数据，在部署之前，先保证 /var/greenplum目录存在且有足够的权限。

# gp-lv.yaml 
### greenplum storage ###
apiVersion: v1
kind: PersistentVolume
metadata:
  name: greenplum-pv
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: greenplum-storage
  local:
    path: /var/greenplum
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master
          - node1
          - node2

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: greenplum-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: greenplum-storage

$ kubectl create -f gp-lv.yaml 
$ kubectl get pv
$ kubectl get pvc

部署greenplum文件

这里采用StatefulSet方式部署，部署三个服务： gp-0作为master，gp-1、gp-2作为segment
master服务开放两个端口：8432为gp数据库的端口、8222为ssh服务端口

# gp.yaml
###  greenplum db ###
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gp

spec:
  selector:
    matchLabels:
      app: gp
  serviceName: gp
  replicas: 3
  template:
    metadata:
      labels:
        app: gp
    spec:
      securityContext:
        # 允许对非root用户访问PersistentVolumeClaim，否则无法写Local PVC
        #runAsUser: 1000
        fsGroup: 1000
      volumes:
      - name: gp-pv-storage
        persistentVolumeClaim:
          claimName: greenplum-claim
      containers:
      - name: gp-container
        image: registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: "/home/gpadmin/master"
          name: gp-pv-storage
        - mountPath: "/home/gpadmin/data"
          name: gp-pv-storage
        ports:
        - name: gp-port
          containerPort: 5432
        - name: ssh-port
          containerPort: 22
---
#service
apiVersion: v1
kind: Service
metadata:
  name: gp
spec:
  selector:
    app: gp
  type: ClusterIP
  clusterIP: None
---
#service
apiVersion: v1
kind: Service
metadata:
  name: gp-out
spec:
  ports:
  - name: gp-port
    port: 8432
    nodePort: 8432
    protocol: TCP
    targetPort: gp-port
  - name: ssh-port
    port: 8222
    nodePort: 8222
    protocol: TCP
    targetPort: ssh-port
  selector:
    app: gp
    statefulset.kubernetes.io/pod-name: gp-0
  type: NodePort

$ kubectl create -f gp.yaml
$ kubectl get pods

初始化gp数据库
通过ssh连接上master服务器（密码是：changeme），进入到服务器内部进行初始化，192.168.x.x为k8s任意一个节点。初始化成功后，会在每个segment节点上生成两个段。

$ ssh -p 8222 [email protected]
$ ./init.sh
20210310:02:50:04:002506 gpstate:gp-0:gpadmin-[INFO]:-Obtaining Segment details from master...
20210310:02:50:04:002506 gpstate:gp-0:gpadmin-[INFO]:-Gathering data from segments...
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:--Master Configuration & Status
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Master host                    = gp-0
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Master postgres process ID     = 2345
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Master data directory          = /home/gpadmin/master/gpseg-1
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Master port                    = 5432
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Master current role            = dispatch
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Greenplum initsystem version   = 6.14.1 build commit:5ef30dd4c9878abadc0124e0761e4b988455a4bd Open Source
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Greenplum current version      = PostgreSQL 9.4.24 (Greenplum Database 6.14.1 build commit:5ef30dd4c9878abadc0124e0761e4b988455a4bd Open Source) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 22 2021 22:11:57
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Postgres version               = 9.4.24
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Master standby                 = No master standby configured
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-Segment Instance Status Report
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Segment Info
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Hostname                          = gp-1.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Address                           = gp-1.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/data/gpseg0
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Port                              = 10000
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Status
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      PID                               = 842
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Configuration reports status as   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Database status                   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Segment Info
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Hostname                          = gp-1.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Address                           = gp-1.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/data/gpseg1
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Port                              = 10001
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Status
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      PID                               = 843
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Configuration reports status as   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Database status                   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Segment Info
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Hostname                          = gp-2.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Address                           = gp-2.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/data/gpseg2
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Port                              = 10000
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Status
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      PID                               = 842
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Configuration reports status as   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Database status                   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-----------------------------------------------------
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Segment Info
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Hostname                          = gp-2.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Address                           = gp-2.gp
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Datadir                           = /home/gpadmin/data/gpseg3
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Port                              = 10001
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-   Status
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      PID                               = 843
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Configuration reports status as   = Up
20210310:02:50:05:002506 gpstate:gp-0:gpadmin-[INFO]:-      Database status                   = Up

恢复数据
部署到k8s中最大的问题就是当pod重建时，容器被“打回原形”，但好在我们将gp的数据保存在宿主机上，但这时不能用上面 init.sh 脚本初始化了，这样会把之前的数据覆盖掉。因此需要对镜像再做修改。

按照前面的方法，启动一个容器，进去添加一个新的脚本 repair.sh ，专门用来修复数据的。

# 用上一个版本的镜像启动
$ docker run -it registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18

在镜像中添加：repair.sh

#!/bin/bash

# whoami: gpadmin

# only run on master
host=`hostname`
if [ $host = "gp-0" ];then
   source /usr/local/greenplum-db/greenplum_path.sh

   # 注意下prepare.sh脚本，这里面有个删除master、data等目录的操作，需要注释掉
   artifact/prepare.sh -s 2 -n 2
   #gpinitsystem -a -c gpinitsystem_config
   source env.sh
   #artifact/postinstall.sh

   #  解决pod重建后，挂载目录权限问题导致启动失败
   sudo chmod 0700 -R $HOME/master/*
   for HOST in `cat $HOME/hostfile`; do
      ssh gpadmin@${HOST} "sudo chmod 0700 -R $HOME/data/*"
      ssh gpadmin@${HOST} "sudo chmod 0700 -R $HOME/master/*"
      ssh gpadmin@${HOST} "sudo chmod 0700 -R $HOME/mirror/*"
   done

   gpstart
fi

提交容器生成一个新的镜像

$ docker commit -a "wangjc" -m "Greenplum on Kubernetes" 135b86314ce1  registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18
$ docker push registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18

使用新镜像部署gp，并新建数据库添加数据，这时将gp-0删除，这里要注意的是：删除pod会导致svc失效，需要重新创建下。

$ kubectl delete pod gp-0
$ kubectl delete -f svc.yaml &&  kubectl create -f svc.yaml      

#连接到gp库的master，进去执行修复操作
$ ssh -p 8222 [email protected]
#执行恢复脚本， 执行到最后需要手工输入一个：y
$ ./repair.sh

支持镜像节点

默认镜像节点功能是关闭的，还是按之前的方式进入容器中修改后重新提交一个新的容器。

修改 vi artifact/prepare.sh 文件添加如下代码

MIRDATASTR=""
for i in $(seq 1 $SEG_NUMPERHOST);  do
    MIRDATASTR="$MIRDATASTR  $PREFIX/mirror"
done

# 增加一个MIRDATASTR替换
sed "s/%%PORT_BASE%%/$PORT_BASE/g; s|%%MIRDATASTR%%|$MIRDATASTR|g; s|%%PREFIX%%|$PREFIX|g;

修改 artifact/gpinitsystem_config_template 文件将所有以MIRROR开头的属性注释打开，并修改镜像目录如下

declare -a MIRROR_DATA_DIRECTORY=(%%MIRDATASTR%%)

修改init.sh 文件，修改3个段服务器，每个服务器分4个段

artifact/prepare.sh -s 3 -n 4

提交容器生成一个新的镜像

$ docker commit -a "wangjc" -m "Greenplum on Kubernetes" 135b86314ce1  registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18
$ docker push registry.cn-hangzhou.aliyuncs.com/9c/greenplum-k8s:6.14.18

Greenplum on Kubernetes

docker-compose部署方式

k8s 部署方式

支持镜像节点

你可能感兴趣的:(Greenplum on Kubernetes)