K8S笔记：基于K8S的Java构建与部署环境

目录

安装Docker

环境：CentOS7 CentOS8

1、安装

官网文档

sudo yum install -y yum-utils
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
#sudo yum install  --allowerasing  docker-ce docker-ce-cli containerd.io
#kubenetes对docker版本有要求，需要先确认需要安装的版本
#CentOS7请修改版本中的el8 --> el7
sudo yum install --allowerasing docker-ce-19.03.15-3.el8 docker-ce-cli-19.03.15-3.el8 containerd.io
sudo systemctl enable --now docker

如果运行docker命令显示无权限错误，可以将当前用户加入docker组

CentOS8

usermod -aG docker 

CentOS7

gpasswd -a  docker

2、修改Cgroup驱动

systemd使用的cgroupfs驱动，如果容器也使用cgroupfs，则节点将有两个不同cgroup管理器，这可能会让节点在资源压力下变得不稳定，所以让容器直接使用systemd。注意不要在节点安装加入集群后修改。(参考)
编辑 /etc/docker/daemon.json (不存在就新建），配置运行参数：

    { "exec-opts": ["native.cgroupdriver=systemd"] }

需要重启Docker，重启后可查看docker info确认修改

systemctl restart docker
docker info

3、修改docker默认网桥docker0网段

如果docker0默认网段与宿主网络冲突会造成容器无法访问宿主机网络的主机，这时需要修改网段。在文件"/etc/docker/daemon.json"中添加配置 “bip”: “ip/netmask”,注意不要与宿主机网段冲突

/etc/docker/daemon.json

{ "bip":"172.17.100.1/24" }

修改后重启docker服务

4、创建运行容器的用户

建议不使用root用户运行容器中的进程

sudo useradd 
sudo passwd 
sudo usermod -aG docker 
su - 

安装测试用docker仓库

docker pull registry:2
sudo mkdir /opt/docker-registry
sudo chown : /opt/docker-registry
docker run -d -u $UID -v /opt/docker-registry:/var/lib/registry \
        -p 5000:5000 --name docker-registry \
        --restart always registry:2

正式环境建议安装harbor

将镜像标志为要推送到私有仓库：

docker tag nginx:latest ip:5000/nginx:latest

push到私有仓库：

docker push ip:5000/nginx:latest

错误处理： http: server gave HTTP response to HTTPS client

• 方法一、编辑daemon.json添加仓库地址

sudo vim /etc/docker/daemon.json

{ "insecure-registries": ["172.17.152.44:5000"] }

• 方法二、修改docker启动参数

sudo vim /usr/lib/systemd/system/docker.service

在 ExecStart后添加参数：–insecure-registries 172.17.152.44:5000

修改后重启Docker服务

注意：ExecStart添加参数后，就不能在daemon.json中重复配置，否则docker服务会启动失败

sudo systemctl daemon-reload
sudo systemctl restart docker

创建部署映像

以下是Image创建栈

docker build

docker build

docker run

clone

volume

gradle build

docker build

copy

Image centos:7

Image tq/centos-temp:7

Image tq/gradle:6

Image tq/jdk:8

Container Builder

Git Server

Java Code

BUILD DEST

Image Java App

1、映像 tq/centos-temp

Dockerfile.temporary

FROM centos:7

RUN set -o errexit \
    && yum install -y -q subversion wget unzip gzip git \
    && yum clean packages

在centos7下运行此build会失败，需要升级systemd模块

创建映像

$ docker build --rm -t 172.17.149.10:5000/tq/centos-temp:7 -f Dockerfile.temp .
$ docker push 172.17.149.10:5000/tq/centos-temp:7

2、 映像 tq/jdk:8

Dockerfile.jdk

FROM 172.17.149.10:5000/tq/centos-temp:7

WORKDIR /root/
RUN set -o errexit \
        && echo "Downloading JDK" \
        && svn -q --username hello --password 123456 \
             --no-auth-cache export svn://192.168.0.100:8000/download/jdk/jdk-8u271-linux-x64.tar.gz \
        && tar -xzf jdk-8u271-linux-x64.tar.gz

FROM centos:7

COPY --from=0 /root/jdk1.8.0_271 /opt/jdk1.8.0_271
ENV JAVA_HOME=/opt/jdk1.8.0_271
ENV PATH="$PATH:$JAVA_HOME/bin"
RUN set -o errexit -o nounset \
    && echo "JAVA_HOME=$JAVA_HOME" \
    && echo "PATH=$PATH" \
    && echo "Testing JDK installation" \
    && java -version

创建映像

$ docker build --rm -t 172.17.149.10:5000/tq/jdk:1.8.0_271 -t 172.17.149.10:5000/tq/jdk:8 -f Dockerfile.jdk .
$ docker push 172.17.149.10:5000/tq/jdk:1.8.0_271
$ docker push 172.17.149.10:5000/tq/jdk:8

3、映像 tq/gradle:6

部署映像创建容器：Dockerfile.gradle

FROM 172.17.149.10:5000/tq/centos-temp:7

ENV GRADLE_VERSION 6.8.2
ARG GRADLE_DOWNLOAD_SHA256=8de6efc274ab52332a9c820366dd5cf5fc9d35ec7078fd70c8ec6913431ee610

WORKDIR /root/
RUN set -o errexit -o nounset \
    && echo "Downloading Gradle" \
    && wget --no-verbose --output-document=gradle.zip "https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" \
    \
    && echo "Checking download hash" \
    && echo "${GRADLE_DOWNLOAD_SHA256} *gradle.zip" | sha256sum --check - \
    \
    && echo "Installing Gradle" \
    && unzip -q gradle.zip \
    && rm gradle.zip
    
FROM 172.17.149.10:5000/tq/jdk:8

RUN set -o errexit -o nounset \
    && echo "Adding gradle user and group" \
    && groupadd --system --gid 1000 gradle \
    && useradd --system --gid gradle --uid 1000 --shell /bin/bash --create-home gradle \
    && mkdir /home/gradle/.gradle \
    && chown --recursive gradle:gradle /home/gradle \
    \
    && echo "Symlinking root Gradle cache to gradle Gradle cache" \
    && ln -s /home/gradle/.gradle /root/.gradle

VOLUME /home/gradle/.gradle

WORKDIR /home/gradle

ENV GRADLE_HOME /opt/gradle
ENV GRADLE_VERSION 6.8.2

COPY --from=0 /root/gradle-${GRADLE_VERSION} ${GRADLE_HOME}
RUN set -o errexit -o nounset \
    && ln --symbolic "${GRADLE_HOME}/bin/gradle" /usr/bin/gradle \
    \
    && echo "Testing Gradle installation" \
    && gradle --version

创建映像

$ docker build --rm -t 172.17.149.10:5000/tq/gradle:6.8.2 -t 172.17.149.10:5000/tq/gradle:6 -f Dockerfile.gradle .
$ docker push  172.17.149.10:5000/tq/gradle:6.8.2
$ docker push  172.17.149.10:5000/tq/gradle:6

4、用映像 ‘tq/gradle:6’ 编译项目

$ cd ~
$ git clone -b release/ver20200807 [email protected]:mygroup/test-server.git project
$ cd ~/project
$ docker run --rm -u gradle -v "$PWD":/home/gradle/project \
        -w /home/gradle/project/test-server myrepo/gradle:6 gradle publishDEV

5、Java App 映像

应用启动脚本 startup.sh

#!/bin/bash

APP_OUT="$APP_HOME"/logs/startup.log
touch $APP_OUT
echo "==== Start app: $(date -Iseconds) ====" >> "$APP_OUT"
jar_file(){
   echo `ls -t $APP_HOME/*.jar 2>/dev/null | head -1`
}
JAR_FILE=$(jar_file)

if [ -z $JAR_FILE ]; then
   echo 'No jar file in APP_HOME: $APP_HOME' > "$APP_OUT"
else
   JAVA_OPTS="-server {{exec_opt}}"
   JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC -XX:+PrintClassHistogram -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCDetails"
   JAVA_OPTS="$JAVA_OPTS -Xloggc:$APP_HOME/logs/gc.log"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port={{jmx_port}} -Dcom.sun.management.jmxremote.rmi.port={{jmx_port}}"
   JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.password.file=$APP_HOME/jmx.password -Dcom.sun.management.jmxremote.access.file=$APP_HOME/jmx.access"

   echo "Using APP_HOME:   $APP_HOME"  >> "$APP_OUT"
   echo "Using JAVA_HOME:  $JAVA_HOME" >> "$APP_OUT"
   echo "Using JAR_FILE:   $JAR_FILE"  >> "$APP_OUT"
   echo "Using SERVER_PORT: $SERVER_PORT" >> "$APP_OUT"
   echo "Using JAVA_OPTS:  $JAVA_OPTS" >> "$APP_OUT"
   
   java $JAVA_OPTS -jar $JAR_FILE
fi

Dockerfile.app.java

FROM 172.17.149.10:5000/tq/jdk:8

ARG BUILD_DEST
ENV LANG en_US.UTF-8    
ENV USER_NAME publish
ENV APP_HOME /home/${USER_NAME}/app

RUN set -o errexit -o nounset \
    && ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
    && echo "Adding $USER_NAME user and group" \
    && groupadd --system --gid 1000 $USER_NAME \
    && useradd --system --gid $USER_NAME --uid 1000 \
          --shell /bin/bash --create-home $USER_NAME \
    && mkdir /home/$USER_NAME/logs

WORKDIR /home/${USER_NAME}
COPY publish/startup.sh /home/${USER_NAME}/
COPY publish/jmx.access /home/${USER_NAME}/
COPY publish/jmx.password /home/${USER_NAME}/
COPY ${BUILD_DEST} app
WORKDIR /home/${USER_NAME}/app
RUN set -o errexit -o nounset \
    && chown --recursive $USER_NAME:$USER_NAME /home/$USER_NAME \
    && chmod u+x ../*.sh \
    && chmod 0400 ../jmx.*

CMD ["../startup.sh"]

创建Java应用映像

$ cd ~
$ docker build --rm \
        -v ./conf:/var/run/publish \
        --build-arg BUILD_DEST=project/test-server/publish/develop \
        --build-arg EXT_JAVA_OPTS="-server -Dfile.encoding=utf-8 -Dspring.profiles.active=development" \
        -t myrepo/test-server:dev-b1 \
        -f Dockerfile.app.java .

6、用docker运行Java应用

$ docker run -d -u publish -p 8314:8314 \
        --name test-server \
        myrepo/test-server:dev-b1

在虚拟环境中需要通过–network host参数使用宿主机IP

$ docker run --rm --network host -it -u publish -p 8314:8314 \
        --name test-server \
        myrepo/test-server:dev-b1

7、部署到Kubenetes

1） 用指令

kubectl create deployment test-server --image=myrepo/test-server:b103

2）用配置文件

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-server-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-server
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: test-server
        branche: release/ver20200807
    spec:
      #hostNetwork: true
      containers:
      - name: test-server
        image: 172.17.152.44:5000/myrepo/test-server:online-b103
        ports:
        - containerPort: 8314

以下是为ansible-deploy自动化部署工具写的配置文件，定义了系统可配置变量

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{apptag}}-{{version}}
spec:
  replicas: {{replicas}}
  selector:
    matchLabels:
      app: {{apptag}}
      version: {{version}}
  minReadySeconds: {{min_ready_seconds}}
  template:
    metadata:
      labels:
        app: {{apptag}}
        version: {{version}}
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
      hostNetwork: {{host_network}}
      containers:
      - name: {{apptag}}
        image: {{docker_registry}}/{{repository}}:{{version}}-b{{build_no}}
        ports:
        - containerPort: {{server_port}}
          name: server
        - containerPort: {{jmx_port}}
          name: jmx
        resources:
          requests:
            memory: "{{request_memory}}"
            cpu: "{{request_cpu}}"
            ephemeral-storage: "{{request_eph_storage}}"
          limits:
            memory: "{{limit_memory}}"
            cpu: "{{limit_cpu}}"
            ephemeral-storage: "{{limit_ephemeral_storage}}"

说明：
1）映像Tag用构建号是因为，用kubectl set image更新应用时，如果image名没有变化，kubectl不会执行更新动作，所以每次构建都要用一个新的Tag名字

运行

kubectl apply -f deployment.yaml

注意：本地虚拟机测试环境，使用 hostNetwork: true 选项让容器使用宿主机(虚拟机)IP，这样服务在注册服务时取到的IP是宿主机IP，可以直接被Kubernetes集群外访问，如果没有使用此选项，则在注册服务时取到的IP是子网10.244.0.0/24内分配的Pod地址，Kubernetes集群外访问这个服务需要配置一个到此子网的路由。

$ route add -p 10.234.0.0 mask 255.255.0.0 172.17.149.10

3）暴露Kubenetes服务

$ kubectl expose deployment/test-server-deployment --type="NodePort" --port 8314

在Docker容器中创建部署映像

为了让DI过程也可以通过编排系统调度，不仅编译需要在容器中，创建部署映像也希望在容器中进行。

有两种方法可以在Docker中运行Docker，Docker In Docker (dind) 和 Docker Outside of Docker (dood)，dind目前已经不再推荐，建议使用更好的解决方案sysbox；其他参考文章：参考1，参考2。

这里将使用dood方案，以下是Image创建栈

docker build

docker build

Image tq/gradle:6

Image tq/docker:gradle

Image tq/publish:gradle

1、映像 myrepo/docker:gradle

docker配置文件./docker/daemon.json

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "bip":"172.17.123.1/24",
    "insecure-registries": ["172.17.149.10:5000"]
}

Dockerfile.docker

FROM 172.17.149.10:5000/tq/gradle:6

RUN set -o errexit -o nounset \
    && echo "Install git" \
    && yum install -y git \
    && echo "Install docker-ce" \
    && yum install -y yum-utils \
    && yum-config-manager \
        --add-repo https://download.docker.com/linux/centos/docker-ce.repo \
    && yum install -y docker-ce docker-ce-cli containerd.io \
    && yum clean packages

COPY ./docker /etc/docker

RUN set -o errexit -o nounset \
    && yum install -y sudo \
    && echo "gradle ALL=NOPASSWD: ALL" >> /etc/sudoers \
    && yum clean packages

脚本解析

1）运行docker build的用户需要加入docker组，否则会提示无权限

创建映像

# docker build --rm -t 172.17.149.10:5000/tq/docker:gradle -f Dockerfile.docker .
# docker push 172.17.149.10:5000/tq/docker:gradle

2、用于构建并发布的映像 tq/publish:java

启动脚本 publish.sh

#!/bin/bash
su - gradle
echo GIT_BRANCHES=$GIT_BRANCHES
echo GIT_URL=$GIT_URL
echo BUILD_PATH=$BUILD_PATH
echo BUILD_TASK=$BUILD_TASK
echo BUILD_DEST=$BUILD_DEST
echo DOCKER_REGISTRY=$DOCKER_REGISTRY
echo REPOSITORY=$REPOSITORY
echo BUILD_VERSION=$BUILD_VERSION
echo BUILD_NO=$BUILD_NO
echo DEPLOY_SERVER=$DEPLOY_SERVER
echo VERSION_ID=$VERSION_ID
echo USER_TOKEN=$USER_TOKEN
set -o errexit -o nounset \
  && git clone -b $GIT_BRANCHES $GIT_URL project \
  && cd project/$BUILD_PATH \
  && gradle $BUILD_TASK \
  && cd \
  && sudo docker build \
        --build-arg BUILD_DEST=project/$BUILD_PATH/$BUILD_DEST \
        -t $DOCKER_REGISTRY/$REPOSITORY:$BUILD_VERSION \
        -t $DOCKER_REGISTRY/$REPOSITORY:$BUILD_VERSION-b$BUILD_NO . \
  && sudo docker push $DOCKER_REGISTRY/$REPOSITORY:$BUILD_VERSION \
  && sudo docker push $DOCKER_REGISTRY/$REPOSITORY:$BUILD_VERSION-b$BUILD_NO \
  && curl -X PUT "$DEPLOY_SERVER/api/versions/$VERSION_ID/buildno?token=$USER_TOKEN&buildno=$BUILD_NO"

Dockerfile.publish.java

FROM 172.17.149.10:5000/tq/docker:gradle

COPY ssh .ssh
RUN set -o errexit -o nounset \
    && echo "Config git check out key" \
    && chmod 700 .ssh \
    && chmod 600 .ssh/id_rsa \
    && echo StrictHostKeyChecking no >> /etc/ssh/ssh_config \
    && echo UserKnownHostsFile /dev/null >> /etc/ssh/ssh_config

VOLUME /publish
COPY publish.sh publish.sh
COPY Dockerfile.app.java Dockerfile
RUN set -o errexit -o nounset \
    && chmod u+x *.sh \
    && chown --recursive gradle:gradle .
CMD ["./publish.sh"]

脚本说明
1）将git迁出代码的秘钥配置到/home/.ssh/id_rsa
2）git clone 避免更新known_hosts的提示
修改/etc/ssh/ssh_config文件的配置，在最后面添加：

StrictHostKeyChecking no
UserKnownHostsFile /dev/null

创建映像

# docker build --rm -t 172.17.149.10:5000/tq/publish:java -f Dockerfile.publish.java .
# docker push 172.17.149.10:5000/tq/publish:java

3、构建代码并发布应用映像

dood的关键点在于挂载宿主机docker.sock

$ docker run --rm -it  -u gradle \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v conf:/publish \
    --env GIT_BRANCHES=release/ver20200807 \
    --env [email protected]:mygroup/test-server.git \
    --env BUILD_PATH=test-server \
    --env BUILD_TASK=publishDEV \
    --env BUILD_DEST=publish/develop \
    --env JAVA_OPTS="-server" \
    --env EXT_JAVA_OPTS="-Dfile.encoding=utf-8 -Dspring.profiles.active=development" \
    --env DOCKER_REGISTRY=172.17.152.44:5000 \
    --env REPOSITORY=myrepo/test-server \
    --env BUILD_VERSION=dev \
    --env BUILD_NO=1 \
    --env DEPLOY_SERVER=192.168.0.100:8062 \
    --env VERSION_ID=65 \
    --env USER_TOKEN=123432 \
    myrepo/publish:java

4、用Kubernetes进行构建并发布Docker映像

配置文件 publish.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: publish-test-server
spec:
  template:
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
      volumes:
      - name: sock
        hostPath:
            path: /var/run/docker.sock
      - name: conf
        hostPath:
            path: /home/op/test-server/conf
      - name: gradle
        hostPath
            path: /opt/gradle
      containers:
      - name: publish-test-server
        image: myrepo/publish:java
        volumeMounts:
        - name: sock
          mountPath: /var/run/docker.sock
        - name: conf
          mountPath: /var/run/publish
        - name: gradle
          mountPath: /home/gradle/.gradle
        env:
        - name: GIT_BRANCHES
          value: release/ver20200807
        - name: GIT_URL
          value: "[email protected]:mygroup/test-server.git"
        - name: BUILD_PATH
          value: test-server
        - name: BUILD_TASK
          value: publishDEV
        - name: BUILD_DEST
          value: publish/develop
        - name: JAVA_OPTS
          value: "-server"
        - name: EXT_JAVA_OPTS
          value: "-Dfile.encoding=utf-8 -Dspring.profiles.active=development"
        - name: DOCKER_REGISTRY
          value: "172.17.152.44:5000"
        - name: REPOSITORY
          value: "myrepo/test-server"
        - name: BUILD_VERSION
          value: online
        - name: VERSION_ID
          value: 36
        - name: BUILD_NO
          value: 103
        - name: DEPLOY_SERVER
          value: "192.168.0.100:8062"
        - name: USER_TOKEN
          value: "1234"
      restartPolicy: Never
  backoffLimit: 1
  activeDeadlineSeconds: 1800
  ttlSecondsAfterFinished: 3600

apiVersion: batch/v1
kind: Job
metadata:
  name: publish-{{apptag}}-{{version}}
spec:
  template:
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
      volumes:
      - name: sock
        hostPath:
            path: /var/run/docker.sock
      - name: gradle
        hostPath:
            path: /opt/gradle
      - name: publish
        configMap:
            name: publish-config
            defaultMode: 256
            items:
            - key: startup.sh
              path: startup.sh
            - key: jmx.password
              path: jmx.password
            - key: jmx.access
              path: jmx.access
      containers:
      - name: publish
        image: {{docker_registry}}/tq/publish:java
        volumeMounts:
        - name: sock
          mountPath: /var/run/docker.sock
        - name: publish
          mountPath: /home/gradle/publish
        - name: gradle
          mountPath: /home/gradle/.gradle
        env:
        - name: GIT_BRANCHES
          value: {{git_branches}}
        - name: GIT_URL
          value: "{{git_url}}"
        - name: BUILD_PATH
          value: {{build_path}}
        - name: BUILD_TASK
          value: {{build_task}}
        - name: BUILD_DEST
          value: {{build_dest}}
        - name: JMX_PORT
          value: "{{jmx_port}}"
        - name: SERVER_PORT
          value: "{{server_port}}"
        - name: DOCKER_REGISTRY
          value: "{{docker_registry}}"
        - name: REPOSITORY
          value: {{repository}}
        - name: BUILD_VERSION
          value: {{version}}
        - name: VERSION_ID
          value: "{{version_id}}"
        - name: BUILD_NO
          value: "{{build_no}}"
        - name: DEPLOY_SERVER
          value: "192.168.0.100:8062"
        - name: USER_TOKEN
          value: "{{user_token}}"
      restartPolicy: Never
  backoffLimit: 0
  activeDeadlineSeconds: {{active}}
  ttlSecondsAfterFinished: {{ttl}}

运行

kubectl apply -f publish.yaml

5、用于持续集成的映像 tq/build:java（仅构建，不发布与部署）

启动脚本 build.sh

#!/bin/bash
su - gradle
echo GIT_BRANCHES=$GIT_BRANCHES
echo GIT_URL=$GIT_URL
echo BUILD_PATH=$BUILD_PATH
echo BUILD_TASK=$BUILD_TASK
set -o errexit -o nounset \
  && git clone -b $GIT_BRANCHES $GIT_URL project \
  && cd project/$BUILD_PATH \
  && gradle $BUILD_TASK

Dockerfile.build.java

FROM 172.17.149.10:5000/tq/gradle:6

COPY ssh .ssh
RUN set -o errexit -o nounset \
    && echo "Install git" \
    && yum install -y git \
    && yum clean packages \
    && echo "Config git check out key" \
    && chmod 700 .ssh \
    && chmod 600 .ssh/id_rsa \
    && echo StrictHostKeyChecking no >> /etc/ssh/ssh_config \
    && echo UserKnownHostsFile /dev/null >> /etc/ssh/ssh_config

COPY build.sh build.sh
RUN set -o errexit -o nounset \
    && chmod u+x *.sh \
    && chown --recursive gradle:gradle .
CMD ["./build.sh"]

脚本说明
1）将git迁出代码的秘钥配置到/home/.ssh/id_rsa
2）git clone 避免更新known_hosts的提示
修改/etc/ssh/ssh_config文件的配置，在最后面添加：

StrictHostKeyChecking no
UserKnownHostsFile /dev/null

创建映像

# docker build --rm -t 172.17.149.10:5000/tq/build:java -f Dockerfile.build.java .
# docker push 172.17.149.10:5000/tq/build:java

6、用Kubernetes进行持续集成（仅仅构建，不发布与部署）

配置文件 build.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: build-{{apptag}}-{{version}}
spec:
  template:
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
      volumes:
      - name: gradle
        hostPath:
            path: /opt/gradle
      containers:
      - name: build
        image: {{docker_registry}}/tq/build:java
        volumeMounts:
        - name: gradle
          mountPath: /home/gradle/.gradle
        env:
        - name: GIT_BRANCHES
          value: {{git_branches}}
        - name: GIT_URL
          value: "{{git_url}}"
        - name: BUILD_PATH
          value: {{build_path}}
        - name: BUILD_TASK
          value: {{build_task}}
      restartPolicy: Never
  backoffLimit: 0
  activeDeadlineSeconds: {{active}}
  ttlSecondsAfterFinished: {{ttl}}

运行

kubectl apply -f build.yaml

将以上流程用AnsibleDeploy集成后的效果展示

AnsibleDeploy是自己开发的一套基于Ansible的自动化部署工具，有简单的项目与主机管理功能，可以通过自定义脚本构建适合自己的部署流程

附录：Tomcat相关脚本

Tomcat应用映像定义

Dockerfile.app.tomcat

FROM 172.17.149.10:5000/tq/centos-temp:7

WORKDIR /root/
RUN set -o errexit \
        && echo "Downloading Tomcat" \
        && svn -q --username hello --password 123456 \
             --no-auth-cache export svn://192.168.0.100:8000/download/tomcat/tomcat8

FROM 172.17.149.10:5000/tq/jdk:8

ARG BUILD_DEST
ENV LANG en_US.UTF-8    
ENV USER_NAME tomcat
ENV APP_HOME /home/${USER_NAME}

RUN set -o errexit -o nounset \
    && ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
    && echo "Adding $USER_NAME user and group" \
    && groupadd --system --gid 1000 $USER_NAME \
    && useradd --system --gid $USER_NAME --uid 1000 \
          --shell /bin/bash --create-home $USER_NAME

WORKDIR ${APP_HOME}
COPY --from=0 /root/tomcat8 ${APP_HOME}
COPY publish/setenv.sh ${APP_HOME}/bin/
COPY publish/server.xml ${APP_HOME}/conf/
COPY publish/jmx.access ${APP_HOME}/bin/
COPY publish/jmx.password ${APP_HOME}/bin/
COPY ${BUILD_DEST} ${APP_HOME}/webapps/app

RUN set -o errexit -o nounset \
    && chown --recursive $USER_NAME:$USER_NAME $APP_HOME \
    && chmod u+x bin/*.sh \
    && chmod 0400 bin/jmx.*
    
CMD ["bin/catalina.sh","run"]

部署任务映像生成

Dockerfile.publish.tomcat

FROM 172.17.149.10:5000/tq/docker:gradle

COPY ssh .ssh
RUN set -o errexit -o nounset \
    && echo "Config git check out key" \
    && chmod 700 .ssh \
    && chmod 600 .ssh/id_rsa \
    && echo StrictHostKeyChecking no >> /etc/ssh/ssh_config \
    && echo UserKnownHostsFile /dev/null >> /etc/ssh/ssh_config

VOLUME /publish
COPY publish.sh publish.sh
COPY Dockerfile.app.tomcat Dockerfile
RUN set -o errexit -o nounset \
    && chmod u+x *.sh \
    && chown --recursive gradle:gradle .
CMD ["./publish.sh"]

创建映像

# docker build --rm -t 172.17.149.10:5000/tq/publish:tomcat -f Dockerfile.publish.tomcat .
# docker push 172.17.149.10:5000/tq/publish:tomcat

环境变量设置脚本setenv.sh

#!/bin/bash

APP_OUT="$APP_HOME"/logs/startup.log
touch $APP_OUT
echo "==== Start app: $(date -Iseconds) ====" >> "$APP_OUT"

CATALINA_OPTS="-server {{exec_opt}}"
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseG1GC -XX:+PrintClassHistogram -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCDetails"
CATALINA_OPTS="$CATALINA_OPTS -Xloggc:$APP_HOME/logs/gc.log"
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false"
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.port={{jmx_port}} -Dcom.sun.management.jmxremote.rmi.port={{jmx_port}}"
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote.password.file=$APP_HOME/bin/jmx.password -Dcom.sun.management.jmxremote.access.file=$APP_HOME/bin/jmx.access"

echo "Using APP_HOME:   $APP_HOME"  >> "$APP_OUT"
echo "Using JAVA_HOME:  $JAVA_HOME" >> "$APP_OUT"
echo "Using JAR_FILE:   $JAR_FILE"  >> "$APP_OUT"
echo "Using SERVER_PORT: $SERVER_PORT" >> "$APP_OUT"
echo "Using CATALINA_OPTS:  $CATALINA_OPTS" >> "$APP_OUT"

export CATALINA_OPTS
export LANG=en_US.UTF-8

服务配置文件server.xml

<Server port="4561" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
  <Service name="Catalina">
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" maxThreads="{{ max_threads }}" minSpareThreads="{{ min_threads }}"/>   
    <Connector port="{{ http_port }}" protocol="HTTP/1.1"
               connectionTimeout="10000"
               executor="tomcatThreadPool"
               acceptorThreadCount="2"
               acceptCount="1024"
               processorCache="6000"
               compression="on"
               compressionMinSize="1024"
               compressableMimeType="text/html,text/xml,text/javascript,text/css,text/plain,application/json"
               redirectPort="{{ https_port }}" />

    <Connector port="4562" protocol="AJP/1.3" redirectPort="{{ https_port }}" />
    <Engine name="Catalina" defaultHost="localhost">
      <Host name="localhost"  appBase="webapps/app" unpackWARs="false" autoDeploy="false">
        <Context path="" docBase=""  reloadable="true" caseSensitive="false" debug="0">Context>
      Host>
    Engine>
  Service>
Server>