百度CCE的k8s部署过程记录

一、机器规划

1、登录百度云账号，购买6台机器，其中master节3台，worker节点3台。
2、所有机器采用共享NAT上网，因此另需购买云NAT网关并绑定IP(需要两个EIP)。
3、购买弹性EIP数量3个，其中两个用于NAT，另外一个用于ingress BLB。
PS：注意所有机器及NAT，必须在同一可通信网段内。

序号	实例名称	IP
1	k8s-hbgs-master1	192.168.64.10
2	k8s-hbgs-master2	192.168.64.8
3	k8s-hbgs-master3	192.168.64.7
4	k8s-hbgs-worker1	192.168.64.9
5	k8s-hbgs-worker2	192.168.64.12
6	k8s-hbgs-worker3	192.168.64.11

PS：CCE集群搭建相对简单，直接购买即可，因此略。

二、容器及镜像

1、采购容器镜像服务CCR ，用于托管docker镜像制品。
2、采用Dockerfile制造docker镜像，并推送至CCR（制作过程略）。
3、采用spring boot 及vue.js前后分离框架。
（1）后端java，镜像制作Dockerfile文件如下：

# VERSION 0.0.1
# Author: liurongming
# 基础镜像使用java
FROM jdk8:v1.8.0_251
# 作者
LABEL maintainer="liurongming"
# VOLUME 指定了临时文件目录为/tmp。
# 其效果是在主机 /var/lib/docker 目录下创建了一个临时文件，并链接到容器的/tmp
VOLUME /tmp

# 工作目录
WORKDIR /online
# 编译指定文件名
# 例子：[--build-arg 'JAR_FILE=xxx.jar']
# 默认加载当前.jar
ARG JAR_FILE='*.jar'
RUN echo ${JAR_FILE}
# 将jar包添加到容器中并更名为app.jar
ADD ${JAR_FILE} app.jar

# 指定启动参数
# 改变使用[-e 'JAVA_OPTS=-Xms512m']
ARG DEAULT_OPTS='-Xms3g -Xmx3g -Xmn1g -Xss1024K -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m'
ENV JAVA_OPTS=$DEAULT_OPTS

# 默认配置sit环境
# 改变使用[-e 'CE=dev']
ARG DEAULT_CE='sit'
ENV CE=$DEAULT_CE

# 更新时区
RUN sh -c 'touch /app.jar; ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime; echo "Asia/Shanghai" > /etc/timezone'

# 设置NACOS环境变量
# 设置通过 [-e 'NACOS_NAMESPACE=value']
ENV NACOS_NAMESPACE=${NACOS_NAMESPACE}
# 设置通过 [-e 'NACOS_HOST=value']
ENV NACOS_HOST=${NACOS_HOST}
# 设置通过 [-e 'NACOS_PORT=value']
ENV NACOS_PORT=${NACOS_PORT}

# 启动入口
# 强制文件编码:UTF-8
# 强制java时区:GMT+08
ENTRYPOINT ["sh","-c","java -jar ${JAVA_OPTS} -Dfile.encoding=UTF-8 -Duser.timezone=GMT+08 -Djava.security.egd=file:/dev/./urandom ./app.jar --spring.profiles.active=$CE"]

提前制作java私有镜像：
准备物料：jdk-8u251-linux-x64.tar.gz 与Dockerfile 处于同级别目录，
执行命令：docker build -t jdk8:v1.8.0_251 .

# CentOS with JDK 8
# Author   liurongming

# build a new image with basic  centos
FROM centos:7
# who is the author
MAINTAINER centos7-java1.8

# make a new directory to store the jdk files
# RUN mkdir /usr/local/java
WORKDIR /usr/local/java


# copy the jdk  archive to the image,and it will automaticlly unzip the tar file
ADD jdk-8u251-linux-x64.tar.gz /usr/local/java/

# make a symbol link
RUN ln -s /usr/local/java/jdk1.8.0_251 /usr/local/java/jdk

# set environment variables
ENV JAVA_HOME /usr/local/java/jdk
ENV JRE_HOME ${JAVA_HOME}/jre
ENV CLASSPATH .:${JAVA_HOME}/lib:${JRE_HOME}/lib
ENV PATH ${JAVA_HOME}/bin:$PATH

CMD ["java","-version"]

（2）前端，镜像制作Dockerfile文件如下：

# VERSION 0.0.1
# Author: liurongming
FROM nginx:stable-alpine 

COPY ./dist /usr/share/nginx/html/
COPY ./docker/default.conf /etc/nginx/conf.d/default.conf
COPY ./docker/nginx.conf /etc/nginx/nginx.conf

# select timezone as Shanghai
RUN sh -c 'ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime; echo "Asia/Shanghai" > /etc/timezone'

EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

在Dockerfile项目同级目录下，建立docker目录下新建default.conf 和nginx.conf
其中nginx.conf内容为：


user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;
    # 是否启动gzip压缩,on代表启动,off代表开启
    gzip  on;

    # 需要压缩的常见静态资源
    gzip_types text/plain application/javascript   application/x-javascript text/css application/xml text/javascript application/x-httpd-php image/jpeg image/gif image/png;

    # 由于nginx的压缩发生在浏览器端而微软的ie6很坑爹,会导致压缩后图片看不见所以该选项是禁止ie6发生压缩
    gzip_disable "MSIE [1-6]\.";

    # 如果文件大于1k就启动压缩
    gzip_min_length 1k;

    # 以16k为单位,按照原始数据的大小以4倍的方式申请内存空间,一般此项不要修改
    gzip_buffers 4 16k;

    # 压缩的等级,数字选择范围是1-9,数字越小压缩的速度越快,消耗cpu就越大
    gzip_comp_level 2;

    include /etc/nginx/conf.d/*.conf;
}

其中default.conf的内容为：

server {
    listen  80;
    server_name localhost;
    # access_log /var/log/nginx/pro.log;

    location  / {
        # 传递真实的请求头信息
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         
        # 限制文件大小为10G git 
        client_max_body_size    10240m; 
        client_body_buffer_size 256k;
        proxy_connect_timeout 1200;
        proxy_read_timeout  1200;
        proxy_send_timeout  6000;
        proxy_buffer_size  32k;
        proxy_buffers   4 64k;
        proxy_busy_buffers_size 128k;
        proxy_temp_file_write_size 10m;
            
        # 允许跨域
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Headers X-Requested-With;
        add_header Access-Control-Allow-Methods GET,POST,OPTIONS;

        # 仅缓存页面
        if ($request_filename ~* .*\.(?:htm|html)$) {
            add_header Cache-Control "private, no-store, no-cache, must-revalidate, proxy-revalidate";
        }
        root   /usr/share/nginx/html;
        index  index.html index.htm;
     }
   
    # 将DNS指向kubernetes集群内的DNS
    resolver kube-dns.kube-system.svc.cluster.local valid=5s;
    set $endpoint_service service-xxx-gateway.xxx-site.svc.cluster.local;

    location ~ ^/service-.*$ {
        # 传递真实的请求头信息
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
        # 调优
        client_max_body_size    10240m; 
        client_body_buffer_size 256k;
        proxy_connect_timeout 1200;
        proxy_read_timeout  1200;
        proxy_send_timeout  6000;
        proxy_buffer_size  32k;
        proxy_buffers   4 64k;
        proxy_busy_buffers_size 128k;
        proxy_temp_file_write_size 10m;

        # 允许跨域   
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Headers X-Requested-With;
        add_header Access-Control-Allow-Methods GET,POST,OPTIONS;
    
        # rewrite ^.+api/?(.*)$ /$1 break;
        include uwsgi_params;
        proxy_pass http://$endpoint_service:8000;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
    
    error_page 500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

PS：其中 location ~ ^/service-.*$ {}可以不用配置，采用ingress 路由转发代替。
极简配置，只配置web前端页面，接口直接做ingress路由转发。

server {
    listen  80;
    server_name localhost;
    # access_log /var/log/nginx/pro.log;

    location  / {
        # 传递真实的请求头信息
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         
        # 限制文件大小为10G git 
        client_max_body_size    10240m; 
        client_body_buffer_size 256k;
        proxy_connect_timeout 1200;
        proxy_read_timeout  1200;
        proxy_send_timeout  6000;
        proxy_buffer_size  32k;
        proxy_buffers   4 64k;
        proxy_busy_buffers_size 128k;
        proxy_temp_file_write_size 10m;
            
        # 允许跨域
        add_header Access-Control-Allow-Origin *;
        add_header Access-Control-Allow-Headers X-Requested-With;
        add_header Access-Control-Allow-Methods GET,POST,OPTIONS;

        # 仅缓存页面
        if ($request_filename ~* .*\.(?:htm|html)$) {
            add_header Cache-Control "private, no-store, no-cache, must-revalidate, proxy-revalidate";
        }
        root   /usr/share/nginx/html;
        index  index.html index.htm;
     }
   
    error_page 500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

二、配套的jenkins配置

1、采用git进行源码托管
2、采用jenkins进行构建
3、登录机器提前镜像docker login 操作。
（1）后端构建jenkinsfile如下：

#!/usr/bin/env groovy

pipeline {
    agent any

    options {
        timestamps() // 日志显示时间
        skipDefaultCheckout() // 禁止默认检出
        disableConcurrentBuilds() // 不允许并行执行Pipeline
        timeout(time: 1, unit: 'HOURS') // 设置超时时间
    }

    environment {
        GIT_CREDENTIAL_KEY = "gitlab_private_key"
        GIT_URL = "ssh://[email protected]:10089/yyjcjfb/xxx/backend/reserved-xxx-cloud.git"
        build_target = "Docker"
    }

    parameters {
        booleanParam(name: 'skip_test', defaultValue: true, description: '你需要在部署之前执行自动化测试么 ?')
        choice(name: 'target_env', choices: ['sit', 'dev','prod'], description: '请选择构建目标环境')
        extendedChoice defaultValue: "reserved-gateway,reserved-service-auth,reserved-service-content,reserved-service-member,reserved-service-travel,reserved-service-equity,reserved-service-police,reserved-service-system,reserved-service-statistics", description: '请选择要发布的项目', multiSelectDelimiter: ',', name: 'choose_project', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value: 'reserved-gateway,reserved-service-auth,reserved-service-content,reserved-service-member,reserved-service-travel,reserved-service-equity,reserved-service-police,reserved-service-system,reserved-service-statistics', visibleItemCount: 10
        listGitBranches branchFilter: '.*', credentialsId: "gitlab_private_key", defaultValue: 'refs/heads/develop', name: 'choose_branch', quickFilterEnabled: false, remoteURL: "ssh://[email protected]:10089/yyjcjfb/xxxx/backend/reserved-xxx-cloud.git", selectedValue: 'DEFAULT', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG'
    }
    stages {
        stage('选择环境') {
            steps {
                timeout(time: 10, unit: 'MINUTES') {
                    script {
                        // 编译环境检测
                        checkBuildEnviroment(build_target)
                    }
                }
            }
        }
        stage('拉取源码') {
            steps {
                // 根据选择分支拉取代码
                checkout([$class           : 'GitSCM',
                          branches         : [[name: "${env.choose_branch}"]],
                          extensions       : [],
                          userRemoteConfigs: [[credentialsId: "${GIT_CREDENTIAL_KEY}",
                                               url          : "${GIT_URL}"]]
                ])
                script  {
                    sh '''
                    # 查看修改配置
                    cd ${WORKSPACE}
                    find ./ -type f -name '*.yml' | grep -vE 'reserved-api|reserved-common|reserved-configure|target'|xargs cat |grep namespace
                '''
                }
            }
        }
        stage('构建制品') {
            steps {
                script {
                    // 构建制品
                    doBuild(build_target)
                }
            }
        }
        stage('部署发布') {
            steps {
                script {
                    // 部署发布
                    doDeploy(target_env)
                }
            }
        }
        stage('结果通知') {
            steps {
                script {
                    // 结果通知
                    doNotify(build_target)
                }
            }
        }
    }

    post {
        always {
            script {
                sh '''
                    # 环境还原
                    cd ${WORKSPACE}
                    git reset --hard
                '''
            }
        }
        success {
            script {
                // 结果通知
                echo "success======"
            }
        }
        failure {
            script {
                // 结果通知
                echo "failure======"
            }
        }
        aborted {
            script {
                // 结果通知
                echo "aborted======"
            }
        }
    }
}

/** 执行环境检测 */
def checkBuildEnviroment(String option) {
    println("选择项目：${env.choose_project}")
    println("选择分支：${env.choose_branch}")

    switch (option) {
        case "Shell":
            println("选择 Shell 原生制品")
            sh '''
                                echo "构建环境检测"
                                pwd
                                java -version
                                mvn -v
                            '''
            break
        case "Docker":
            println("选择 Docker 镜像制品")
            sh '''
                                echo "构建环境检测"
                                pwd
                                java -version
                                mvn -v
                                docker -v
                                sudo docker info
                            '''
            break
    }
}

/** 执行编译 */
def doBuild(String option) {
    if (option == 'Docker') {
        sh '''
            # 镜像构建
            # mvn -f pom.xml clean package -Dmaven.test.skip=true -U docker:build
            mvn -f pom.xml clean package -Dmaven.test.skip=true -U 
        '''
    } else {
        sh '''
           # 原生构建
           mvn -f pom.xml clean package -Dmaven.test.skip=true -U 
        '''
    }
}

/** 执行发布 */
def doDeploy(String option) {
    println("部署环境为：${option} ...") 
    if (option == 'prod') {
        echo "选择生产环境部署发布中..."
    } else if(option == 'sit') {
        echo "选择测试环境部署发布中..."
        def tasks = [:]
        for (curProject in  choose_project.tokenize(',')) {
           def sendProject = curProject
           println("正在执行部署：${sendProject} ...") 
           tasks["deploying-${sendProject}"] = {           
               dir("${sendProject}") {
                    sh '''
                        cur_dir=`pwd`
                        if [ ! -d $cur_dir ];then
                            echo "${cur_dir} is  not exist"
                            exit 1
                        fi
                        
                        echo "当前路径为：${cur_dir}" 
                        cp ../Dockerfile ./target/
                        
                        cur_pj=`echo ${cur_dir} | awk -F '/' '{print \$NF}' |xargs`
                        cur_jar=`find ./target/ -maxdepth 1 -type f -name '*.jar'|awk -F '/' '{print \$NF}'|xargs`
                        
                        cd ./target/
                        # sudo docker images |grep ${cur_pj} | awk '{print \$3}' | xargs sudo docker rmi
                        
                        sudo docker build --build-arg "JAR_FILE=${cur_jar}" -t ccr地址/pri-nsp-xx命名空间/${cur_pj} .
                        sudo docker push ccr地址/pri-nsp-xx命名空间/${cur_pj}:latest
                    '''
               }
            }
        }
        
        parallel tasks
    } else {
        echo "选择开发环境部署发布中..."
        def tasks = [:]
        for (curProject in  choose_project.tokenize(',')) {
           def sendProject = curProject
           println("正在执行部署：${sendProject} ...") 
           tasks["deploying-${sendProject}"] = {           
               dir("${sendProject}") {
                    sh '''
                        cur_dir=`pwd`
                        if [ ! -d $cur_dir ];then
                            echo "${cur_dir} is  not exist"
                            exit 1
                        fi
                        
                        echo "当前路径为：${cur_dir}" 
                        cp ../Dockerfile ./target/
                        
                        cur_pj=`echo ${cur_dir} | awk -F '/' '{print \$NF}' |xargs`
                        cur_jar=`find ./target/ -maxdepth 1 -type f -name '*.jar'|awk -F '/' '{print \$NF}'|xargs`
                        
                        cd ./target/
                        # sudo docker images |grep ${cur_pj} | awk '{print \$3}' | xargs sudo docker rmi
                        
                        sudo docker build --build-arg "JAR_FILE=${cur_jar}" -t ccr地址/ccr-pri-nsp-dev/${cur_pj} .
                        sudo docker push ccr地址/ccr-pri-nsp-dev/${cur_pj}:latest
                    '''
               }
            }
        }
        
        parallel tasks
    }
}


/** 结果通知 */
def doNotify(String option) {
    if (option == 'Shell') {
        echo "结果通知..."
    } else {
        echo "结果通知..."
    }
}

（2）前端构建jenkinsfile如下：

#!/usr/bin/env groovy

pipeline {
    agent any

    options {
        timestamps() // 日志显示时间
        skipDefaultCheckout() // 禁止默认检出
        disableConcurrentBuilds() // 不允许并行执行Pipeline
        timeout(time: 1, unit: 'HOURS') // 设置超时时间
    }

    environment {
        GIT_CREDENTIAL_KEY = "gitlab_private_key"
        GIT_URL = "ssh://[email protected]:10089/yyjcjfb/xxxx/web/reserved-web-manage.git"
        build_target = "Docker"
    }

    parameters {
        listGitBranches branchFilter: '.*', credentialsId: "gitlab_private_key", defaultValue: 'refs/heads/release', name: 'choose_branch', quickFilterEnabled: false, remoteURL: "ssh://[email protected]:10089/yyjcjfb/xxxx/web/reserved-web-manage.git", selectedValue: 'DEFAULT', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH_TAG'
        choice(name: 'target_env', choices: ['sit', 'dev','prod'], description: '请选择构建目标环境')
    }
    stages {
        stage('选择环境') {
            steps {
                timeout(time: 10, unit: 'MINUTES') {
                    script {
                        // 编译环境检测
                        checkBuildEnviroment(build_target)
                    }
                }
            }
        }
        stage('拉取源码') {
            steps {
                // 根据选择分支拉取代码
                checkout([$class           : 'GitSCM',
                          branches         : [[name: "${env.choose_branch}"]],
                          extensions       : [],
                          userRemoteConfigs: [[credentialsId: "${GIT_CREDENTIAL_KEY}",
                                               url          : "${GIT_URL}"]]
                ])
            }
        }
        stage('构建制品') {
            steps {
                script {
                    // 构建制品
                    doBuild(target_env)
                }
            }
        }
        stage('部署发布') {
            steps {
                script {
                    // 部署发布
                    doDeploy(target_env)
                }
            }
        }
        stage('结果通知') {
            steps {
                script {
                    // 结果通知
                    doNotify(build_target)
                }
            }
        }
    }

    post {
        always {
            script {
                sh '''
                    # 环境还原
                    cd ${WORKSPACE}
                    git reset --hard
                '''
            }
        }
        success {
            script {
                // 结果通知
                echo "success======"
            }
        }
        failure {
            script {
                // 结果通知
                echo "failure======"
            }
        }
        aborted {
            script {
                // 结果通知
                echo "aborted======"
            }
        }
    }
}

/** 执行环境检测 */
def checkBuildEnviroment(String option) {
    println("选择项目：${env.choose_project}")
    println("选择分支：${env.choose_branch}")

    switch (option) {
        case "Shell":
            println("选择 Shell 原生制品")
            sh '''
                                echo "构建环境检测"
                                pwd
                                node -v
                                npm config set registry https://registry.npm.taobao.org
                                npm config get registry
                            '''
            break
        case "Docker":
            println("选择 Docker 镜像制品")
            sh '''
                                echo "构建环境检测"
                                pwd   
                                node -v
                                npm config set registry https://registry.npm.taobao.org
                                npm config get registry
                                docker -v
                            '''
            break
    }
}

/** 执行编译 */
def doBuild(String option) {
    if (option == 'prod') {
         sh '''
            # 镜像构建
           echo "docker build"
           cd $WORKSPACE && rm -rf dist && npm install && npm run build:prod
           sudo docker build -t ccr地址/ccr-pri-nsp-prod/reserved-web-manage .
           
        '''
    } else if (option == 'sit') {
        sh '''
            # 镜像构建
           echo "docker build"
           cd $WORKSPACE && rm -rf dist && npm install && npm run build:prod
           sudo docker build -t ccr地址/pri-nsp-hbgs/reserved-web-manage .
           
        '''
    } else {
        sh '''
            # 镜像构建
           echo "docker build"
           cd $WORKSPACE && rm -rf dist && npm install && npm run build:prod
           sudo docker build -t ccr地址/ccr-pri-nsp-dev/reserved-web-manage .
           
        '''
    }
}

/** 执行发布 */
def doDeploy(String option) {
    println("部署环境为：${option} ...") 
    if (option == 'prod') {
        echo "选择生产环境部署发布中..."
        sh '''
           # 上传镜像   
           sudo docker push ccr地址/ccr-pri-nsp-prod/reserved-web-manage:latest
        '''
    } else if(option == 'sit'){
       echo "选择测试环境部署发布中..."
       sh '''
           # 上传镜像   
           sudo docker push ccr地址/pri-nsp-hbgs/reserved-web-manage:latest
        '''
    } else{
        echo "选择开发环境部署发布中..."
        sh '''
           # 上传镜像   
           sudo docker push ccr地址/ccr-pri-nsp-dev/reserved-web-manage:latest
        '''
    }
}

/** 结果通知 */
def doNotify(String option) {
    if (option == 'Shell') {
        echo "结果通知..."
    } else {
        echo "结果通知..."
    }
}

四、部署服务

1、部署服务相对比较简单，之前才考百度云的文档部署即可。
2、部署完成以后，直接重启容器即可进行相应的更新（前提是设置好策略）
3、部署步骤：配置环境变量configmap》新建命名空间》新建无状态工作负载》新建NodePort服务》新建Ingress 》绑定外网IP。

PS：（1）如需新建LoadBalancer服务时，去掉下面这行注释,目的是不创建EIP节省资源

# service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true" # LB Service 不分配 EIP

（2）直接将访问网关的api指向k8s内部的网关服务，前提是已经创建好网关服务器的NodePort服务。如访问：/service-* 的接口时，全部执向service-reserved-gateway服务及端口。这样做的好处时，访问接口时不经过前端的nginx代理，直接走ingress代理，就可以采用第二章节的极简单的web配置，当前端项目停止后接口仍能够访问，这样减少代理层级链路同时也提升转发性能，继而减少出错的端口及减少对前端的依赖。

百度CCE的k8s部署过程记录

一、机器规划

二、容器及镜像

二、配套的jenkins配置

四、部署服务

你可能感兴趣的:(百度CCE的k8s部署过程记录)