GitLab CI/CD实战

GitLab CI/CD实战

前置知识点(Tips)

GitLab CI / CD

GitLab CI / CD是GitLab内置的工具，用于通过连续方法进行软件开发：

1. 持续集成 Continuous Integration (CI)
2. 持续交付 Continuous Delivery (CD)
3. 持续部署 Continuous Deployment (CD)

Runner

GitLab Runner是一个开源项目，用于运行您的作业并将结果发送回GitLab。它与GitLab CI / CD结合使用，GitLab CI / CD是GitLab随附的用于协调作业的开源持续集成服务。
docker安装教程

Docker

Docker 是个划时代的开源项目，它彻底释放了计算虚拟化的威力，极大提高了应用的维护效率，降低了云计算应用开发的成本！使用 Docker，可以让应用的部署、测试和分发都变得前所未有的高效和轻松！

Compose

Docker Compose 是 Docker 官方编排（Orchestration）项目之一，负责快速的部署分布式应用。

我们今天在gitlab新建一个简单的后端项目flask-ci-test，实现flask+mysql+redis（flask是python的一种比较流行的web框架）的简单实战部署。

1.安装Runner

我们推荐采用了docker安装runner方式。

1. 使用本地系统卷安装启动Runner容器

Linux

   docker run -d --name gitlab-runner --restart always \
     -v /srv/gitlab-runner/config:/etc/gitlab-runner \
     -v /var/run/docker.sock:/var/run/docker.sock \
     gitlab/gitlab-runner:latest

Mac（On macOS, use /Users/Shared instead of /srv.）

   docker run -d --name gitlab-runner --restart always \
     -v /Users/Shared/gitlab-runner/config:/etc/gitlab-runner \
     -v /var/run/docker.sock:/var/run/docker.sock \
     gitlab/gitlab-runner:latest

2. 注册runner
   2.1. 运行注册命令

Linux

docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

Mac（On macOS, use /Users/Shared instead of /srv.）

docker run --rm -it -v /Users/Shared/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

配置项目runner的URL和token

Settings -> CI/CD -> Runners(Collapse)

2.2. 输入您的GitLab实例URL：

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )
https://gitlab.com

2.3. 输入你配置项目runner的token：

Please enter the gitlab-ci description for this runner
[hostname] my-runner

2.4. 输入与Runner关联的标签，稍后您可以在GitLab的UI中进行更改

Please enter the gitlab-ci tags for this runner (comma separated):
flask-ci-test

2.5. 输入Runner执行器

Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell:
docker

2.6. 如果您选择Docker作为执行程序，则系统会要求您提供默认图片，以供未在.gitlab-ci.yml中定义图片的项目使用：

Please enter the Docker image (eg. ruby:2.6):
alpine:latest

3. 一行注册命令

sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --executor "docker" \
  --docker-image alpine:latest \
  --description "docker-runner" \
  --tag-list "docker,aws" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

4. 更新配置

docker restart gitlab-runner

5. GitLab Runner读取日志配置

docker logs gitlab-runner

6. 查看runner配置(Mac)

cat /Users/Shared/gitlab-runner/config/config.toml

3. 使用Compose配置数据库

3.1. docker-compose.yml

version: '3.1'

services:
  db:
    image: postgres:10
    environment:
      POSTGRES_PASSWORD: xxxxxx
      POSTGRES_USER: postgres
      PGDATA: /data
    volumes:
      - "${PG_DATA}:/data"
    ports:
      - "${PG_PORT}:5432"
    expose:
      - "5432"
    container_name: ${NET}_db
    networks:
      - my_network

  redis:
    image: redis:5
    environment:
      REDIS_PASSWORD: "xxxxxx"
    command: [
      "bash", "-c",
      '
         docker-entrypoint.sh
         --requirepass "$$REDIS_PASSWORD"
      '
    ]
    volumes:
      - "${REDIS_DATA}:/data"
    expose:
      - "6379"
    ports:
      - "${REDIS_PORT}:6379"
    container_name: ${NET}_redis
    networks:
      - my_network

networks:
  my_network:
    external:
      name: ${NET}

3.2. env.sh（环境变量配置【数据库端口，挂在磁盘】）

#!/usr/bin/env bash
export NET=flask_ci_test_dev
export PG_PORT=20001
export REDIS_PORT=20002
export PG_DATA=/data/${NET}_db
export REDIS_DATA=/data/${NET}_redis

3.3. 部署mysql，redis

source ./env.sh
mkdir ${NET} && cp ./docker-compose.yml ./${NET} && cd ${NET}
docker network ls | grep ${NET} > /dev/null 2>&1 || docker network create ${NET}
docker-compose pull
docker-compose up -d --force-recreate

4.flask-ci-test(配置)

4.1. requirements.txt (Python 需要安装)

click==7.1.2
Flask==1.1.2
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
psycopg2==2.7.4
psycopg2-binary==2.7.4
redis==2.10.6
six==1.15.0
Werkzeug==1.0.1

4.2. hello.py(Python代码【Flask】)

"""
@author: magician
@file:   hello.py.py
@date:   2020/6/29
"""
import psycopg2
import redis
from flask import Flask

app = Flask(__name__)


def connect_pg():
    """
    connect pgsql TODO: 连接做成单例
    :return:
    """
    conn = psycopg2.connect(dbname="postgres",
                            user="postgres",
                            password="huansi@2017",
                            host="flask_ci_test_dev_db",
                            port="5432")

    return conn


def connect_redis():
    """
    connect redis
    :return:
    """
    r = redis.Redis(host='flask_ci_test_dev_redis',
                    port=6379,
                    db=0,
                    password='huansi@2017')

    return r


@app.route('/')
def hello_world():
    return 'Hello, World!'


@app.route('/')
def hello_db(tag):
    """
    hello db
    :param tag:
    :return:
    """
    if tag == 'pgsql':
        cur = connect_pg().cursor()
        cur.execute("SELECT content FROM ci_content;")
        ci_content = cur.fetchone()
        content = str(ci_content[0]) if ci_content else '*'
    elif tag == 'redis':
        r = connect_redis()
        r.set('content', 'Hello,Redis!')
        content = r.get('content')
    else:
        content = 'Hello,World!'

    return content

4.3. Dockfile(Docker部署脚本)

FROM python:3.5

WORKDIR /app

COPY . /app

RUN pip3 install -r requirements.txt -i https://pypi.douban.com/simple

ENV C_FORCE_ROOT true

ENV DISPLAY=":0" C_FORCE_ROOT=true PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

RUN cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime


EXPOSE 5000

CMD ["/bin/bash","docker_entry.sh"]

4.4. docker_entry.sh(gun运行后端项目)

#!/usr/bin/env bash

#if [ "$MIGRATE" = "1" ]; then
#    python3 manage.py db migrate --rev-id 1234
#    python3 manage.py db upgrade
#    python3 manage.py load_data
#fi

#python3 manage.py del_version

#nohup celery worker -B -A celery_worker.celery --loglevel debug > celery.log 2>&1 &
#
#python wsgi.py

if [ "$RUN" = "celery" ];then
   celery worker -B -A celery_worker.celery --loglevel INFO
#    python wsgi.py
else
   gunicorn -b 0.0.0.0:5000 -w 25 hello:app  --log-level debug
fi

4.5. .gitlab-ci.yml(GitLab CICD脚本)

image: docker:19.03.8

stages:
  - build
  - deploy

variables:
  APP_NAME: flask_ci_test
  DOCKER_TLS_CERTDIR: ""
  IMAGE_NAME: ${APP_NAME}_api:${CI_COMMIT_REF_NAME}
  CON_NAME: ${APP_NAME}_api_${CI_COMMIT_REF_NAME}

build:
  stage: build
  retry: 2
  tags:
    - flask_ci_test
  script:
    - docker build -t ${IMAGE_NAME} .

deploy-dev:
  stage: deploy
  retry: 2
  tags:
    - flask_ci_test
  variables:
    APP_PORT: 20112
    DB_HOST: ${APP_NAME}_${CI_COMMIT_REF_NAME}_db
    DB_PORT: 5432
    REDIS_HOST: ${APP_NAME}_${CI_COMMIT_REF_NAME}_redis
    REDIS_PORT: 6379
  script:
    - |
      docker ps -a | grep -i ${CON_NAME} > /dev/null 2>&1
      docker stop ${CON_NAME}
      docker rm ${CON_NAME}
    - >
      docker run --name ${CON_NAME} --network ${APP_NAME}_${CI_COMMIT_REF_NAME} -d -p ${APP_PORT}:5000
      -e "REDIS_HOST=${REDIS_HOST}" -e "REDIS_PORT=${REDIS_PORT}"
      -e "DB_HOST=${DB_HOST}" -e "DB_PORT=${DB_PORT}"
      $IMAGE_NAME
  only:
    - dev

4.6. run.sh(本地运行项目脚本)

export FLASK_APP=hello.py
flask run

5.项目最终效果

1. CICD效果(每次推代码，自动部署)
   
2. 网页效果

• http://127.0.0.1:20112/
• http://127.0.0.1:20112/mysql
• http://127.0.0.1:20112/redis
  

6.参考文档：

1. GitLab CICD
2. GitLab Runner
3. Docker — 从入门到实践

本文有点长，感谢大家阅读，喜欢的读者麻烦点个赞，GitLab CI/CD实战到此完美实现！