2022-06-10

Ansible playbook 自动化部署方案

1. 方案背景

当前Jlite非标部署比较复杂，需要部署的服务比较多，每个服务又有很多配置。一是部署时间比较长、二是部署不够标准(基础软件版本、日志文件配置等不统计)。为了使部署更加简单、方便、快捷，特提供本部署方案。

ansible playbook自动化部署技术目前比较成熟、文档齐全，固本次选择ansible playbook为自动化部署技术方案。

2. Ansible 简介

Ansible是一种常用的自动运维化工具，基于python开发，分布式，无需客户端，轻量级，配置语言采用YAML。

2.1. Ansible的特性：

1. 模块化：调用特定的模块，完成特殊的任务。
2. Paramiko(python对ssh的实现)，PyYaml，jinja2(模块语言)三个关键模块。
3. 支持自定义模块，可使用任何编程语言写模块。
4. 基于python语言实现。
5. 部署简单，基于python和SSH(默认已安装)，agentless，无需代理不依赖KPI(无需SSL)。
6. 安全，基于OpenSSH
7. 幂等性：一个任务执行一次和执行n遍效果一样，不因重复执行带来意外情况。
8. 支持playbook编排任务，YAML格式，编排任务，支持丰富的数据结构。
9. 较强大的多层解决方案role。

2.2 Ansible的作用目标：

1. 自动化部署APP
2. 自动化管理配置项
3. 自动化的持续交付
4. 自动化的云服务管理

3. Playbook简介

playbook是ansible用于配置，部署和管理托管主机剧本，通过playbook的详细描述，执行其中一系列tasks，可以让远程主机达到预期状态。playbook是由一个或多个"play"组成的列表。

也可以说，playbook字面意思及剧本，现实中由演员按剧本表演，在ansible中由计算机进行安装，部署应用，提供对外服务，以及组织计算机处理各种各样的事情

3.1. Playbook 核心元素

• Hosts

  执行的远程主机列表(应用在哪些主机上)

• Tasks

  任务集

• Variables

  内置变量或自定义变量在playbook中调用

• Templates

  可替换模板文件中的变量并实现一些简单逻辑的文件

• Handlers/Notify

  两个为一个组合，由特定条件触发的操作，满足条件执行，不满足不执行。

• Tags

  指定某条任务执行，用于选择运行playbook中的部署代码。

4. 部署组件图

结合Ansible Playbook、Docker、Docker Compose、Docker Swarm集群、Nacos功能特点。可以使用如下组件图方案对Jlite实施自动化部署。

1654850242(1).png

4.1. 本地nacos配置信息说明

因为Ansible Playbook并没有提供一个界面配置，所以我们使用Nacos作为所有部署时参数的配置中心。

• 以客户为 worksapce(工作空间)，SAAS环境的话，每个SAAS环境为一个worksapce
• 每个服务为一个group
• 一个服务的不同配置为dataId

4.1.1 远程服务器资源

• 远程服务器资源配置信息 remote-hosts.yaml

  ---
  remote_hosts:
    swarm:
      # docker swarm 集群的leader服务器ip地址
      leader:
      - 10.10.205.115
    # docker swarm 集群的manager服务器ip地址
    manager:
      - 10.10.205.114
    # docker swarm 集群的worker服务器ip地址
    worker:
      - 10.10.205.95
      - 10.10.205.94
  ...
  

4.1.2 服务部署信息

• 需要部署的服务参数 service-docker-compose-parameter.yaml ，playbook用于生成最终的docker-compose.yaml文件。

  ---
  # 根docker-compose.yaml文件类似
  services:
    #jlite服务
    jlite:
      #版本号
      version: 3.7.11
      #外发时的镜像文件名
    image: osdfo3234234220220611
    #端口号
    ports:
        - "1081:1081"
    deploy:
      #节点数
        replicas: 2
        
    #redis服务
    redis:
      ports:
        - "6397:6397"
    password: jsicp
    
    #lucene服务
    lucene:
      ports:
        - "8108:8108"
  ...
  

4.1.3 每个服务依赖的配置

每个服务自已依赖的nacos配置不变，只需要要本地配置就好，自动化脚本自动下载后上传到远程部署服务器的nacos上。

5. 服务的 docker-compose.yaml

每个服务的docker-compose.yaml文件由研发提供，可以在里面可以使用参数占位符。参数由前面的service-docker-compose-parameter.yaml定义。

version: '3'

services:
  jlite:
    image: {{ services.jlite.image }}
    container_name: jlite_node_noe
    dns_search: .
    volumes: 
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - /home/jht/logs/jlite:/home/jht/logs
    environment:
      - TZ=Asia/Shanghai
    ports: {{ services.jlite.ports }}
    networks:
      - swarm_net
    deploy:
      mode: replicated
      replicas: {{ services.jlite.deploy.replicas }}
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      update_config:
        parallelism: 2
        delay: 5s
    depends_on:
      - redis
      - lucene
      - emqx
      - nginx
  redis:
    image: redis_docker_images_name
    container_name: redis_container
    environment:
      - TZ=Asia/Shanghai
    ports: {{ services.redis.ports }}
    networks:
      - swarm_net
    deploy:
      mode: replicated
      replicas: {{ services.redis.deploy.replicas }}
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
      update_config:
        parallelism: 2
        delay: 5s

6. 安装包

安装包由OA流程走完后，运维工具跑定时任务，定时打镜像到仓库中。

7. 日志文件

可以在docker swarm的集群服务器上共享挂载磁盘的方式，集中在一起。如下

services:
  jlite:
    volumes:
      - /home/docker/logs/jlite:/home/jht/logs

8. 自动化部署流程图

image.png

9. ansible playbook 环境安装配置

9.1 获取读取nacos配置文件 playbook脚本

---
- hosts: localhost
  remote_user: jht
  vars:
    - workspace: 93549ceb-522d-4137-b02d-7aa05ab10852
  tasks:
    - name: login nacos
      uri:
        url: http://10.10.204.114:8848/nacos/v1/auth/users/login?username=nacos&password=nacos
        method: post
        return_content: yes
        status_code: 200
      register: login
    - debug:
        var: login.json.accessToken
    - name: download config
      uri:
        url: http://10.10.204.114:8848/nacos/v1/cs/configs?show=all&dataId=serviceConfiguration&group=ansible&tenant={{ workspace }}&accessToken={{ login.json.accessToken }}&namespaceId={{ workspace }}
        method: get
        return_content: yes
        status_code: 200
      register: config
    - debug:
        var: config.json.content
    - name: write file
      file:
        path: /home/jht/playbook/templates
        state: directory
    - name: create file
      file:
        path: /home/jht/playbook/templates/serviceConfiguration.yaml
        state: touch
    - name: write file content
      blockinfile:
        path: /home/jht/playbook/templates/serviceConfiguration.yaml
        block: "{{ config.json.content }}"

9.2 安装docker

- name: install required packages
  yum:
    name: "{{ item }}"
    state: present
  with_items:
    - yum-utils
    - device-mapper-persistent-data
    - lvm2
- name: add docker repo to /etc/yum.repos.d
  shell: yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  register: result
- name: debug
  debug:
    var: not result.failed
- name: install docker-ce
  yum:
    name: docker-ce
    state: present
  when: not result.failed
- name: install docker-ce-cli
  yum:
    name: docker-ce-cli
    state: present
  when: not result.failed
- name: install containerd.io
  yum:
    name: containerd.io
    state: present
  when: not result.failed
- name: create /etc/docker
  file:
    path: "{{ docker_config_dir }}"
    state: directory
- name: start docker service
  systemd:
    name: docker
    state: started
    enabled: true
- name: provide docker-ce configfile
  template:
    src: daemon.json.j2
    dest: /etc/docker/daemon.json
  notify: restart docker

9.3 安装docker compose

- name: copy docker-compose-Linux-x86_64 to /usr/local/bin
  copy:
    src: ../files/docker-compose-linux-x86_64
    dest: "{{ docker_compose_dir }}/docker-compose-Linux-x86_64"

- name: install docker-compose
  shell: cd "{{ docker_compose_dir }}" && mv docker-compose-Linux-x86_64 docker-compose && chmod +x docker-compose

9.4 安装docker swarm leader

---
- name: Create leader node of docker swarm cluster
  hosts: leader
  remote_user: jht
  gather_facts: yes
  become: yes
  become_method: sudo
  become_user: root
  vars:
    ip_addr: "{{ ansible_default_ipv4['address'] }}"
  tasks:
    - name: show ip_add
      debug:
        var: ip_addr

    - name: Ensure docker is running
      become: yes
      service:
        name: docker
        state: started

    - name: Get docker info
      shell: docker info
      register: docker_info

    - name: show docker_info
      debug:
        var: docker_info

    - name: Create docker swarm leader node
      shell: docker swarm init --advertise-addr {{ ip_addr }}
      when: "docker_info.stdout.find('Swarm: inactive') != -1"

    - name: Get docker swarm manager token
      shell: docker swarm join-token -q manager
      register: manager_token

    - name: Get docker swarm worker token
      shell: docker swarm join-token -q worker
      register: worker_token

    - name: Output tokens
      debug:
        msg: "{{ manager_token.stdout }} ----------------- {{ worker_token.stdout }}"

    - name: Add tokens to dummy host, to be shared between multiple hosts
      add_host:
        name: swarm_tokens_host
        worker_token: "{{ worker_token.stdout }}"
        manager_token: "{{ manager_token.stdout }}"
        leader_ip_addr: "{{ ip_addr }}"

    - name: Show nodes of docker swarm
      shell: docker node ls

9.5 设置 docker swarm manager

---
- name: Create manager nodes of docker swarm cluster
  hosts: managers
  remote_user: jht
  become: true
  become_method: sudo
  become_user: root
  vars:
    manager_token: "SWMTKN-1-0xwaprlwrnc1ep0839lpuwgl2muq10vxu09forqgh7ojo4hx89-2fx5aapfq3uwq65slca5ph99r"
    leader_ip_addr: "10.10.205.115"
  tasks:
    - name: Ensure docker is running
      service:
        name: docker
        state: started

    - name: Get docker info
      shell: docker info
      register: docker_info

    - name: Join as manager node
      shell: docker swarm join --token {{ manager_token }} {{ leader_ip_addr }}:2377
      when: "docker_info.stdout.find('Swarm: inactive') != -1"
      retries: 3
      delay: 20
...

9.6 设置 docker swarm workers

---
- name: Create docker swarm worker nodes
  gather_facts: yes
  hosts: workers
  remote_user: jht
  become: yes
  become_method: sudo
  become_user: root
  vars:
    worker_token: "SWMTKN-1-0xwaprlwrnc1ep0839lpuwgl2muq10vxu09forqgh7ojo4hx89-5yps35hmuo6eda9cncl2vra39"
    leader_ip_addr: "10.10.205.115"
  tasks:
    - name: Ensure docker is running
      become: yes
      service:
        name: docker
        state: started

    - name: Get docker info
      shell: docker info
      register: docker_info

    - name: Join as worker node
      shell: docker swarm join --token {{ worker_token }} {{ leader_ip_addr }}:2377
      when: "docker_info.stdout.find('Swarm: inactive') != -1"
      retries: 3
      delay: 20
...

10. 部署实例

10.1 部署 nginx

• docker-compose.yaml

  version: '3.6'
  
  services:
    nginx:
      image: nginx:1.17.6-alpine
      ports:
        - "80:80"
      networks:
        - swarm_net
      deploy:
        mode: replicated
        replicas: 2
        restart_policy:
          condition: on-failure
          delay: 5s
          max_attempts: 3
          window: 120s
        update_config:
          parallelism: 2
          delay: 5s
  networks:
    swarm_net:
      driver: overlay
  

• deploy-nginx-service.yml

---
- name: Deploy services to docker swarm cluster
  gather_facts: no
  hosts: leader
  remote_user: jht
  become: true
  become_method: sudo
  become_user: root
  vars:
    src_docker_compose: ./docker-compose.yml
    dest_docker_compose: ~/docker-compose.yml
    stack_name: first_stack
  tasks:
    - name: Upload docker compose file
      template:
        src: "{{ src_docker_compose }}"
        dest: "{{ dest_docker_compose }}"

    - name: Deploy services
      shell: docker stack deploy --with-registry-auth --prune --compose-file {{ dest_docker_compose }} {{ stack_name }}

    - name: Pause for 10 seconds to wait for services being running
      pause:
        seconds: 10

    - name: Ensure services deployed
      shell: docker service ls
      register: service_output

    - name: Output service list
      debug:
        msg: "{{ service_output.stdout_lines }}"

    - name: Show stacks
      shell: docker stack ls
...