Nginx 、MySQL、Django 在 Docker-compose 中的部署

概述

本文主要记录了在 Linux 3.10.0-1062.el7.x86_64 下使用 docker-compose 搭建服务的过程，由于公司服务器在内网中，搭建镜像及下载依赖时需要外部代理，如果本机服务无相关限制，可联通外网，配置代理部分忽略即可，其实 firewall 为开启，selinux 为 enforcing 状态。

部署中主要涉及以下内容：

1. Nginx 代理 MySQL 和 django 服务以及 https 自签名的部署。
2. 使用 gunicorn 部署 django 服务。
3. MySQL 相关初始化配置，及建权过程。
4. 将 .py 文件转为 .pyc 文件

Docker 安装

如果之前安装过的话，需要删除老版本 docker:

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

安装构建 docker 所需要的工具;

 sudo yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

添加 docker 源:

sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

安装 docker ce:

sudo yum install docker-ce docker-ce-cli containerd.io

配置 docker:

sudo systemctl start docker
sudo systemctl enable docker

安装 docker-compose:

# download
sudo curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
# apply exe permissions
sudo chmod +x /usr/local/bin/docker-compose
# add link
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
# check 
docker-compose --version

如果在安装中下无法下载相应服务，则表明可能需要配置相应代理或开放相关 ACL 配置，关于相关配置过程可参见Centos7 下代理配置。

docker 的代理配置也需要单独做出来，可以参见docker 代理脱坑这篇。

项目结构

下面是 docker-compose 工程构建后完整目录：

├── config
│   ├── auto_start_script.sh
│   ├── mysql
│   │   ├── init
│   │   │   └── init.sql
│   │   ├── my.cnf
│   │   └── my.conf
│   ├── nginx
│   │   ├── conf.d
│   │   │   ├── certs
│   │   │   │   ├── cmi_sdn.crt
│   │   │   │   ├── cmi_sdn.key
│   │   │   │   └── dhparam.pem
│   │   │   ├── ssl.conf
│   │   │   └── ssl.conf.bk
│   │   ├── default.d
│   │   │   └── ssl-redirect.conf
│   │   ├── log
│   │   │   ├── access.log
│   │   │   └── error.log
│   │   └── nginx.conf
│   └── requirements.txt
├── docker-compose.yml
├── Dockerfile
├── db_storage # 创建文件夹就好，用于挂载 mysql 数据
└── project
    ├── your_project
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── manage.py
    └── static
    └── frontend

tree -I '*project*|*frontend*|*db_storage*'

• config 目录：存放 nginx 配置, requirement, mysql 配置
• docker-compose.yml: 用于编排和管理容器的 yaml 文件
• Dockerfile: 用于编写 django 镜像的文件
• project：django 项目代码

Dockerfile 编写

编写 django 的运行环境，如不需要代理可将环境变量删除

FROM python:3.6.8

ENV MY_PROXY_URL="http://173.39.112.117:80"
ENV HTTP_PROXY=$MY_PROXY_URL \
    HTTPS_PROXY=$MY_PROXY_URL \
    FTP_PROXY=$MY_PROXY_URL \
    http_proxy=$MY_PROXY_URL \
    https_proxy=$MY_PROXY_URL \
    ftp_proxy=$MY_PROXY_URL

WORKDIR /src

COPY /config/requirements.txt /

RUN pip install --no-cache-dir  -r /requirements.txt

ENV MY_PROXY_URL=
ENV HTTP_PROXY=$MY_PROXY_URL \
    HTTPS_PROXY=$MY_PROXY_URL \
    FTP_PROXY=$MY_PROXY_URL \
    http_proxy=$MY_PROXY_URL \
    https_proxy=$MY_PROXY_URL \
    ftp_proxy=$MY_PROXY_URL

创建 config/requirements.txt 文件，并添加所需要的 python 依赖包，部署选用 gunicorn. 文件如下：

Django==2.1.7
djangorestframework==3.9.2
djangorestframework-jwt==1.11.0
mysqlclient==1.4.2
PyJWT==1.7.1
requests==2.21.0
gunicorn==19.9.0
django-role-permissions==2.2.1
aiohttp==3.5.4
apscheduler==3.6.3
networkx==2.4
numpy==1.17.4

Nginx 配置

在 nginx/conf.d/certs 目录下，使用 openssl 进行自签名，生成需要的证书：

sudo openssl req   -x509   -nodes   -days 365   -newkey rsa:2048   -keyout example.key   -out example.crt;

sudo openssl dhparam -out dhparam.pem 2048

编写 nginx 配置文件 config/nginx/conf.d/ssl.conf：

upstream web_server {
  # ip_hash;
  server web:8000  fail_timeout=100s;
  #server 10.89.200.40:8000 backup;
  keepalive 300;
}

upstream topo {
  # ip_hash;
  server topo:7777  fail_timeout=100s;
  #server 10.89.200.40:8000 backup;
  keepalive 300;
}

server {
    listen 443 http2 ssl;
    listen [::]:443 http2 ssl;

    server_name server_IP_address;

    ssl_certificate /etc/nginx/conf.d/certs/cmi_sdn.crt;
    ssl_certificate_key /etc/nginx/conf.d/certs/cmi_sdn.key;
    ssl_dhparam /etc/nginx/conf.d/certs/dhparam.pem;
    ########################################################################
    # from https://cipherli.st/                                            #
    # and https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html #
    ########################################################################

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    # Disable preloading HSTS for now.  You can use the commented out header line that includes
    # the "preload" directive if you understand the implications.
    #add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    ##################################
    # END https://cipherli.st/ BLOCK #
    ##################################

    location / {
            proxy_set_header Host $host:$server_port;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Real-PORT $remote_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://web_server/;
            proxy_connect_timeout 159s;
            proxy_send_timeout   600;
            proxy_read_timeout   600;
            keepalive_timeout 600s;
  }


  listen 8000;
  server_name localhost;

  location /static {
    autoindex on;
    alias /src/static;
  }
}


server {
    listen 666 http2 ssl;
    listen [::]:666 http2 ssl;

    server_name server_IP_address;

    ssl_certificate /etc/nginx/conf.d/certs/cmi_sdn.crt;
    ssl_certificate_key /etc/nginx/conf.d/certs/cmi_sdn.key;
    ssl_dhparam /etc/nginx/conf.d/certs/dhparam.pem;
    ########################################################################
    # from https://cipherli.st/                                            #
    # and https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html #
    ########################################################################

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    # Disable preloading HSTS for now.  You can use the commented out header line that includes
    # the "preload" directive if you understand the implications.
    #add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
    #add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    ##################################
    # END https://cipherli.st/ BLOCK #
    ##################################

    add_header X-Frame-Options sameorigin always;

    location = / {
        root   /src/frontend/dist;
        index  index.html index.htm;
    }


    location ^~ /index.html {
        root   /src/frontend/dist;
        index  index.html index.htm;

    }

    location ^~ /favicon.png {
        root /src/frontend/dist;
    }

    location ^~ /static {
        alias /src/frontend/dist/static;
    }

    location ^~ /topo/get_topology/ {
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Real-PORT $remote_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://topo;
   }

   location ^~ /topo/update_topology/ {
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Real-PORT $remote_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://topo;
   }

    location ~* /.*  {
        proxy_set_header Host $host:$server_port;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Real-PORT $remote_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://web_server;
        proxy_connect_timeout 159s;
        proxy_send_timeout   600;
        proxy_read_timeout   600;
        keepalive_timeout 600s;
    }

  server_name localhost;
}

注意这里的 web 是 docker-compose 编排中的 django 所在容器名称。

topo 是我这里启用的另一服务，忽略即可。

编写 default.d/ssl-redirect.conf 添加 http 到 https 的自动跳转

return 301 https://$host$request_uri/;

编写 config/nginx/nginx.conf 配置入口文件

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

stream {
    # log_format  main  'mysql - $remote_addr - [$time_local]  ';
    # access_log  /var/log/nginx/access.log  main;

    upstream mysql {
       # hash $remote_addr consistent;
       server 10.124.207.155:3306;
       server 10.124.207.154:3306 backup;
    }
    server {
       listen 3306;#公网机器监听端口
       proxy_connect_timeout 10s;
       proxy_timeout 800s;
       proxy_pass mysql;
    }
}

mysql 为 docker-compose 文件下的名称

创建 config/nginx/log/access.log or error.log

MySQL 配置

编写 config/mysql/my.conf

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

#server-id=1
log_bin                 = mysql-bin   # 开启二进制log文件
binlog_format           = mixed
#relay-log               = relay-bin
#relay-log-index         = slave-relay-bin.index
#auto-increment-offset   = 1     # 从 1 开始
#auto-increment-increment = 2    # 每次增长 2

character-set-server=utf8

[client]
default-character-set=utf8

[mysql]
default-character-set=utf8

[mysqld]
skip-name-resolve

编写初始化 config/mysql/init/init.sql

CREATE USER 'test'@'%';

GRANT ALL PRIVILEGES ON *.* To 'test'@'%' IDENTIFIED BY 'testPass1!';

FLUSH PRIVILEGES;

CREATE DATABASE IF NOT EXISTS test default charset utf8 COLLATE utf8_general_ci;

编写 docker-compose 文件：

version: '3'

services:
   mysql:
        image: ctg/mysql:5.7.29
        container_name: "mysql"
        environment:
            MYSQL_ROOT_PASSWORD: "MyNewPass1!"
        restart: always
        ports:
            - "3306:3306"
        volumes:
            - "/etc/localtime:/etc/localtime:ro"
            - "./db_storage:/var/lib/mysql"
            - "./config/mysql/my.cnf:/etc/my.cnf"
            - "./config/mysql/init:/docker-entrypoint-initdb.d/"
        networks:
            - app_net

   proxy_server:
        image: ctg/nginx:latest
        ports:
            - "443:443"
            - "3307:3307"
            - "444:444"
            - "666:666"
        volumes:
            - "/etc/localtime:/etc/localtime:ro"
            - "./project:/src"
            - "./config/nginx/conf.d:/etc/nginx/conf.d"
            - "./config/nginx/default.d:/etc/nginx/default.d"
            - "./config/nginx/log:/var/log/nginx"
            - "./config/nginx/nginx.conf:/etc/nginx/nginx.conf"
            - "./frontend:/var/frontend"
        container_name: "proxy_server"
        depends_on:
            - web
        networks:
            - app_net
   web:
        image: ctg/web:latest
        container_name: "ctg_backend"
        command: bash -c "python manage.py makemigrations && python manage.py migrate && python manage.py collectstatic --noinput && gunicorn -c gunicorn_config.py ctg.wsgi"

        ports:
            - "8000:8000"
        volumes:
            - "/etc/localtime:/etc/localtime:ro"
            - "./project:/src"
            - "/proc:/hostip/:ro"
        restart: always
        privileged: true
        networks:
            - app_net

   topo:
       image: ctg/topo:latest
       container_name: "ctg_topo"
       command: bash -c "python /ctg_topo/manage.py runserver 0.0.0.0:7777"

       ports:
           - "7777:7777"
       volumes:
           - "/etc/localtime:/etc/localtime:ro"
           - "./topo_project/topo_server:/ctg_topo"
       restart: always
       networks:
            - app_net

networks:
    app_net:
        driver: bridge
        ipam:
            config:
                - subnet: 172.26.0.1/16

由于 nginx 和 django 是分开部署的，记得把 django 层收集的静态文件也给 nginx 拷贝一份，否则会导致 admin 管理界面样式丢失。我这里的 frontend 是又另外的 portal 界面，忽略即可

想着修改 django 项目的 settings.py 文件，添加如下内容：

ALLOW_HOSTS = ['web']
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')

运行

构建镜像：

docker-compose build

如果上述构建成功的话，就可以运行容器了：

docker-compose up -d

如果 docker-compose 的容器是以守护进程运行的话，可以使用下面的命令查看容器的日志：

docker-compose logs -f web

配置开机自启动

# 配置 docker 自启动
systemctl enable docker

配置docker-compose自启动可参考 自启动. 编写 config/auto_start_script.sh 文件。

将 python 转为 pyc 文件

在 django manage.py 同级别执行

# 生成 pyc 文件: 
python -m compileall -b .

# 删除 py 文件: 
find . -name "*.py" |xargs rm -rf

# 删除 pycache 目录:
find . -name "pycache" |xargs rm -rf

# 最后将 docker-compose.yml 文件中涉及到的 py 文件 换成 pyc. 
需要注意的是 gunicorn_config.py 要备份一份，该文件不要转成 pyc.

如果想要实现通过 Jenkins 自动编译加部署，可以参考Jenkins 实现 CI/CD.

参考

how-to-create-a-self-signed-ssl-certificate-for-nginx-on-centos-7

docker ce install