ARM架构构建Python Django项目镜像

ARM架构构建Python Django项目镜像

一、前期准备

• ARM服务器
• Docker容器
• Docker-compose 编排
• Miniconda3 Python3.9 基础环境

二、创建项目

声明：本地开发环境为M1芯片机器，所需包基本与ARM服务器一直，本篇文章主要介绍在拥有基本代码后，如何基于服务器重新安装所有包，也特意模仿在部署或者安装过程中所遇到的坑，以及处理方法。

基础环境这里就不详细搭建，只附上本文中环境版本：

1. 创建虚拟环境

# 创建项目所需要的环境以及包，并指定python版本
conda create -n huxing python=3.9.12

2. 查看现有虚拟环境

conda info --envs
# 或者
conda info list

3. 触发已创建的环境

conda activate huxing

4. cd 到 /root/miniconda3/envs/ 可以看到我们创建的虚拟环境（这个目录就是安装minicnda路径）

5. cd 到 /root/miniconda3/envs/huxing/bin 可以看到当前虚拟环境所有依赖包

三、安装所有依赖包

1. 安装Django（pip 源记得更换国内），Django版本最好不要夸大版本，改动比较多。

# 安装到指定目录，并设置镜像源，以及安装包版本
pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ django==2.2.5

2. 安装pands

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ pandas

3. 安装dxfgrabber

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ dxfgrabber

4. 安装pymysql

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ pymysql

5. 安装opencv

# 这里使用conda进行安装，pip arm架构太过麻烦。opencv 依赖包比较多，耐心等待
conda install -c https://conda.anaconda.org/menpo opencv

安装后：

6. 安装image

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ image

7. 安装oss2

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ oss2

8. 安装tqdm

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ tqdm

9. 安装scipy

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ scipy

10. 安装tensorflow

pip install --target=/root/miniconda3/envs/huxing/bin -i https://pypi.tuna.tsinghua.edu.cn/simple/ tensorflow

四、准备构建镜像

1. 下载minicoda3.tar.gz 官网网址

2. 创建目录

mkdir -p /data/ocr

3. 解压minicoda3

tar -xvf miniconda39.tar.gz

4. 压缩刚才创建的虚拟环境

cd /root/miniconda3/envs
tar -cvf huxing.tar.gz huxing/

5. 拷贝虚拟环境至/data/ocr/miniconda3

cp huxing.tar.gz /data/ocr/miniconda3/envs
cd /root/miniconda3/envs
tar -xvf huxing.tar.gz
rm -rf huxing.tar.gz

6. 创建Django配置文件

cd /data/ocr/miniconda3/etc
vim gunicorn.py

# config file of gunicorn for model web application

# django project dir
pythonpath = "/opt/huxing/"

# django settings
django_settings = "recognisehome.settings.production"

# the process name application, for top,ps command. need "pip install setproctitle" module 
proc_name = "huxing"

# listen address
bind = "0.0.0.0:8000"

# Run user and group of the application
#umask = 0022

# number of work process
workers = 4

# connections
worker_connections = 2048
backlog = 2048

# need "pip install gevent"
#worker_class = "gevent"
worker_class = "sync"

# The maximum number of requests a worker will process before restarting.
max_requests = 5000

# The number of seconds to wait for requests on a Keep-Alive connection
keepalive = 600

# Timeout for graceful workers restart.
# Generally set to thirty seconds. How max time worker can handle request after got restart signal. If the time is up worker will be force killed.
timeout = 300
gracefule_timeout = 5

# preload app code in master, please set False if set True, the reload will not real reload the django settings
preload_app = False

# Daemon able
daemon = False

# It's needed for decide the real REMOTE_ADDR
forwarded_allow_ips = "*"

# Log
accesslog = "/var/log/gunicorn/access.log"
access_log_format = '%(p)s %(h)s - %(t)s [%(D)s] "%(r)s" %(s)s %(b)s'
errorlog = "/var/log/gunicorn/error.log"
loglevel = "info"

import os
# By default the work process is set to be 0, even define umask = 0022 in config settings.
# So use the callback function "post_worker_init" to reset the umask, it works fine.
#def post_worker_init(worker):
#    os.umask(umask)

os.environ.setdefault('DJANGO_SETTINGS_MODULE', django_settings)

7. 压缩整个miniconda3

tar -cvf miniconda3.tar.gz miniconda3

8. 创建Dockerfile

# arm架构镜像
FROM centos:latest

# 将宿主机minicoda3 以及 虚拟环境进行压缩，并解压到容器的 /root目录
ADD miniconda3.tar.gz /root
# 将代码 解压到 容器 /opt下
ADD huxing.tar.gz /opt

ENV PATH="/root/miniconda3/bin:$PATH"
ENV DJANGO_SETTINGS_MODULE="recognisehome.settings.production"

RUN mkdir -p /var/log/django && mkdir -p /var/log/gunicorn
RUN conda init bash && source /root/.bashrc && conda activate huxing

CMD /root/miniconda3/envs/huxing/bin/gunicorn --preload -c /root/miniconda3/etc/gunicorn.py recognisehome.wsgi

# 在Debian 10中为vim启用鼠标复制粘贴
vim /etc/vim/vimrc.local

source $VIMRUNTIME/defaults.vim
let skip_defaults_vim = 1
if has('mouse')
    set mouse=r
endif

9. 创建项目配置文件，后期映射到容器，方便更改。

cd /data/ocr
mkdir conf
cd conf
vim production.py

# encoding:utf-8
# 线上配置

DEBUG = False

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'OPTIONS': {
            'init_command': 'SET storage_engine=INNODB',
            "init_command": "SET sql_mode='STRICT_TRANS_TABLES'",
        },
        'NAME': 'hx-recognize',                      # Or path to database file if using sqlite3.
        'USER': 'xxx',                      # Not used with sqlite3.
        'PASSWORD': 'xxx',              # Not used with sqlite3.
        'HOST': 'xxx',                 # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '3306',
    }
}

# 阿里云配置 
ALIYUN_ACCESS_KEY_ID = "xxx"
ALIYUN_ACCESS_KEY_SECRET = "xxx"
# OSS配置
# aliyun oss bucket name
OSS_MEIDA_BUCKET_NAME = 'xx'
OSS_STORE_BUCKET_NAME = 'xx'

OSS_BUCKET_ENDPOINT = 'https://xxx.com/'
DBJ_MEDIA_PATH = 'dbj/img/'
DBJ_STORE_PATH = 'dbj/store/'

# aliyun oss media url： 这里的{bucket-name}与上面保持一致就可以
MEDIA_URL = u'http://xxx.com/'


try:
    from .settings_local import *
except ImportError:
    pass

10. 创建docker-compose.yml

version: "3.0"
  
services:
  # django huxing
  huxing:
    build:
      context: .
    image: orc_huxing:alpha
    container_name: orc_huxing
    restart: always
    ports:
      - 8000:8000
    networks:
      - ibimfish
    volumes:
      - "./conf/production.py:/opt/huxing/recognisehome/settings/production.py"

networks:
  ibimfish:
    external: true

整个目录结构，miniconda39.tar.gz 是从官网下载的原始版本

五、构建运行镜像

1. 运行docker-compose

docker-compose up -d
# 需要拷贝的两个tar包比较大，耐心等待

六、启动报错之问题排查

1. gunicorn模块未安装

**问题分析：**Docker-compose运行后，通过docker ps 可看到容器启动失败了，查询日志发现没有这个文件，原来忘记忘记安装gunicorn模块了

**解决方式：**从新安装gunicorn模块并按照上面步骤进行压缩成tar包，并重新运行容器。（记得删除你失败的镜像）

#停止容器删除镜像
docker stop orc_huxing && docker rm orc_huxing && docker rmi orc_huxing:alpha
# 下载包
pip install --target=/root/miniconda3/envs/huxing/bin gunicorn
# 拷贝环境到制作镜像的目录
cp -r huxing/ /data/ocr/miniconda3/envs/
# 重新压缩
tar -cvf miniconda3.tar.gz miniconda3
# 重新运行
docker-compose up -d
docker logs -f orc_huxing

2. 启动成功，没有任何日志，调用接口报错500，也没有日志

**问题分析：**docker 搜集不到gunicorn日志

**解决方式：**进入容器内部查看gunicorn日志

docker exec -it orc_huxing bash

找到问题，缺少共享库，那就安装

yum install mesa-libGL -y

不过报错了，原来是yum没有设置源

设置源，并重新下载。

sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-*
sed -i -e "s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g" /etc/yum.repos.d/CentOS-*
yum install mesa-libGL -y

或者重新设置Dockerfile文件 一劳永逸

FROM centos:latest
  
ADD miniconda3.tar.gz /root
ADD huxing.tar.gz /opt

ENV PATH="/root/miniconda3/bin:$PATH"
ENV DJANGO_SETTINGS_MODULE="recognisehome.settings.production"

RUN sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-* &&\
    sed -i -e "s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g" /etc/yum.repos.d/CentOS-* &&\
    yum install mesa-libGL -y

RUN mkdir -p /var/log/django && mkdir -p /var/log/gunicorn
RUN conda init bash && source /root/.bashrc && conda activate huxing

CMD /root/miniconda3/envs/huxing/bin/gunicorn --preload -c /root/miniconda3/etc/gunicorn.py recognisehome.wsgi

最后重新运行，启动成功。

识别个图片试试：

速度还行。首次识别没有预加载模型，耗时过多，后期再进行优化。