标注环境CVAT在Ubuntu18.04下搭建

CVAT Ubuntu 18.04 标注环境搭建

记录一次在Ubuntu 18.04下搭建CVAT 环境的过程，将会在本月内再次搭建一次

环境准备

1. Ubuntu环境搭建
2. Anaconda 下载
3. CVAT 下载
4. 换源路径

Ubuntu环境搭建

传送门链接: link.
U盘启动制作工具: link.

换源路径

推荐Ubuntu清华源 link.

# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse

# 预发布软件源，不建议启用
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse

推荐Anaconda清华源 link.

conda config --set show_channel_urls yes
nano ~/.condarc

# 在.condarc中贴入如下源信息

channels:
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

# 在.condarc追加如下超时时间

remote_connect_timeout_secs: 40.0
remote_read_timeout_secs: 100.0

Anaconda 下载安装

下载对应的版本 ，路径来源（清华源）
Anaconda3-2019.10-Linux-x86_64.sh .
Anaconda3-2019.10-Windows-x86_64.exe .

CVAT 下载

git clone https://github.com/opencv/cvat

基础环境配置

Ubuntu Docker 环境配置

# 摘自 https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md

sudo apt-get update
sudo apt-get install -y \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg-agent \
  software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io


# docker进程使用Unix Socket而不是TCP端口。
# 而默认情况下，Unix socket属于root用户，需要root权限才能访问。
# add user in docker group
sudo groupadd docker     #添加docker用户组
sudo gpasswd -a $USER docker     #将登陆用户加入到docker用户组中
newgrp docker     #更新用户组
docker ps    #测试docker命令是否可以使用sudo正常使用

# install docker-compose
pip install docker-compose -i https://pypi.tuna.tsinghua.edu.cn/simple

★ CVAT Dockerfile修改

原生的Dockerfile会自动去官方源去做更新，因为国内网络问题，所以要换源
步骤如下

1. 在CVAT目录下，放入source.list
   
   source内容是清华源，对应Ubuntu版本

2. 修改如下位置的Dockerfile
   
   

3. 变动后结果如下

FROM ubuntu:16.04

ARG http_proxy
ARG https_proxy
ARG no_proxy
ARG socks_proxy

ENV TERM=xterm \
    http_proxy=${http_proxy}   \
    https_proxy=${https_proxy} \
    no_proxy=${no_proxy} \
    socks_proxy=${socks_proxy}

ENV LANG='C.UTF-8'  \
    LC_ALL='C.UTF-8'

ARG USER
ARG DJANGO_CONFIGURATION
ENV DJANGO_CONFIGURATION=${DJANGO_CONFIGURATION}

# Install necessary apt packages
RUN apt-get update && \
    apt-get install -yq \
        python-software-properties \
        software-properties-common \
        wget && \
    add-apt-repository ppa:mc3man/xerus-media -y && \
    add-apt-repository ppa:mc3man/gstffmpeg-keep -y && \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq \
        apache2 \
        apache2-dev \
        libapache2-mod-xsendfile \
        supervisor \
        ffmpeg \
        gstreamer0.10-ffmpeg \
        libldap2-dev \
        libsasl2-dev \
        python3-dev \
        python3-pip \
        unzip \
        unrar \
        p7zip-full \
        vim && \
    add-apt-repository --remove ppa:mc3man/gstffmpeg-keep -y && \
    add-apt-repository --remove ppa:mc3man/xerus-media -y && \
    rm -rf /var/lib/apt/lists/*

# Add a non-root user
ENV USER=${USER}
ENV HOME /home/${USER}
WORKDIR ${HOME}
RUN adduser --shell /bin/bash --disabled-password --gecos "" ${USER}

COPY components /tmp/components

# OpenVINO toolkit support
ARG OPENVINO_TOOLKIT
ENV OPENVINO_TOOLKIT=${OPENVINO_TOOLKIT}
RUN if [ "$OPENVINO_TOOLKIT" = "yes" ]; then \
        /tmp/components/openvino/install.sh; \
    fi

# CUDA support
ARG CUDA_SUPPORT
ENV CUDA_SUPPORT=${CUDA_SUPPORT}
RUN if [ "$CUDA_SUPPORT" = "yes" ]; then \
        /tmp/components/cuda/install.sh; \
    fi

# Tensorflow annotation support
ARG TF_ANNOTATION
ENV TF_ANNOTATION=${TF_ANNOTATION}
ENV TF_ANNOTATION_MODEL_PATH=${HOME}/rcnn/inference_graph
RUN if [ "$TF_ANNOTATION" = "yes" ]; then \
        bash -i /tmp/components/tf_annotation/install.sh; \
    fi

ARG WITH_TESTS
RUN if [ "$WITH_TESTS" = "yes" ]; then \
        wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
        echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main' | tee /etc/apt/sources.list.d/google-chrome.list && \
        wget -qO- https://deb.nodesource.com/setup_9.x | bash - && \
        apt-get update && \
        DEBIAN_FRONTEND=noninteractive apt-get install -yq \
            google-chrome-stable \
            nodejs && \
        rm -rf /var/lib/apt/lists/*; \
        mkdir tests && cd tests && npm install \
            eslint \
            eslint-detailed-reporter \
            karma \
            karma-chrome-launcher \
            karma-coveralls \
            karma-coverage \
            karma-junit-reporter \
            karma-qunit \
            qunit; \
        echo "export PATH=~/tests/node_modules/.bin:${PATH}" >> ~/.bashrc; \
    fi

# Install and initialize CVAT, copy all necessary files
COPY cvat/requirements/ /tmp/requirements/
COPY supervisord.conf mod_wsgi.conf wait-for-it.sh manage.py ${HOME}/
RUN pip3 install --no-cache-dir -r /tmp/requirements/${DJANGO_CONFIGURATION}.txt  -i https://pypi.tuna.tsinghua.edu.cn/simple

# Install git application dependencies
RUN apt-get update
RUN apt-get install apt-transport-https
ADD sources.list /etc/apt/
RUN apt-get update && \
    apt-get install -y ssh netcat-openbsd git curl zip  && \
#    wget -qO /dev/stdout https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
    apt-get install -y git-lfs && \
    git lfs install && \
    rm -rf /var/lib/apt/lists/* && \
    if [ -z ${socks_proxy} ]; then \
        echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30\"" >> ${HOME}/.bashrc; \
    else \
        echo export "GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no -o ConnectTimeout=30 -o ProxyCommand='nc -X 5 -x ${socks_proxy} %h %p'\"" >> ${HOME}/.bashrc; \
    fi

# Download model for re-identification app
ENV REID_MODEL_DIR=${HOME}/reid
RUN if [ "$OPENVINO_TOOLKIT" = "yes" ]; then \
        mkdir ${HOME}/reid && \
        wget https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/person-reidentification-retail-0079/FP32/person-reidentification-retail-0079.xml -O reid/reid.xml && \
        wget https://download.01.org/openvinotoolkit/2018_R5/open_model_zoo/person-reidentification-retail-0079/FP32/person-reidentification-retail-0079.bin -O reid/reid.bin; \
    fi

# TODO: CHANGE URL
ARG WITH_DEXTR
ENV WITH_DEXTR=${WITH_DEXTR}
ENV DEXTR_MODEL_DIR=${HOME}/dextr
RUN if [ "$WITH_DEXTR" = "yes" ]; then \
        mkdir ${DEXTR_MODEL_DIR} -p && \
        wget https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip -O ${DEXTR_MODEL_DIR}/dextr.zip && \
        unzip ${DEXTR_MODEL_DIR}/dextr.zip -d ${DEXTR_MODEL_DIR} && rm ${DEXTR_MODEL_DIR}/dextr.zip; \
    fi

COPY ssh ${HOME}/.ssh
COPY cvat/ ${HOME}/cvat
COPY cvatjs/ ${HOME}/cvatjs
COPY tests ${HOME}/tests
# Binary option is necessary to correctly apply the patch on Windows platform.
# https://unix.stackexchange.com/questions/239364/how-to-fix-hunk-1-failed-at-1-different-line-endings-message
RUN patch --binary -p1 < ${HOME}/cvat/apps/engine/static/engine/js/3rdparty.patch
RUN chown -R ${USER}:${USER} .

# RUN all commands below as 'django' user
USER ${USER}

RUN mkdir data share media keys logs /tmp/supervisord
RUN python3 manage.py collectstatic

EXPOSE 8080 8443
ENTRYPOINT ["/usr/bin/supervisord"]

★ CVAT docker-compose.yml修改

1. 变动如下

# cvat ports 下增加
    ports:
      - "0.0.0.0:9000:8080"
# 使外网可以访问 例如192.168.1.100:9000

# cvat environment 下增加
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: "*"
# 使外网可以访问

2. 变动后结果如下

#
# Copyright (C) 2018 Intel Corporation
#
# SPDX-License-Identifier: MIT
#
version: "2.3"

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:10.3-alpine
    networks:
      default:
        aliases:
          - db
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
    volumes:
      - cvat_db:/var/lib/postgresql/data

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0.5-alpine
    networks:
      default:
        aliases:
          - redis
    restart: always

  cvat:
    container_name: cvat
    image: cvat
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    ports:
      - "0.0.0.0:9000:8080"
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
        TF_ANNOTATION: "no"
        USER: "django"
        DJANGO_CONFIGURATION: "production"
        WITH_TESTS: "no"
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: "*"
    volumes:
      - /cvat/data:/home/ai/cvatdata/data
      - /cvat/keys:/home/ai/cvatdata/keys
      - /cvat/logs:/home/ai/cvatdata/logs
      - /cvat/Users:/home/ai/cvatdata/models

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_models:

耐心等待与使用

docker-compose build

安装过程中，不可避免有些遗漏的地方也是用官方源去下载的，不过数量较少，所以也没有完全去把所有的源都换了。
在网络限制在400kb/s时，docker-compose build一次大概45min

可能遇到的问题

1. 创建超级用户失败

docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'

错误信息是：LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "auth_user...

处理办法

# 1. 进入到docker镜像中
docker exec -it cvat /bin/bash
# 2. python3 ~/manage.py createsuperuser
# 再运行一次，看看具体的错误信息，一般情况下，是要进行一次migrate
python3 ~/manage.py migrate
# 3. 然后再次运行创建超级用户的命令
python3 ~/manage.py createsuperuser

2. 上述操作之后，依旧没法访问设定的9000端口

处理办法1

# 1. 还是进入到docker镜像中
docker exec -it cvat /bin/bash
# 2. 启动django
python3 manage.py runserver 0.0.0.0:8080
然后按住ctrl + p + q 退出当前docker exec 环境

3. 20191210 增加因PC突然断电，然后无法启动的处理办法

# 不要用 -d
docker-compose up
# 此时注意看打印的日志，可能会多如下日志
cvat          | 2019-12-10 10:23:33,918 DEBG 'runserver' stderr output:
cvat          | [Tue Dec 10 10:23:33.918640 2019] [core:warn] [pid 18:tid 140294961167296] AH00098: pid file /tmp/mod_wsgi-localhost:8080:1000/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
# 怎么办？
# 0 照常启动 不后台
docker-compose up
# 1 还是进入到docker里
docker exec -it cvat /bin/bash
# 2 删掉或备份 mod_wsgi_localhost:8080:1000
cd /tmp
rm -rf mod_wsgi-localhost\:8080\:1000/
# 3 停止docker-compose
ctrl c
# 4 重启
docker-compose up -d

4. 启动后页面展示不全，日志展示

[2020-05-27 12:53:59,190] WARNING django.request: Not Found: /static/CACHE/js/239c24a082e8.js
[2020-05-27 12:53:59,195] WARNING django.request: Not Found: /static/CACHE/js/d91026595cdb.js
[2020-05-27 12:53:59,197] WARNING django.server: "GET /static/CACHE/js/239c24a082e8.js HTTP/1.1" 404 77
[2020-05-27 12:53:59,197] WARNING django.server: "GET /static/CACHE/js/d91026595cdb.js HTTP/1.1" 404 77

原因未知。。。
解决办法：
回到docker-compose.yml所在的目录，docker-compose restart

创建超级用户

方法有两种

1. 直接用bash

docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'

2. 进入到镜像内运行

# 1. 进入到docker镜像中
docker exec -it cvat /bin/bash
# 2. 运行命令
python3 ~/manage.py createsuperuser

创建普通用户

1. 打开admin网页

http://192.168.1.100:9000/admin/

可以探索的使用方法

1. 与git集成
2. 自动创建标注任务
3. 自动标注

！对了！在chrome下使用这个CVAT