天池热身赛-布匹瑕疵目标检测
1、检测代码
代码源自datawhale官方提供baseline: https://github.com/datawhalechina/team-learning-cv/tree/master/DefectDetection
数据处理代码都写好了，太感动了 _
baseline使用的是yolov5，我的显卡只有一个1080Ti，所以先选择yolov5s进行训练，设置训练50个epoch， 图片大小设置为512x512。
这部分内容主要参考了https://blog.csdn.net/qq_26751117/article/details/113853150
- 数据处理：主要是将比赛方提供的数据格式转化为yolo需要的格式，先使用convertTrainLabel.py转化，然后在运行process_data_yolo.py，就得到了数据，存放位置为process_data文件夹；注意需要修改process_data_yolo中val字段，全部改为train字段，运行两次，分别得到验证和训练的数据文件。
- 预训练权重：尝试了一下不加载预训练权重，效果不是很好，可能是因为本来数据就比较少，还是需要进行迁移学习的。所以想办法下载了yolov5s.pt文件，进行了加载。由于模型比较小，可以设置较大的batch size, 这里是16。这里借上边那个文章的图，需要简单修改一下加载权重的部分。
- 运行，简单修改了一下一些报错的点，然后就可以运行了yolov5s模型了。
2、docker提交
我的dockerfile文件：
# Base Images
FROM registry.cn-shanghai.aliyuncs.com/tcc-public/pytorch:1.4-cuda10.1-py3
ADD . /workspace
WORKDIR /workspace
RUN pip install -r requirements.txt
CMD ["sh", "run.sh"]
开始构建：
(torch16) pdluser@pdluser-System-Product-Name:~/project/tianchi_demo$ sudo docker build -t registry.cn-shenzhen.aliyuncs.com/nine_percent/tianchi_submit:1.0 .
[sudo] pdluser 的密码：
Sending build context to Docker daemon 6.778GB
Step 1/5 : FROM registry.cn-shanghai.aliyuncs.com/tcc-public/pytorch:1.4-cuda10.1-py3
---> 76c152fbfd03
Step 2/5 : ADD . /workspace
---> 10ca596f6d20
Step 3/5 : WORKDIR /workspace
---> Running in 37a88d04d2a9
Removing intermediate container 37a88d04d2a9
---> 7f7982fbfaba
Step 4/5 : RUN pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple --ignore-installed PyYAML
---> Running in 877004f83473
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Downloading https://mirrors.aliyun.com/pypi/packages/ec/d6/a82d191ec058314b2b7cbee5635150f754ba1c6ffc05387bc9a57efe48b8/cryptacular-1.5.5.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting zope.sqlalchemy
Downloading https://mirrors.aliyun.com/pypi/packages/fa/83/459decec1dd2c14d60f9a360fff989c128abe545a1554a1da64b054a55d4/zope.sqlalchemy-1.3-py2.py3-none-any.whl
Collecting velruse>=1.0.3
Downloading https://mirrors.aliyun.com/pypi/packages/8f/0b/d47ea894587f3155f8c4520aa74d57c856189d0bbe27e831881d655a3386/PasteDeploy-2.1.1-py2.py3-none-any.whl
Building wheels for collected packages: cryptacular
Building wheel for cryptacular (PEP 517): started
Building wheel for cryptacular (PEP 517): finished with status 'done'
Created wheel for cryptacular: filename=cryptacular-1.5.5-cp37-abi3-manylinux2010_x86_64.whl size=52452 sha256=93037b68313c3d86df4c8cab9d0cc0866d1579cb7399410c7903b56eb2ff0067
Stored in directory: /root/.cache/pip/wheels/dd/c7/11/721f100da8477396b1f8fcfa2d23c801d5bac07d0e2d82dc0d
Successfully built cryptacular
Building wheels for collected packages: apex, velruse, pbkdf2, anykeystore
Building wheel for apex (setup.py): started
Building wheel for apex (setup.py): finished with status 'done'
Created wheel for apex: filename=apex-0.9.10.dev0-cp37-none-any.whl size=46468 sha256=c68745de219dd6169195cfec426e528cd5f5f932bd3cb7ddbc22817a9827cfea
Stored in directory: /root/.cache/pip/wheels/b8/f0/7a/2fc4cf8a70bfc0981f7009a2146685d06ee220398c0b780acf
Building wheel for velruse (setup.py): started
Building wheel for velruse (setup.py): finished with status 'done'
Created wheel for velruse: filename=velruse-1.1.1-cp37-none-any.whl size=50923 sha256=c300b70b745467b6b075bec09d6b2a11ab3524f6de31605431a62308613648e3
Stored in directory:
Successfully built apex velruse pbkdf2 anykeystore
Installing collected packages: PyYAML, Cython, numpy, opencv-python, typing-extensions, torch, pyparsing, kiwisolver, six, cycler, pillow
Removing intermediate container 877004f83473
---> 5c40d92c4bc1
Step 5/5 : CMD ["sh", "run.sh"]
---> Running in 41c2daf77fbc
Removing intermediate container 41c2daf77fbc
---> 603e3fe4452c
Successfully built 603e3fe4452c
Successfully tagged registry.cn-shenzhen.aliyuncs.com/nine_percent/tianchi_submit:1.0
在构建完镜像以后，进入镜像：
先查看一下对应的ID:
pdluser@pdluser-System-Product-Name:~$ sudo docker images
[sudo] pdluser 的密码：
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.cn-shenzhen.aliyuncs.com/nine_percent/tianchi_submit 1.0 b773b4e52e7a 4 minutes ago 11.2GB
f99df53cc33c 23 hours ago 7.92GB
registry.cn-shanghai.aliyuncs.com/tcc-public/pytorch 1.4-cuda10.1-py3 76c152fbfd03 13 months ago 7.56GB
registry.cn-shanghai.aliyuncs.com/tcc-public/python 3 a4cc999cf2aa 21 months ago 929MB
进入第一个镜像，b7:
(torch16) pdluser@pdluser-System-Product-Name:~/project/tianchi_demo$ sudo docker run -it b7 /bin/bash
root@2a128d20af63:/workspace#
在这里运行run.sh，测试成功就可以提交了。
下一步将镜像推送到Registry:
$ sudo docker login --username=用户名 registry.cn-shenzhen.aliyuncs.com
$ sudo docker tag [ImageId] registry.cn-shenzhen.aliyuncs.com/nine_percent/tianchi_submit:[镜像版本号]
$ sudo docker push registry.cn-shenzhen.aliyuncs.com/nine_percent/tianchi_submit:[镜像版本号]
3、遇到的问题
在进行build的时候，发现以下问题，ERROR: Double requirement given: PyYAML>=5.3 (from -r requirements.txt (line 10)) (already in PyYAML, name=‘PyYAML’)
通过把yaml的等级要求去掉，就不会报错了。
用到opencv的时候也出现了报错：
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.7/site-packages/cv2/__init__.py", line 5, in
from .cv2 import *
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
解决方案是在dockerfile中添加以下内容：
RUN apt update
RUN apt install libgl1-mesa-glx
RUN apt-get install -y libglib2.0-0
但是会遇到以下问题：
这样改动dockfile，避免交互：
RUN DEBIAN_FRONTEND=noninteractive apt update -y
RUN DEBIAN_FRONTEND=noninteractive apt install libgl1-mesa-glx -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y libglib2.0-0
第一次提交出错：
啊，连续两次错误了。
之后调节了一下文件存放位置，和对应的命令，终于提交成功了，可喜可贺可喜可贺。
吐槽：这个docker虽然很不错，但是入门还是有一定门槛的，我总结了一下使用过程中经常用到的知识点：https://blog.csdn.net/DD_PP_JJ/article/details/113902874 可以参考一下。 整个过程docker卡壳时间比较久，查看了一下群友推荐的资料，感觉还是对docker了解比较局限，每次build的时候都需要从远端下载镜像，非常麻烦，每次build要花很久很久，遇到了许多dockerfile的相关的问题，每次处理都需要重新下载，感觉很麻烦。提交比赛结果也是非常漫长，push要花很久时间，好麻烦。另外，不知道为何我再运行docker -v的时候发现映射不到容器中，这个问题还没有解决。