django api_使用Django API Deeplabv3进行语义分割

django api

介绍 (Introduction)

Image segmentation has been a hot topic for a while now. Various uses cases involving segmentation had emerged in a bunch of different areas, machine vision, medical imaging, object detection, recognition tasks, traffic control systems, video surveillance and, a lot more. The intuition behind these intelligent systems is to capture the diverse components that form the image and therefore, teach computer vision models to grasp more insight and a better understanding of the scene and the context.

图像分割一直是一个热门话题。 涉及细分的各种用例已经出现在许多不同的领域中，包括机器视觉，医学成像，对象检测，识别任务，交通控制系统，视频监控等等。 这些智能系统的直觉是捕获形成图像的各种组件，因此，教计算机视觉模型以掌握更多的见解以及对场景和上下文的更好理解。

Melody Jacob on Melody Jacob的原始照片，左为 Unsplash on the left, segmented version on the right Unsplash ，右为分段图

The two types of image segmentation commonly used are:

常用的两种图像分割类型是：

• Semantic Segmentation: identifying different classes in the image and segment accordingly

  语义分割 ：识别图像中的不同类别并相应地进行分割

• Instance Segmentation: determine priorly the different classes in the image and recognize the number of instances each class contains. The image is decomposed in multiple labeled regions relating to the different class instances the model was trained on.

  实例分割 ：事先确定图像中的不同类，并识别每个类包含的实例数。 图像在与模型训练所依据的不同类实例相关的多个标记区域中分解。

For this article, I will use the Pytorch implementation of the Google DeepLab V3 segmentation model to customize the background of an image. The intention is to segment the foreground and detach it from the rest while replacing the remaining background with a whole different picture. The model will be served through a Django REST API.

对于本文，我将使用Google DeepLab V3细分模型的Pytorch实现来自定义图像的背景。 目的是分割前景并将其与其余部分分离，同时用另一张图片替换其余背景。 该模型将通过Django REST API提供服务。

总览 (Overview)

1. A bit of background on DeepLab V3
   DeepLab V3的背景知识
2. Use the DeepLab V3-Resnet101 implementation from Pytorch
   使用Pytorch的DeepLab V3-Resnet101实现
3. Set up the Django API
   设置Django API
4. Conclusion
   结论

You can check the entire code for this project under my Gihut repo.

您可以在我的Gihut存储库下检查该项目的完整代码。

1. DeepLab V3的一些背景知识 (1. A bit of background on DeepLab V3)

Segmentation models use fully convolutional neural networks FCNN during a prior image detection stage where masks and boundaries are put in place then, the inputs are processed through a vastly deep network where the accumulated convolutions and poolings cause the image to importantly decrease its resolution and quality, hence results are yield with a high loss of information. DeepLab models address the challenge leveraging on Atrous convolutions and Atrous Spatial Pyramid Pooling (ASPP) architectures.

分割模型在先前的图像检测阶段使用全卷积神经网络FCNN，在此阶段放置了遮罩和边界，然后通过一个非常深的网络对输入进行处理，在该网络中累积的卷积和池化会严重降低图像的分辨率和质量，因此，结果是信息丢失率很高的结果。 DeepLab模型利用Atrous卷积和Atrous空间金字塔池(ASPP)架构来应对挑战。

Torchvision 0.3: segmentation, detection models, new datasets and more Torchvision 0.3：细分，检测模型，新数据集等

Four main stages are involved during the forward process in the DeepLab architecture:

DeepLab架构的转发过程涉及四个主要阶段：

• Extract the image features with the backbone models. The backbone used in our case is the Resnet 101 object detection model, serving as a first convolution pipe to catch and mask the important feature maps.
  使用主干模型提取图像特征。 在我们的案例中使用的主干是Resnet 101对象检测模型，它用作捕获和掩盖重要特征图的第一个卷积管道。
• To control the size of the output feature maps Atrous convolutions are used in the last few layers of the backbone.
  为了控制输出特征图的大小，在主干的最后几层中使用Atrous卷积。
• In the last stages, ASPP architectures classify the different pixels of the output image and process it through 1 x 1 convolution layers to recover their original size.
  在最后阶段，ASPP体系结构对输出图像的不同像素进行分类，并通过1 x 1卷积层对其进行处理，以恢复其原始大小。

Rethinking Atrous Convolution for Semantic Image Segmentation重新思考用于语义图像分割的Atrous卷积

2.使用Pytorch的DeepLab V3-Resnet101实现 (2. Use the DeepLab V3-Resnet101 implementation from Pytorch)

Let’s kick off the process by creating a Pytorch module that wraps the original DeepLab V3 model. Originally, the Pytorch team already propose their implementation of Google DeepLab V3 architecture pre-trained on the COCO dataset along with various backbones to choose from. For our specific task, we will go with the deeplabv3-resnet101 pre-trained module easily loadable from torchvision.models.segmentation API package.

让我们通过创建包装原始DeepLab V3模型的Pytorch模块开始这一过程。 最初，Pytorch团队已经提出了在COCO数据集上预先训练的Google DeepLab V3架构的实现方案，以及可供选择的各种主干。 对于我们的特定任务，我们将使用可从torchvision.models.segmentation API软件包轻松加载的deeplabv3-resnet101预训练模块。

创建您的python虚拟环境 (Create your python virtual environment)

Set up your python virtual environment using pip virtualenv and install all the required packages:

使用pip virtualenv设置python虚拟环境并安装所有必需的软件包：

i. Specify the path of your python virtual env:

我 指定您的python虚拟环境的路径：

virtualenv absolute/path/to/virtual/env/pytorch-en

virtualenv absolute/path/to/virtual/env/pytorch-en

ii. Activate your virtual env:

ii。 激活您的虚拟环境：

source /absolute/path/pytorch-env/bin/activate

source /absolute/path/pytorch-env/bin/activate

iii. Install the required libraries:

iii。 安装所需的库：

pip install torch torchvision

pip install torch torchvision

pip install opencv-python

pip install opencv-python

pip install numpy

pip install numpy

pip install Pillow=2.2.1

pip install Pillow=2.2.1

实施数据集样本 (Implement a Dataset sample)

Before coding the Pytorch wrapper, you will need to define a Pytorch dataset class to sample the input image file into a high-level object with multiple methods. That way, the model will be dealing with sampled objects instead of converting and handling all the related file stuff.

在对Pytorch包装器进行编码之前，您需要定义一个Pytorch数据集类，以使用多种方法将输入图像文件采样到高级对象中。 这样，模型将处理采样的对象，而不是转换和处理所有相关的文件内容。

def __init__(self, root_dir, image_file, device) The init method takes care of converting the image file into a Pillowimage object and then into a torch tensor, defining internally a set of preprocessing rules to transform the initial tensor into a more suitable input for the deeplabv3 model. The purpose is to generate a well-shaped tensor, dimension wise, to avoid any kind of annoying mismatching between the image files and the model.

def __init__(self, root_dir, image_file, device) init方法负责将图像文件转换为Pillow图像对象，然后转换为割炬张量，内部定义了一组预处理规则，以将初始张量转换为更合适的输入用于deeplabv3模型。 目的是在尺寸方向上生成形状良好的张量，以避免图像文件与模型之间的任何烦人的不匹配。

为DeepLab V3推理创建Pytorch包装器模块 (Create the Pytorch wrapper module for DeepLab V3 inference)

The SemanticSeg(nn.Module)wrapper module has three main methods:

SemanticSeg(nn.Module)包装器模块具有三种主要方法：

• def __init__(self, pretrained, device) Initialization with pre-trained value to load an already trained deeplabv3-resnet101 module, and device argument to specify CPU or GPU acceleration when dealing with inference.

  def __init__(self, pretrained, device)使用预训练的值进行初始化，以加载已训练的deeplabv3-resnet101模块，并使用设备参数指定处理推理时的CPU或GPU加速。

• def forwar(self, input) Applying module inference on a SegmentationSample input, and returning a tensor of predictions.

  def forwar(self, input)在SegmentationSample输入上应用模块推论，并返回预测的张量。

• def load_model(self, pretrained=False) Load the deeplabv3-resnet101 module from torchvision.models.segmentation API.

  def load_model(self, pretrained=False)从torchvision.models.segmentation API加载deeplabv3-resnet101模块。

Add a method to help process the outcome. Keep in mind that the output tensors have 21 channels matching the prediction results for each target class the model was trained on. Accordingly, we need to decode the tensor shape to be able to output a proper image result.

添加一种方法来帮助处理结果。 请记住，输出张量具有21个通道，这些通道与训练模型的每个目标类的预测结果匹配。 因此，我们需要解码张量形状以能够输出适当的图像结果。

background_custom(self, input_image, source, background_source, channel=21) The method takes the output tensor image with [1, 21, H, W] shape, the path to the image file, the path to the background image file, and the number of channels that are already set to 21. The goal is to extract the person channel (class 15) from the output, exclude all the remaining channels tagging them as the background, and finally merge the defined background with a new image source.

background_custom(self, input_image, source, background_source, channel=21)该方法采用[1, 21, H, W] background_custom(self, input_image, source, background_source, channel=21) [1, 21, H, W]形状的输出张量图像，图像文件的路径，背景图像文件的路径以及已设置为21的通道数。目标是从输出中提取人员通道( 类15 )，排除所有将其标记为背景的其余通道，最后将定义的背景与新图像源合并。

设置Django API (Set up the Django API)

We will be using the Django REST Framework to build a simple API for serving our model. The key idea is to configure all the required files, including the models, routing pipes, and views so that we can easily test the inference through an easy forward POST and GET requests.

我们将使用Django REST框架来构建用于服务模型的简单API。 关键思想是配置所有必需的文件，包括模型，路由管道和视图，以便我们可以通过简单的POST和GET请求轻松测试推理。

You can also follow the amazing tutorial by Bennett Garner that covers in detail all the required steps to get your API running.

您还可以遵循Bennett Garner的精彩教程，其中详细介绍了使API运行所需的所有步骤。

Typically, an API is a window into a database. The API backend handles querying the database and formatting the response. What you receive is a static response, usually in JSON format, of whatever resource you requested.

通常，API是进入数据库的窗口。 API后端处理查询数据库和格式化响应。 您收到的是您请求的任何资源的静态响应，通常为JSON格式。

设置Django项目 (Set up Django project)

Install djangoand djangorestframework : pip install django pip install djangorestframework .

安装django和djangorestframework ： pip install django pip install djangorestframework 。

Head up to the location you will be working on and start your project: django-admin startproject semanticsegmentation .

前往您要处理的位置并启动您的项目： django-admin startproject semanticsegmentation 。

Within semanticsegmentation/ folder, you will find a manage.py script to run your localhost project: python manage.py runserver .

在semanticsegmentation/文件夹中，您将找到一个manage.py脚本来运行您的localhost项目： python manage.py runserver 。

On 127.0.0.1:8080 port you should land on the welcoming Django page:

在127.0.0.1:8080端口上，您应该进入欢迎的Django页面：

Django Localhost on 127.0.0.1:8080 Django Localhost于127.0.0.1:8080

From now on your Django backend is correctly set up and running.

从现在开始，您的Django后端已正确设置并正在运行。

创建您的API应用 (Create your API app)

You will generate a new application specifically dedicated to the API management inside the previous folder.

您将在上一个文件夹中生成一个专门用于API管理的新应用程序。

Run the following command to create the folder: python manage.py startapp api

运行以下命令创建文件夹： python manage.py startapp api

Register the new API app with the project folder: Go to semanticsegmentation/settings.py and add the path to the API,

将新的API应用注册到项目文件夹：转到semanticsegmentation/settings.py然后将路径添加到API，

INSTALLED_APPS = [
'api.apps.ApiConfig',
'django.contrib.admin',
'django.contrib.auth',
...
]

定义模型并使用Django ORM进行管理 (Define your models and manage them with Django ORM)

Now in order to save and retrieve objects from the Django database, you need to build a model class that represents the entity we will be going to process in the API requests. For our particular case, we need to post images, apply the model inference to get the semantic filter and, then recover them back. Consequently, the best fit would be a class with two models.FileFieldattributes to upload the input and output image files to the server, a specific id models.UUIDFieldlinked to each stored image, a name CharFieldto identify them, and optionally, a DateTimeFieldto save the exact time when they were stored.

现在，为了从Django数据库中保存和检索对象，您需要构建一个模型类，该模型类表示我们将在API请求中处理的实体。 对于我们的特殊情况，我们需要发布图像，应用模型推断以获得语义过滤器，然后将其恢复。 因此，最合适的是有两个一类models.FileField属性，输入和输出的图像文件上传到服务器，一个特定的ID models.UUIDField链接到每个存储的图像，名称CharField识别它们，以及可选的DateTimeField保存确切的存储时间。

Define in a separate script the methods that handle the file upload logic:

在单独的脚本中定义处理文件上传逻辑的方法：

get_input_image_path

get_input_image_path

get_output_image_path

get_output_image_path

Migrate your changes to the database: python manage.py makemigrations

迁移对数据库的更改： python manage.py makemigrations

python manage.py migrate

python manage.py migrate

Define the serializers that match the model class, one serializer for the input image and another for the output result:

定义与模型类匹配的序列化器，一个用于输入图像的序列化器，另一个用于输出结果的序列化器：

Register your new model with the admin site: admin.site.register(ImageSegmentation)

在管理站点注册新模型： admin.site.register(ImageSegmentation)

建立您的意见 (Build your views)

We will be creating simple get and post methods that handle simple operations:

我们将创建处理简单操作的简单get和post方法：

• POST api_view: It sends two file images, processes them applying the model inference, and saves them to their corresponding input/output folders.

  POST api_view ：它发送两个文件图像，使用模型推断对其进行处理，然后将它们保存到其相应​​的输入/输出文件夹中。

• GET api_view: Retrieve the saved image and serve them as static resources.

  GET api_view：检索保存的图像并将其用作静态资源。

配置URL端点并运行您的API (Configure the URL endpoints and run your API)

1. Set up the url patterns in the semanticsegmentation/urls.py file:

   在semanticsegmentation/urls.py文件中设置URL模式：

urlpatterns = [
   path('admin', admin.site.urls),
   path(r'bg_custom_api', include('API.urls'))
]

2. define the address of your API endpoints in the api.urls file:

2.在api.urls文件中定义API端点的地址：

urlurlpatterns = [
 path(r'test/', views.test_api, name='test_api_communication'),
 path(r'images/', views.get_images, name='get_images'),
 path(r'api/', views.run_inference, name='run_inference_on_images'),
]

在Postman上运行您的API (Run your API on Postman)

Download Postman and start testing your API locally.

下载Postman并开始在本地测试您的API。

Postman) post request on 127.0.0.1:8000/api/inference/ Postman )在127.0.0.1:8000/api/inference/上的发布请求

• Run Grayscale Inference on 127.0.0.1:8080/api/grayscale/

  在127.0.0.1:8080/api/grayscale/上运行灰度推断

Original Photo by Calvin Lupiya on Unsplash in the left upper corner, the grayscaled version on its right, and original colored Photo by CDC on Unsplash on the lower-left corner with its segmented version on the right. 卡尔文·鲁皮亚 ( Calvin Lupiya)的原始照片 位于左上角的 “ Unsplash ”上，右侧为灰度版本，左下角为“ Unsplash ”的 CDC原始彩色照片，其右下为分段图像。

• Run Background custom inference on 127.0.0.1:8080/api/inference/

  在127.0.0.1:8080/api/inference/上运行Background自定义推断

Photo by Joanna Nix-Walkup on Unsplash and Photo by Kiana Bosman on Unsplash Joanna Nix-Walkup在 Unsplash上​​的照片和 Kiana Bosman在 Unsplash上的 照片

结论 (Conclusion)

Once you have set up all the required components for the API to run correctly, you can start experimenting with some really cool features with DeepLab V3, ranging from background custom, background grayscale, background blur and, much other creative stuff I’m sure you can come up with.

设置好API所需的所有组件以使其正常运行后，您就可以开始使用DeepLab V3尝试一些非常酷的功能，包括背景自定义，背景灰度，背景模糊以及许多其他有创意的东西。可以提出来。

A natural step after building the API for model serving would be to develop a small mobile application as a client that interacts with Django backend, hence you can test and fetch all the results previously seen.I’m willing to consider this option for my next article, so stay tuned for the second part.

构建用于模型服务的API之后，自然而然的步骤是开发一个小型移动应用程序作为与Django后端交互的客户端，因此您可以测试并获取以前看到的所有结果。我愿意在下一个中考虑使用此选项文章，敬请期待第二部分。

If you have any questions regarding the code, please get in touch with me and, don’t hesitate to e-mail me at [email protected]

如果您对代码有任何疑问，请与我联系，不要犹豫，通过[email protected]向我发送电子邮件。

The whole project code for this article can be found under my GitHub repo:

可以在我的GitHub存储库中找到本文的整个项目代码：

本文参考 (References for this article)

• DeepLabV3 model with a ResNet-101 backbone by Pytorch team

  Pytorch团队使用ResNet-101主干网的DeepLabV3 模型

• [2017 arXiv] [DeepLabv3] Rethinking Atrous Convolution for Semantic Image Segmentation by the Google Team

  [2017 arXiv] [DeepLabv3]由Google团队重新考虑Atrous卷积以进行语义图像分割

• DeepLab V3 Atrous convolutions by Sik-Ho Tsang

  曾锡豪的 DeepLab V3 Atrous卷积

翻译自: https://towardsdatascience.com/semantic-segmentation-using-a-django-api-deeplabv3-7b7904ddfed9

django api