真实案例扫描APP开发——基于实例分割实现拍照文档实时边缘检测（C++/JNI实现）
前言
这是一个安卓NDK的项目，想要实现的效果就是拍照扫描，这里只涉及到的只有边缘检测，之后会写文档滤镜、证件识别与证件1比1打印，OCR、版面分析之后的文档还原。我的开发环境是Android Studio 北极狐，真机是华为mate 30 pro，系统是HarmonyOS 4.0.0, NDK 是21.1.6352462这个版本，可实现CPU与GPU、NPU推理，推理速度与精度可以按真机去匹配，测试的效果不输于当前市面上排APP市场靠前的几个商业应用。实现的效果如下，有书本和单开页面两种效果，运行速度可以看看下面的视频：
手机拍照扫描边缘检测效果
一、边缘检测
1.传统方法
传统的边缘检测方案的难度和局限性：
-
阈值参数的依赖： Canny算法的检测效果依赖于一系列阈值参数的设置，这包括高阈值和低阈值。这些阈值通常是经验值，需要通过调试和实验来确定。引入额外步骤可能导致引入新的阈值参数，使得调参更为复杂。
-
阈值参数数量的限制： 阈值参数的数量不能太多，因为过多的参数会增加调试的复杂性，难以准确设置。同时，过多的参数可能降低算法的鲁棒性，使其在不同场景下难以表现理想效果。
-
算法鲁棒性的问题： 尽管存在一系列阈值参数，最终只有一组或少数几组固定的组合。这可能导致算法在某些场景下的鲁棒性受到影响，使得在一些情况下边缘检测效果不理想。
-
边缘图上的数学模型复杂： Canny算法在边缘图上建立复杂的数学模型，这增加了算法的实现难度。对于某些情况，算法可能无法提供令人满意的边缘检测结果。
综合而言，Canny算法在实际应用中需要仔细调参，并且其性能可能受到特定场景的限制。在某些复杂情况下，可能需要考虑其他边缘检测方法或进一步改进以提高算法的鲁棒性。
传统的检测分割方法计算成本较低，但效果相对较差，往往难以满足我们的要求。相反，深度学习为主的端到端检测分割方法计算成本较高，通常需要GPU进行计算，但其效果卓越。目前，实际应用中基本上都采用以深度学习为主的端到端分割算法，这使得图像分割在实际落地应用中取得了显著的进展。
2.图像分割
图像分割是将一幅数字图像划分为多个组成部分（一系列像素或超像素）的过程。分割的目标是通过简化和/或变换图像，使其转换为更有意义和更易于分析的内容表示。图像分割通常用于定位图像中目标和边界（如线和曲面）的位置。更准确地说，图像分割是为图像中的每个像素分配标签，其中具有相同标签的像素具有相似的特征。在图像分割领域中有多种技术，包括基于区域的分割技术、边界检测分割技术和基于聚类的分割技术。
图像分割领域的一些经典算法：
-
阈值技术： 该技术的主要目的是确定图像的最佳阈值。强度值超过阈值的像素被分配为1，其余像素的强度值被分配为零，从而形成二值图。选择阈值的方法包括Otsu、k均值聚类和最大熵法。
-
运动与交互分割： 该技术基于图像中的运动进行分割。其思想是在假设目标是运动的情况下找出两幅图中的差异，从而确定目标位置。
-
边界检测： 包括多种数学方法，其目的是标记图像中亮度变化剧烈或具有不连贯性的区域中的点。边界检测通常是其他分割技术的前提步骤，因为区域边界和边之间存在很高的关联性。
-
区域增长方法： 建立在相邻像素具有相似像素值的假设上。该方法比较每个像素与其相邻像素，如果满足相似性标准，则像素被划分到一个或多个相邻点组成的聚类中。相似性标准的选择对结果影响很大，容易受到噪声的影响。
此外，还有其他一些未在上文中提及的图像分割方法，如双聚类方法、快速匹配法、分水岭变换法等。这些方法在不同的应用场景中具有各自的优势和适用性。
二. Enet实例分割
1. 算法简介
ENet算法旨在实现快速的语义分割，同时考虑了分割的准确性和实时性能。其基本网络结构为编码-解码结构，通过下采样实现像素级的分类和上采样实现图像目标的定位。为了提高实时性，ENet采用了一些策略来减少计算量和提高采样速度。
在下采样层，滤波器操作扩大了图片的感受野，允许网络收集更多的目标上下文信息。然而，在图像语义分割中，下采样操作有两个缺点：
-
空间信息的损失： 由于特征图分辨率的降低，导致了空间信息的损失。
-
模型尺寸和计算量增加： 对于整张图像的像素分割，需要输入和输出具有相同的分辨率，因此在进行下采样操作后同样需要上采样与之匹配，这增加了模型尺寸和计算量。
对于第一个问题，FCN算法融合不同编码层产生的特征图，但这会增加网络的参数量，不利于语义分割的实时性。对于第二个问题，SegNet算法通过在最大池化层中保存特征元素索引，在解码器中进行搜索，实现解码特征的稀疏上采样图。然而，下采样仍然会损害目标空间信息的精度。
2、ENet算法网络结构
ENet算法在设计上采用了一些关键策略，以实现高效的实时语义分割：
-
初始化模块设计： ENet在初始化模块中采用了一个池化操作和一个步长为2的卷积操作并行，然后将结果特征图进行合并。这种设计旨在在网络早期使用较小尺寸和较少数量的特征图，从而大大减少了网络的参数量，提高了网络的运行速度。
-
扩张卷积的下采样： 在下采样过程中，ENet算法使用了扩张卷积（Dilated Convolution），这有助于平衡图像分辨率和图像感受野。扩张卷积允许在不降低特征图分辨率的同时扩大图像目标的感受野，提高了对更广泛上下文信息的学习能力。
-
不对称的编码器-解码器结构： ENet的结构不同于对称的结构，它包含了一个大的编码器和一个小的解码器。这样的设计有助于降低ENet网络的参数量，同时保持了对输入图像的有效表示。这种不对称结构的设计是为了更好地平衡模型的复杂性和实时性能。
3. 网络实现代码
import torch.nn as nn
import torch
class InitialBlock(nn.Module):
def __init__(self,
in_channels,
out_channels,
bias=False,
relu=True):
super().__init__()
if relu:
activation = nn.ReLU
else:
activation = nn.PReLU
# Main branch - As stated above the number of output channels for this
# branch is the total minus 3, since the remaining channels come from
# the extension branch
self.main_branch = nn.Conv2d(
in_channels,
out_channels - 3,
kernel_size=3,
stride=2,
padding=1,
bias=bias)
# Extension branch
self.ext_branch = nn.MaxPool2d(3, stride=2, padding=1)
# Initialize batch normalization to be used after concatenation
self.batch_norm = nn.BatchNorm2d(out_channels)
# PReLU layer to apply after concatenating the branches
self.out_activation = activation()
def forward(self, x):
main = self.main_branch(x)
ext = self.ext_branch(x)
# Concatenate branches
out = torch.cat((main, ext), 1)
# Apply batch normalization
out = self.batch_norm(out)
return self.out_activation(out)
class RegularBottleneck(nn.Module):
def __init__(self,
channels,
internal_ratio=4,
kernel_size=3,
padding=0,
dilation=1,
asymmetric=False,
dropout_prob=0,
bias=False,
relu=True):
super().__init__()
# Check in the internal_scale parameter is within the expected range
# [1, channels]
if internal_ratio <= 1 or internal_ratio > channels:
raise RuntimeError("Value out of range. Expected value in the "
"interval [1, {0}], got internal_scale={1}."
.format(channels, internal_ratio))
internal_channels = channels // internal_ratio
if relu:
activation = nn.ReLU
else:
activation = nn.PReLU
# Main branch - shortcut connection
# Extension branch - 1x1 convolution, followed by a regular, dilated or
# asymmetric convolution, followed by another 1x1 convolution, and,
# finally, a regularizer (spatial dropout). Number of channels is constant.
# 1x1 projection convolution
self.ext_conv1 = nn.Sequential(
nn.Conv2d(
channels,
internal_channels,
kernel_size=1,
stride=1,
bias=bias), nn.BatchNorm2d(internal_channels), activation())
# If the convolution is asymmetric we split the main convolution in
# two. Eg. for a 5x5 asymmetric convolution we have two convolution:
# the first is 5x1 and the second is 1x5.
if asymmetric:
self.ext_conv2 = nn.Sequential(
nn.Conv2d(
internal_channels,
internal_channels,
kernel_size=(kernel_size, 1),
stride=1,
padding=(padding, 0),
dilation=dilation,
bias=bias), nn.BatchNorm2d(internal_channels), activation(),
nn.Conv2d(
internal_channels,
internal_channels,
kernel_size=(1, kernel_size),
stride=1,
padding=(0, padding),
dilation=dilation,
bias=bias), nn.BatchNorm2d(internal_channels), activation())
else:
self.ext_conv2 = nn.Sequential(
nn.Conv2d(
internal_channels,
internal_channels,
kernel_size=kernel_size,
stride=1,
padding=padding,
dilation=dilation,
bias=bias), nn.BatchNorm2d(internal_channels), activation())
# 1x1 expansion convolution
self.ext_conv3 = nn.Sequential(
nn.Conv2d(
internal_channels,
channels,
kernel_size=1,
stride=1,
bias=bias), nn.BatchNorm2d(channels), activation())
self.ext_regul = nn.Dropout2d(p=dropout_prob)
# PReLU layer to apply after adding the branches
self.out_activation = activation()
def forward(self, x):
# Main branch shortcut
main = x
# Extension branch
ext = self.ext_conv1(x)
ext = self.ext_conv2(ext)
ext = self.ext_conv3(ext)
ext = self.ext_regul(ext)
# Add main and extension branches
out = main + ext
return self.out_activation(out)
class DownsamplingBottleneck(nn.Module):
def __init__(self,
in_channels,
out_channels,
internal_ratio=4,
return_indices=False,
dropout_prob=0,
bias=False,
relu=True):
super().__init__()
# Store parameters that are needed later
self.return_indices = return_indices
# Check in the internal_scale parameter is within the expected range
# [1, channels]
if internal_ratio <= 1 or internal_ratio > in_channels:
raise RuntimeError("Value out of range. Expected value in the "
"interval [1, {0}], got internal_scale={1}. "
.format(in_channels, internal_ratio))
internal_channels = in_channels // internal_ratio
if relu:
activation = nn.ReLU
else:
activation = nn.PReLU
# Main branch - max pooling followed by feature map (channels) padding
self.main_max1 = nn.MaxPool2d(
2,
stride=2,
return_indices=return_indices)
# Extension branch - 2x2 convolution, followed by a regular, dilated or
# asymmetric convolution, followed by another 1x1 convolution. Number
# of channels is doubled.
# 2x2 projection convolution with stride 2
self.ext_conv1 = nn.Sequential(
nn.Conv2d(
in_channels,
internal_channels,
kernel_size=2,
stride=2,
bias=bias), nn.BatchNorm2d(internal_channels), activation())
# Convolution
self.ext_conv2 = nn.Sequential(
nn.Conv2d(
internal_channels,
internal_channels,
kernel_size=3,
stride=1,
padding=1,
bias=bias), nn.BatchNorm2d(internal_channels), activation())
# 1x1 expansion convolution
self.ext_conv3 = nn.Sequential(
nn.Conv2d(
internal_channels,
out_channels,
kernel_size=1,
stride=1,
bias=bias), nn.BatchNorm2d(out_channels), activation())
self.ext_regul = nn.Dropout2d(p=dropout_prob)
# PReLU layer to apply after concatenating the branches
self.out_activation = activation()
def forward(self, x):
# Main branch shortcut
if self.return_indices:
main, max_indices = self.main_max1(x)
else:
main = self.main_max1(x)
# Extension branch
ext = self.ext_conv1(x)
ext = self.ext_conv2(ext)
ext = self.ext_conv3(ext)
ext = self.ext_regul(ext)
# Main branch channel padding
n, ch_ext, h, w = ext.size()
ch_main = main.size()[1]
padding = torch.zeros(n, ch_ext - ch_main, h, w)
# Before concatenating, check if main is on the CPU or GPU and
# convert padding accordingly
if main.is_cuda:
padding = padding.cuda()
# Concatenate
main = torch.cat((main, padding), 1)
# Add main and extension branches
out = main + ext
return self.out_activation(out), max_indices
class UpsamplingBottleneck(nn.Module):
def __init__(self,
in_channels,
out_channels,
internal_ratio=4,
dropout_prob=0,
bias=False,
relu=True):
super().__init__()
# Check in the internal_scale parameter is within the expected range
# [1, channels]
if internal_ratio <= 1 or internal_ratio > in_channels:
raise RuntimeError("Value out of range. Expected value in the "
"interval [1, {0}], got internal_scale={1}. "
.format(in_channels, internal_ratio))
internal_channels = in_channels // internal_ratio
if relu:
activation = nn.ReLU
else:
activation = nn.PReLU
# Main branch - max pooling followed by feature map (channels) padding
self.main_conv1 = nn.Sequential(
nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=bias),
nn.BatchNorm2d(out_channels))
# Remember that the stride is the same as the kernel_size, just like
# the max pooling layers
self.main_unpool1 = nn.MaxUnpool2d(kernel_size=2)
# Extension branch - 1x1 convolution, followed by a regular, dilated or
# asymmetric convolution, followed by another 1x1 convolution. Number
# of channels is doubled.
# 1x1 projection convolution with stride 1
self.ext_conv1 = nn.Sequential(
nn.Conv2d(
in_channels, internal_channels, kernel_size=1, bias=bias),
nn.BatchNorm2d(internal_channels), activation())
# Transposed convolution
self.ext_tconv1 = nn.ConvTranspose2d(
internal_channels,
internal_channels,
kernel_size=2,
stride=2,
bias=bias)
self.ext_tconv1_bnorm = nn.BatchNorm2d(internal_channels)
self.ext_tconv1_activation = activation()
# 1x1 expansion convolution
self.ext_conv2 = nn.Sequential(
nn.Conv2d(
internal_channels, out_channels, kernel_size=1, bias=bias),
nn.BatchNorm2d(out_channels))
self.ext_regul = nn.Dropout2d(p=dropout_prob)
# PReLU layer to apply after concatenating the branches
self.out_activation = activation()
def forward(self, x, max_indices, output_size):
# Main branch shortcut
main = self.main_conv1(x)
main = self.main_unpool1(
main, max_indices, output_size=output_size)
# Extension branch
ext = self.ext_conv1(x)
ext = self.ext_tconv1(ext, output_size=output_size)
ext = self.ext_tconv1_bnorm(ext)
ext = self.ext_tconv1_activation(ext)
ext = self.ext_conv2(ext)
ext = self.ext_regul(ext)
# Add main and extension branches
out = main + ext
return self.out_activation(out)
class ENet(nn.Module):
def __init__(self, num_classes, encoder_relu=False, decoder_relu=True):
super().__init__()
self.initial_block = InitialBlock(3, 16, relu=encoder_relu)
# Stage 1 - Encoder
self.downsample1_0 = DownsamplingBottleneck(
16,
64,
return_indices=True,
dropout_prob=0.01,
relu=encoder_relu)
self.regular1_1 = RegularBottleneck(
64, padding=1, dropout_prob=0.01, relu=encoder_relu)
self.regular1_2 = RegularBottleneck(
64, padding=1, dropout_prob=0.01, relu=encoder_relu)
self.regular1_3 = RegularBottleneck(
64, padding=1, dropout_prob=0.01, relu=encoder_relu)
self.regular1_4 = RegularBottleneck(
64, padding=1, dropout_prob=0.01, relu=encoder_relu)
# Stage 2 - Encoder
self.downsample2_0 = DownsamplingBottleneck(
64,
128,
return_indices=True,
dropout_prob=0.1,
relu=encoder_relu)
self.regular2_1 = RegularBottleneck(
128, padding=1, dropout_prob=0.1, relu=encoder_relu)
self.dilated2_2 = RegularBottleneck(
128, dilation=2, padding=2, dropout_prob=0.1, relu=encoder_relu)
self.asymmetric2_3 = RegularBottleneck(
128,
kernel_size=5,
padding=2,
asymmetric=True,
dropout_prob=0.1,
relu=encoder_relu)
self.dilated2_4 = RegularBottleneck(
128, dilation=4, padding=4, dropout_prob=0.1, relu=encoder_relu)
self.regular2_5 = RegularBottleneck(
128, padding=1, dropout_prob=0.1, relu=encoder_relu)
self.dilated2_6 = RegularBottleneck(
128, dilation=8, padding=8, dropout_prob=0.1, relu=encoder_relu)
self.asymmetric2_7 = RegularBottleneck(
128,
kernel_size=5,
asymmetric=True,
padding=2,
dropout_prob=0.1,
relu=encoder_relu)
self.dilated2_8 = RegularBottleneck(
128, dilation=16, padding=16, dropout_prob=0.1, relu=encoder_relu)
# Stage 3 - Encoder
self.regular3_0 = RegularBottleneck(
128, padding=1, dropout_prob=0.1, relu=encoder_relu)
self.dilated3_1 = RegularBottleneck(
128, dilation=2, padding=2, dropout_prob=0.1, relu=encoder_relu)
self.asymmetric3_2 = RegularBottleneck(
128,
kernel_size=5,
padding=2,
asymmetric=True,
dropout_prob=0.1,
relu=encoder_relu)
self.dilated3_3 = RegularBottleneck(
128, dilation=4, padding=4, dropout_prob=0.1, relu=encoder_relu)
self.regular3_4 = RegularBottleneck(
128, padding=1, dropout_prob=0.1, relu=encoder_relu)
self.dilated3_5 = RegularBottleneck(
128, dilation=8, padding=8, dropout_prob=0.1, relu=encoder_relu)
self.asymmetric3_6 = RegularBottleneck(
128,
kernel_size=5,
asymmetric=True,
padding=2,
dropout_prob=0.1,
relu=encoder_relu)
self.dilated3_7 = RegularBottleneck(
128, dilation=16, padding=16, dropout_prob=0.1, relu=encoder_relu)
# Stage 4 - Decoder
self.upsample4_0 = UpsamplingBottleneck(
128, 64, dropout_prob=0.1, relu=decoder_relu)
self.regular4_1 = RegularBottleneck(
64, padding=1, dropout_prob=0.1, relu=decoder_relu)
self.regular4_2 = RegularBottleneck(
64, padding=1, dropout_prob=0.1, relu=decoder_relu)
# Stage 5 - Decoder
self.upsample5_0 = UpsamplingBottleneck(
64, 16, dropout_prob=0.1, relu=decoder_relu)
self.regular5_1 = RegularBottleneck(
16, padding=1, dropout_prob=0.1, relu=decoder_relu)
self.transposed_conv = nn.ConvTranspose2d(
16,
num_classes,
kernel_size=3,
stride=2,
padding=1,
bias=False)
def forward(self, x):
# Initial block
input_size = x.size()
x = self.initial_block(x)
# Stage 1 - Encoder
stage1_input_size = x.size()
x, max_indices1_0 = self.downsample1_0(x)
x = self.regular1_1(x)
x = self.regular1_2(x)
x = self.regular1_3(x)
x = self.regular1_4(x)
# Stage 2 - Encoder
stage2_input_size = x.size()
x, max_indices2_0 = self.downsample2_0(x)
x = self.regular2_1(x)
x = self.dilated2_2(x)
x = self.asymmetric2_3(x)
x = self.dilated2_4(x)
x = self.regular2_5(x)
x = self.dilated2_6(x)
x = self.asymmetric2_7(x)
x = self.dilated2_8(x)
# Stage 3 - Encoder
x = self.regular3_0(x)
x = self.dilated3_1(x)
x = self.asymmetric3_2(x)
x = self.dilated3_3(x)
x = self.regular3_4(x)
x = self.dilated3_5(x)
x = self.asymmetric3_6(x)
x = self.dilated3_7(x)
# Stage 4 - Decoder
x = self.upsample4_0(x, max_indices2_0, output_size=stage2_input_size)
x = self.regular4_1(x)
x = self.regular4_2(x)
# Stage 5 - Decoder
x = self.upsample5_0(x, max_indices1_0, output_size=stage1_input_size)
x = self.regular5_1(x)
x = self.transposed_conv(x, output_size=input_size)
return x
if __name__ == '__main__':
x = torch.randn(1, 3, 256, 256)
net = ENet(13)(x)
print(net.shape) # torch.Size([1, 13, 256, 256])
4、不对称卷积：
标准卷积权重存在相当数量的冗余。通过将一个滤波器为nxn的卷积层分解成两个连续的卷积层，其中一个卷积层具有一个nx1的滤波器，另一个卷积层具有一个1xn的滤波器，可以减少这种冗余信息。这种分解卷积也被称为不对称卷积。ENet算法中采用了n=5的不对称卷积，通过这两个步骤产生的计算量与一个3x3的卷积层相似。这有助于增加模型学习函数的多样性，并增加感受野。
5、扩张卷积：
为了避免特征图的过度下采样，ENet算法使用了扩张卷积替代了最小分辨率操作阶段中的几个编码模型的主要卷积层。这种操作使得精度显著提升。
6、正则化：
由于大多数像素级分割数据集相当较小，复杂的神经网络模型容易过拟合，从而导致模型泛化能力下降。正则化参数相当于对参数引入先验分布，调节模型允许存储的信息量，并对其加以约束，从而降低模型的复杂度，有助于减少过拟合。
7、模型推理：
ENet在CmaVid数据集上训练后的模型大小为4.27M，非常轻量。这使得ENet在嵌入式系统和移动设备上的推理过程更加高效。
三、NDK开发
1.NCNN
NCNN是一个轻量级的深度学习框架，专为移动设备和嵌入式系统设计。它在模型推理过程中注重速度和轻量级，采用C++实现，支持多种硬件平台，包括CPU、GPU、以及一些专用的神经网络加速器。
-
轻量级设计： NCNN的设计目标是轻量级，使其适用于移动设备和嵌入式系统。它采用C++实现，具有较小的代码体积和内存占用。
-
硬件加速支持： NCNN支持多种硬件平台，包括通用的CPU和GPU，以及一些专用的神经网络加速器，如华为的NPU（神经处理单元）等。
-
高性能： NCNN在模型推理过程中注重性能，采用了一系列优化策略，包括内存重用、多线程并发等，以提高推理速度。
-
跨平台支持： NCNN在多个操作系统上运行，包括Linux、Windows、Android等，为开发者提供了较大的灵活性。
-
丰富的模型支持： NCNN支持多种深度学习模型的加载和推理，包括常见的卷积神经网络（CNN）、循环神经网络（RNN）等。
-
模型转换工具： NCNN提供了模型转换工具，支持从主流深度学习框架（如TensorFlow、Caffe、ONNX等）中导入模型，以便在NCNN上进行推理。
2. 模型转换
将ONNX模型转换为NCNN模型可以通过使用NCNN框架提供的工具进行。NCNN提供了一个名为onnx2ncnn的工具，可以将ONNX模型转换为NCNN模型。
onnx2ncnn input.onnx output.param output.bin
input.onnx是输入的 ONNX 模型文件。output.param和output.bin是输出的 NCNN 模型文件，其中output.param包含了网络结构的参数，output.bin包含了模型的权重参数。
3.NDK推理代码
#pragma once
#include "net.h"
#include #include "DocumentEdge.h"
namespace SCAN
{
struct Line
{
cv::Point _p1;
cv::Point _p2;
cv::Point _center;
Line(cv::Point p1, cv::Point p2)
{
_p1 = p1;
_p2 = p2;
_center = cv::Point((p1.x + p2.x) / 2, (p1.y + p2.y) / 2);
}
};
DocumentEdge::DocumentEdge()
{
}
DocumentEdge::~DocumentEdge()
{
}
int DocumentEdge::read_model(std::string param_path, std::string bin_path, ncnn::Net& net, bool use_gpu)
{
bool has_GPU = false;
#if NCNN_VULKAN
ncnn::create_gpu_instance();
has_GPU = ncnn::get_gpu_count() > 0;
#endif
bool to_use_GPU = has_GPU && use_gpu;
net.opt.use_vulkan_compute = to_use_GPU;
int rb = -1;
int rm = -1;
rb = net.load_param(param_path.c_str());
rm = net.load_model(bin_path.c_str());
if (rb < 0 || rm < 0)
{
return -1;
}
if (to_use_GPU)
{
return 1;
}
return 0;
}
int DocumentEdge::read_model(std::string edge_model_parma,std::string edge_model_bin,
std::string mid_model_parma,std::string mid_model_bin,bool use_gpu, int book_mid)
{
if (book_mid == 1)
{
read_model(mid_model_parma, mid_model_bin, mid_net, use_gpu);
}
read_model(edge_model_parma, edge_model_bin, edge_net, use_gpu);
return 0;
}
int DocumentEdge::inference(ncnn::Net& net, cv::Mat& cv_src, cv::Mat& cv_dst, int target_size)
{
if (cv_src.empty())
{
return -20;
}
ncnn::Mat in = ncnn::Mat::from_pixels_resize(cv_src.data, ncnn::Mat::PIXEL_BGR,
cv_src.cols, cv_src.rows, target_size, target_size);
const float norm_vals[3] = { 1 / 255.f, 1 / 255.f, 1 / 255.f };
in.substract_mean_normalize(0, norm_vals);
ncnn::Extractor ex = net.create_extractor();
ex.set_num_threads(threads);
ncnn::Mat out;
ex.input("input.1", in);
ex.extract("887", out);
cv::Mat cv_seg = cv::Mat::zeros(cv::Size(out.w, out.h), CV_8UC1);
for (int i = 0; i < out.h; ++i)
{
for (int j = 0; j < out.w; ++j)
{
const float* bg = out.channel(0);
const float* fg = out.channel(1);
if (bg[i * out.w + j] < fg[i * out.w + j])
{
cv_seg.data[i * out.w + j] = 255;
}
}
}
cv::resize(cv_seg, cv_dst, cv::Size(cv_src.cols, cv_src.rows), cv::INTER_LINEAR);
return 0;
}
int DocumentEdge::targetArea(cv::Mat &cv_src, cv::Mat& cv_seg, int e = 7, int d = 5)
{
cv::Mat cv_temp = cv_src.clone();
//计算分割到的面积
float seg_total = countNonZero(cv_temp);
float pix_total = cv_src.cols * cv_src.rows;
int index = 0;
if (seg_total >= pix_total * 0.9)
{
e = 7;
d = 7;
index = 9;
}
else if ((seg_total < pix_total * 0.9) && (seg_total >= pix_total * 0.7))
{
e = 7;
d = 5;
index = 8;
}
else if ((seg_total < pix_total * 0.7) && (seg_total >= pix_total * 0.5))
{
e = 6;
d = 5;
index = 7;
}
else if ((seg_total < pix_total * 0.5) && (seg_total >= pix_total * 0.3))
{
e = 4;
d = 3;
index = 6;
}
else
{
e = 1;
d = 1;
index = 0;
}
cv::Mat cv_dilate, cv_erode;
cv::Mat element_e = getStructuringElement(cv::MORPH_RECT, cv::Size(e, e), cv::Point(-1, -1));
cv::Mat element_d = getStructuringElement(cv::MORPH_RECT, cv::Size(d, d), cv::Point(-1, -1));
cv::erode(cv_src, cv_dilate, element_e);
cv::dilate(cv_dilate, cv_seg, element_d);
cv::threshold(cv_seg, cv_seg, 100, 255, cv::THRESH_BINARY);
return index;
}
//线排序Y轴
static bool cmp_y(const Line& p1, const Line& p2)
{
return p1._center.y < p2._center.y;
}
//线排序X轴
static bool cmp_x(const Line& p1, const Line& p2)
{
return p1._center.x < p2._center.x;
}
//点排序
static bool point_y(const cv::Point& p1, const cv::Point& p2)
{
return p1.y < p2.y;
}
//点排序
static bool point_x(const cv::Point& p1, const cv::Point& p2)
{
return p1.x < p2.x;
}
//两条线的交点
static cv::Point2f computeIntersect(Line& l1, Line& l2)
{
int x1 = l1._p1.x;
int x2 = l1._p2.x;
int y1 = l1._p1.y;
int y2 = l1._p2.y;
int x3 = l2._p1.x, x4 = l2._p2.x, y3 = l2._p1.y, y4 = l2._p2.y;
if (float d = (x1 - x2) * (y3 - y4) - (y1 - y2) * (x3 - x4))
{
cv::Point2f pt;
pt.x = ((x1 * y2 - y1 * x2) * (x3 - x4) - (x1 - x2) * (x3 * y4 - y3 * x4)) / d;
pt.y = ((x1 * y2 - y1 * x2) * (y3 - y4) - (y1 - y2) * (x3 * y4 - y3 * x4)) / d;
return pt;
}
return cv::Point2f(-1, -1);
}
static bool cmpPointX(const cv::Point& p1, const cv::Point& p2)
{
return p1.x < p2.x;
}
//点排序
static void sortPoint(std::vector<cv::Point>& old_point, std::vector<cv::Point>& new_point)
{
sort(old_point.begin(), old_point.end(), cmpPointX);
new_point = std::vector<cv::Point>(4);
if (old_point.at(0).y < old_point.at(1).y)
{
new_point.at(0) = old_point.at(0);
new_point.at(2) = old_point.at(1);
}
else
{
new_point.at(2) = old_point.at(0);
new_point.at(0) = old_point.at(1);
}
if (old_point.at(2).y < old_point.at(3).y)
{
new_point.at(1) = old_point.at(2);
new_point.at(3) = old_point.at(3);
}
else
{
new_point.at(1) = old_point.at(3);
new_point.at(3) = old_point.at(2);
}
}
//得到直线斜率
static std::vector <float> getLinesArctan(std::vector<cv::Vec4f> lines)
{
float k = 0; //直线斜率
std::vector <float> lines_arctan;//直线斜率的反正切值
for (unsigned int i = 0; i < lines.size(); i++)
{
k = (double)(lines[i][3] - lines[i][1]) / (double)(lines[i][2] - lines[i][0]); //求出直线的斜率
lines_arctan.push_back(atan(k));
}
return lines_arctan;
}
//得到线的角度
static void getLinesAngle(std::vector<cv::Vec4f> lines, std::vector<double>& angle)
{
//显示每条直线的角度
std::vector <float> lines_arctan;//直线斜率的反正切值
lines_arctan = getLinesArctan(lines);
for (unsigned int i = 0; i < lines.size(); i++)
{
angle.push_back(lines_arctan[i] * 180.0 / 3.1415926);
}
}
static double angle(cv::Point pt1, cv::Point pt2, cv::Point pt0)
{
double dx1 = pt1.x - pt0.x;
double dy1 = pt1.y - pt0.y;
double dx2 = pt2.x - pt0.x;
double dy2 = pt2.y - pt0.y;
return (dx1 * dx2 + dy1 * dy2) / sqrt((dx1 * dx1 + dy1 * dy1) * (dx2 * dx2 + dy2 * dy2) + 1e-10);
}
/// box(contours.size());
for (int i = 0; i < contours.size(); i++)
{
approxPolyDP(cv::Mat(contours[i]), contours[i], 3, true);
cv::Rect rect = boundingRect(cv::Mat(contours[i]));
int area_t = rect.area();
if (area_t >= area)
{
area = area_t;
}
}*/
//auto t1 = cv::getTickCount();
//std::cout << "elapsed time: " << (t1 - t0) * 1000.0 / cv::getTickFrequency() << "ms" << std::endl;
return 0;
}
///