小程序sketch
This is the 2nd part of a 3 part series for Sketch3D, an application to perform sketch-based authoring in an augmented reality (AR) environment to turn 2D sketches into 3D objects.
这是Sketch3D的3部分系列的第2部分， Sketch3D是一个在增强现实(AR)环境中执行基于草图的创作以将2D草图转换为3D对象的应用程序。
第一部分涵盖了什么？ (What Was Covered in Part 1?)
- Introduction into the Unity Editor (AR/VR Development Tool) Unity编辑器简介(AR / VR开发工具)
- Using the Vuforia AR Engine to detect Image Targets and anchor virtual content 使用Vuforia AR Engine检测图像目标并锚定虚拟内容
- Setting up the application for deployment on an Android device 设置应用程序以在Android设备上进行部署
- Easy start-up through Github (if Unity and Vuforia are already installed) 通过Github轻松启动(如果已经安装了Unity和Vuforia)
第2部分介绍了什么？ (What’s Covered in Part 2?)
- Communication between a Python server and an AR Android application written in C# Python服务器和用C＃编写的AR Android应用程序之间的通信
- Computer vision techniques to process 2D sketches 用于处理2D草图的计算机视觉技术
- Dynamic rendering of 3D objects in the Unity environment 在Unity环境中动态渲染3D对象
- and of course… Code! 当然是……代码！
第3部分介绍了什么？ (What’s Covered in Part 3?)
- The basics of Facebook’s PyTorch library, a deep learning toolbox, for training a 2D segmentation network. Facebook的PyTorch库的基础知识，这是一个深度学习工具箱，用于训练2D分割网络。
- The heavy data augmentation used to create a data set suitable for performing annotation segmentation 用于创建适合执行注释分段的数据集的大量数据扩充
Training and testing of the U-Net 2D segmentation network in Google Colab environment
在Google Colab环境中培训和测试U-Net 2D细分网络
- Deployment and integration of the segmentation network into Sketch3D 将细分网络部署并集成到Sketch3D中
资源资源 (Resources)
Github link: Sketch3D project repository
Github链接 ： Sketch3D项目存储库
Google Colab: A Google colab notebook (ipynb) to train the segmentation network.
Google Colab ：用于训练细分网络的Google colab笔记本(ipynb)。
第1章。设计一个简单的Python服务器 (Section 1. Designing a Simple Python Server)
The project is separated into two parts: a Python server and the Android client application written in C# using the Unity development platform. The purpose of the Python server is to process the sketch captured by the camera of the Android phone and to send the necessary information to construct a 3D object back to the client. When the target image is detected, the client constructs the 3D object and renders it in a mixed reality environment viewable on the phone. Let’s begin with a very simple HTTP Python server to introduce the client/server architecture.
该项目分为两部分：Python服务器和使用Unity开发平台以C＃编写的Android客户端应用程序。 Python服务器的目的是处理由Android手机的摄像头捕获的草图，并发送必要的信息以将3D对象构造回客户端。 当检测到目标图像时，客户端将构建3D对象，并在可在手机上查看的混合现实环境中渲染该3D对象。 让我们从一个非常简单的HTTP Python服务器开始，以介绍客户端/服务器体系结构。
You can replace the IP address field in the main method to the IP address of your device (the IP address can be found by running the terminal command ifconfig on Linux devices or ipconfig on Windows/Mac devices). The handler() class is used to define custom behavior to handle incoming HTTP requests. In the example above, the function do_GET() is used to handle GET requests and do_POST() is used to handle POST requests. The self.path instance variable associated with our Handler instance can be used to route specific requests to the appropriate handler based on the path property.
您可以将main方法中的IP地址字段替换为设备的IP地址(可以通过在Linux设备上运行terminal命令ifconfig或在Windows / Mac设备上运行ipconfig来找到IP地址)。 handler()类用于定义自定义行为以处理传入的HTTP请求。 在上面的示例中，函数do_GET()用于处理GET请求，而do_POST()用于处理POST请求。 与我们的Handler实例相关联的self.path实例变量可用于基于path属性将特定请求路由到适当的处理程序。
The API is actually quite simple and is summarized below. The HTTP GET /data request acquires the information necessary to construct the 3D object and the HTTP POST request sends an image to the server to be processed. Note: The HTTP GET/annotation request will be discussed in Part 3. when annotation support is introduced.
该API实际上非常简单，下面进行总结。 HTTP GET /data请求获取构造3D对象所需的信息，并且HTTP POST请求将图像发送到要处理的服务器。 注意：引入注释支持时，将在第3部分中讨论HTTP GET /annotation请求。
第2节。处理草图 (Section 2. Processing Sketches)
The processing routines are encapsulated inside a class called Processor(). Upon receiving an image from the Android application, the server uses the Processor instance to maintain the state of the application and perform the necessary computer vision tasks. Sketch3D works by having the user take a picture of the side and front of the 3D object to construct. When an image is sent to the server, the Processor instance keeps track of the type of image (“Side” or “Front”) and any useful information that is computed during the processing pipeline, such as the location of the corners of the sketch.
处理例程封装在称为Processor()的类中。 从Android应用程序收到图像后，服务器使用Processor实例维护应用程序的状态并执行必要的计算机视觉任务。 Sketch3D的工作原理是让用户拍摄要构造的3D对象的侧面和正面的图片。 将图像发送到服务器时，Processor实例会跟踪图像的类型(“ Side”或“ Front”)以及在处理管线期间计算出的任何有用信息，例如草图拐角的位置。
处理管道高级概述 (High-Level Overview of Processing Pipeline)
The general processing pipeline to convert a sketch into a 3D virtual object is as follows:
将草图转换为3D虚拟对象的常规处理管道如下：
- The Android application sends an image to the server via HTTP POST request. Android应用程序通过HTTP POST请求将图像发送到服务器。
- The Processor instance crops the image to the region of interest. Processor实例将图像裁剪到感兴趣的区域。
- The Processor instance looks for any annotations with defined behavior and removes the annotations from the image (Part 3 discusses the annotation feature). Processor实例将查找具有已定义行为的所有注释，然后从图像中删除这些注释(第3部分讨论了注释功能)。
- The Processor instance extracts the corners of the sketch. Processor实例提取草图的角。
- The Processor instance constructs a list of faces, where each element of the list is a set of 3D points that define the vertices of that face. Processor实例构造一个面列表，其中列表中的每个元素都是一组3D点，这些点定义了该面的顶点。
- The server encodes and sends the information to construct the 3D object to the client through the HTTP GET /data request. 服务器通过HTTP GET / data请求对信息进行编码并将其发送以构造3D对象给客户端。
接收和裁剪图像文件 (Receiving and Cropping an Image File)
The server uses the following code snippet to handle the POST request that contains an image file (png) captured by the AR Android application.
服务器使用以下代码段处理POST请求，该请求包含AR Android应用程序捕获的图像文件(png)。
def deal_post_data(self):
ctype, pdict = cgi.parse_header(self.headers['Content-Type'])
pdict['boundary'] = bytes(pdict['boundary'], "utf-8")
pdict['CONTENT-LENGTH'] = int(self.headers['Content-Length'])
if ctype == 'multipart/form-data':
form = cgi.FieldStorage( fp=self.rfile, headers=self.headers, environ={'REQUEST_METHOD':'POST', 'CONTENT_TYPE':self.headers['Content-Type'], })
# typeOfImage = form["Type"].value + '.png'
bbox = {
'x':int(form["x"].value),
'y':int(form["y"].value),
'width':int(form["width"].value),
'height':int(form["height"].value)
}
try:
byteThings = b""
if isinstance(form["file"], list):
for record in form["file"]:
#open("./%s"%name, "wb").write(record.file.read())
byteThings += record.file.read()
else:
#open("./%s"%name, "wb").write(form["file"].file.read())
byteThings += form["file"].file.read()
nparr = np.fromstring(byteThings, np.uint8)
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
cropped_img = processor.crop(img,bbox['x'],bbox['width'],bbox['y'],bbox['height'])
processor.set_image(form["Type"].value,cropped_img)
except IOError:
return (False, "Can't create file to write, do you have permission to write?")
return (True, "Files uploaded",form["Type"].value)
def do_POST(self):
r, info,typeOfImage = self.deal_post_data()
print(r, info, "by: ", self.client_address)
processor.processImage(typeOfImage)
f = io.BytesIO()
message = ''
if r:
self.send_response(200)
self.end_headers()
message = processor.message()
else:
self.send_response(500)
self.end_headers()
message = 'error'
f.write(message.encode())
self.wfile.write(f.getvalue())
f.close()The image is cropped after being read into a Numpy matrix using OpenCV. The coordinates of the cropping method are provided as additional information in the POST request. These coordinates are determined by the client user-interface.
使用OpenCV将其读取到Numpy矩阵后，将图像裁剪。 裁剪方法的坐标在POST请求中作为附加信息提供。 这些坐标由客户端用户界面确定。
Upon clicking either button in the upper-left or upper-right corner, which capture an image for the side and front of our 3D object respectively, a screen shot of the user-interface and the bounding box coordinates are sent to the server. The coordinates of the cropping function are based on the coordinates defining the bounding box shown in light-gray on the user interface (Figure 1).
单击左上角或右上角的任一按钮(分别捕获3D对象的侧面和正面的图像)后，用户界面的屏幕截图和边框坐标将发送到服务器。 裁剪功能的坐标基于定义用户界面( 图1 )上以浅灰色显示的边界框的坐标。
提取草图的角 (Extracting the Corners of the Sketch)
Now that we have isolated the sketch in our region of interest, we now want to extract its corners so that we can reconstruct the polygon. The following code snippet below is used to extract the corners of the polygon from the sketch.
既然我们已经在感兴趣的区域中隔离了草图，我们现在想要提取其角，以便我们可以重建多边形。 下面的以下代码段用于从草图中提取多边形的角。
import cv2
import numpy as np
def calculate_hull(img,sensitivity=0.25):
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
gray = np.float32(gray)
corners = cv2.goodFeaturesToTrack(gray, 100, sensitivity, 10)
corners = np.int0(corners)
hull = cv2.convexHull(corners)
return hullIn the code, the first step is to convert the image into a gray scale. The OpenCV goodFeaturesToTrack() function is used to detect corners in our image. Then, the convexHull() function finds the smallest convex polygon that encloses the set of points extracted in the previous step. In mathematical terms, a convex hull is a shape with no interior angles greater than 180 degrees. You can imagine this function as attempting to tighten a closed loop around the detected corners so long as the surface of the loop never trends inwards or forms a ‘V’ — any points touching this loop are kept and the rest of the points are discarded. This is useful for several reasons. First, if some points were detected inside of the sketch they will be ignored by this function because touching these points would cause a ‘V’ or an interior angle greater than 180 degrees. Furthermore, convexHull returns a set of points that are in a clockwise orientation, which is useful because in Unity in order to define a polygon, you must specify the points in a clockwise, winding order in the direction that you are facing the polygon in the virtual environment.
在代码中，第一步是将图像转换为灰度。 OpenCV goodFeaturesToTrack() 函数用于检测图像中的角。 然后，函数凸凸convexHull()找到包含上一步中提取的点集的最小凸多边形。 用数学术语来说，凸包是没有内角大于180度的形状。 您可以想象此功能是尝试在检测到的角上收紧一个闭合环路，只要环路的表面从不向内弯曲或形成“ V”形-保留与该环路相关的所有点，其余的点均被丢弃。 出于几个原因，这很有用。 首先，如果在草图内部检测到某些点，则此功能将忽略它们，因为触摸这些点将导致“ V”或大于180度的内角。 此外，convertHull返回一组沿顺时针方向的点，这很有用，因为在Unity中，为了定义多边形，您必须在面向多边形的方向上以顺时针旋转的顺序指定点虚拟环境。
Winding order must be specified because it determines whether or not an object is being seen from the front or the back. This requirement is kind of confusing at first but what it means simply is that in Unity defining a polygon that can be viewed from any angle requires defining two sets of vertices, one in clockwise, winding order and the other in counter-clockwise winding order. If only one is specified, there is the unexpected behavior that viewing from one side of the polygon will result in an invisible polygon (weird!). It’s not how things work in real-life, but in our virtual environment these are the rules.
必须指定绕线顺序，因为它确定是从正面还是背面看到物体。 这项要求起初有点令人困惑，但其简单含义是，在Unity中定义可以从任何角度查看的多边形都需要定义两组顶点，一组顶点按顺时针缠绕顺序排列，另一组按逆时针缠绕顺序排列。 如果仅指定一个，则发生意外的行为，即从多边形的一侧查看将导致不可见的多边形(怪异！)。 这不是现实生活中的工作方式，而是在我们的虚拟环境中，这是规则。
创建3D面 (Creating 3D Faces)
Let’s say at this point the client has sent over two pictures, a sketch of the front and side of the 3D object to create, and the server has detected the corners of these sketches. The android application now detects the Image Target and wants to construct the 3D object. To do so, the server must construct a set of faces that define a 3D object using only the vertices defining the front and side face. The code snippet below describes this process at a high-level.
假设此时客户已发送了两张图片，即要创建的3D对象的正面和侧面的草图，并且服务器已检测到这些草图的角。 android应用程序现在检测到图像目标，并希望构造3D对象。 为此，服务器必须构造一组仅使用定义正面和侧面的顶点定义3D对象的面。 下面的代码片段从高层次描述了此过程。
def create3DFaces(sideHull,frontHull):
sideHull = normalize(sideHull)
frontHull = normalize(frontHull)
sideHull = addZAxis(sideHull)
frontHull = addZAxis(frontHull)
frontHull = rotate_by_90(frontHull)
frontHull,sideHull,hull_back = match_front_face(frontHull,sideHull)
faces = construct_faces(frontHull,hull_back)
faces = scale_down_faces(faces)
return facesFirst, the vertices are normalized across the width and height of the input sketch. Next, a z-axis is added to each point to create 3D points. Then, the front face is rotated by 90 degrees around the z-axis using a rotation matrix. Next, the back face is extruded and the points of the front, side, and extruded back face are matched together. Finally, the remaining sides of the 3D object are constructed and a list is returned, where each element of the list is a set of points defining a face of our 3D object. Each point is also scaled by a constant factor to make the object realistically rendered in our mixed reality environment.
首先，在输入草图的宽度和高度上对顶点进行归一化 。 接下来，将z轴添加到每个点以创建3D点。 然后，使用旋转矩阵使正面绕z轴旋转90度 。 接下来， 将背面挤压，并将正面，侧面和挤压背面的点匹配在一起 。 最后， 构造3D对象的其余部分并返回一个列表 ，其中列表中的每个元素都是一组点，这些点定义了3D对象的面。 每个点还按恒定因子缩放，以使对象在我们的混合现实环境中逼真的渲染。
As a side, the back face is extruded by considering the scale of the side and number of points. For example, if the side has 4 corners, then the relative size of the back face to the front face is the relative size of the front 2 corners to the back 2 corners. If the side has 3 corners, then the back face is a single point. Fewer than 3 corners or greater than 4 corners is not supported by Sketch3D.
作为侧面，通过考虑侧面的比例和点数来拉伸背面。 例如，如果侧面有4个角，则背面与正面的相对大小是前2个角与背面2个角的相对大小。 如果侧面有3个角，则背面为单点。 Sketch3D不支持少于3个角或大于4个角。
通过3D面Kong发送 (Sending over the 3D Faces)
Finally, we are at the last step of the processing pipeline. At this point, we have a list faces. All we have to do now is send this list to the Android client and let it deal with constructing the object. To send the list over, a custom encoding method is defined in our GET /data request handler. The snippet is below.
最后，我们处于处理流程的最后一步。 至此，我们有了列表面Kong。 现在我们要做的就是将此列表发送到Android客户端，并让其处理对象的构造。 为了发送列表，在我们的GET / data请求处理程序中定义了一个自定义编码方法。 该代码段如下。
def get_mesh(self):
if processor.ready_to_send():
faces = processor.get_faces()
"""Respond to a GET request."""
self.send_response(200)
self.send_header("Content-type", "text")
self.end_headers()
for i in range(len(faces)):
self.wfile.write("N".encode())
self.wfile.write(str(len(faces[i])).encode())
self.wfile.write(",".encode())
for j in range(len(faces[i])):
self.wfile.write(json.dumps(faces[i][j]).encode())
if j != len(faces[i])-1:
self.wfile.write(",".encode())
else:
self.send_response(500)
self.send_header("Content-type", "text")
self.end_headers()
self.wfile.write("Error: Do not have enough pictures to construct 3D object.".encode())The server sends a 200 response code if the side and front sketch have been successfully processed and a 500 if either the side or front face sketch have not been successfully processed. The android application, which will be discussed next, will alert the user if a 500 code has been received.
如果已成功处理了侧面和正面草图，则服务器将发送200响应代码，如果尚未成功处理侧面或正面草图，则服务器将发送500 。 接下来将要讨论的android应用程序将在收到500码后提醒用户。
You did it! That’s all for the server. The detailed implementations of each function discussed can be found in the Github project in the process.py and server.py files. Now, onto the client!
你做到了！ 这就是服务器的全部内容。 可以在Github项目的process.py和server.py文件中找到讨论的每个功能的详细实现。 现在，到客户身上！
第3部分。ARAndroid应用程序 (Section 3. The AR Android Application)
This section covers the basics of the AR android application. The AR application is created in the Unity game engine using the Vuforia developer package to provide some handy AR computer-vision functionalities. The application is written in C# per Unity. The major functions of the AR application are outlined below.
本节介绍AR android应用程序的基础知识。 使用Vuforia开发人员软件包在Unity游戏引擎中创建AR应用程序，以提供一些便捷的AR计算机视觉功能。 该应用程序是根据Unity使用C＃编写的。 AR应用程序的主要功能概述如下。
- Defining an interactive user-interface (buttons to capture the photos, alerts of errors, etc.) 定义交互式用户界面(用于捕获照片的按钮，错误警报等)
- Communication with the server (sending images of the sketch via POST requests and receiving data through GET requests) 与服务器通信(通过POST请求发送草图图像，并通过GET请求接收数据)
- Reconstruction of a 3D object in our mixed reality environment 在我们的混合现实环境中重建3D对象
Note: Part I discussed setting up the AR application, such as defining an image target, defining an empty mesh used to create a 3D object, etc. This part is purely code based; however if the Unity environment was not populated with the correct objects/configurations, this code will not work.
注意：第一部分讨论了如何设置AR应用程序，例如定义图像目标，定义用于创建3D对象的空网格等。 但是，如果未使用正确的对象/配置填充Unity环境，则此代码将不起作用。
案例研究：构建多维数据集 (Case Study: Constructing a Cube)
The user-interface can be split into three parts: when the image target is not detected, when the image target is detected, and when there is an error. Let’s walk through a simple example that hits all of these parts: constructing a cube.
用户界面可以分为三个部分：未检测到图像目标时，何时检测到图像目标时以及出现错误时。 让我们来看一个简单的示例，它涉及所有这些部分：构造一个多维数据集。
To construct a cube, a photo of a square is sent as both the side and front face. Let’s imagine that at first, before collecting either the side and front face, the user points the camera of the phone to the target image. They will be presented with the following alert in Figure 2, informing them that the server does not have enough information to construct the 3D object.
为了构建一个立方体，发送一张正方形的照片作为侧面和正面。 让我们想象一下，首先，在收集侧面和正面之前，用户将手机的相机对准目标图像。 在图2中，将向他们显示以下警报，通知他们服务器没有足够的信息来构造3D对象。
Now, let’s assume that the user then captures an image of the side and the server successfully extracts its four corners. Now, the user wants to take an image of the front square. Figure 3. shows what this UI would look like at this point. The side button is “green” indicating that the side corners are known by the server and the front button is “red” indicating that the user must still capture an image of the front sketch. If the “red” button is clicked in the example UI shown in Figure 3. the contents in the gray-box will be sent to and processed by the server. If the ‘green’ button is clicked, the information defining the front face will be overridden.
现在，假设用户随后捕获了侧面的图像，并且服务器成功提取了其四个角。 现在，用户想要拍摄前广场的图像。 图3.显示了此时的UI。 侧面按钮为“绿色”，表示服务器已知侧面角，而正面按钮为“红色”，表示用户仍必须捕获正面草图的图像。 如果在图3所示的示例UI中单击“红色”按钮，则灰色框中的内容将发送到服务器并由服务器处理。 如果单击“绿色”按钮，则定义正面的信息将被覆盖。
Assuming that both sketches have now been successfully processed, upon detecting the target image, the 3D object will be constructed as shown in Figure 4. The user can then interact with the object in three different ways: scaling, translation, and rotation. For each manipulation, the user specifies which axis (X, Y, or Z) to perform the operation. In the case of scaling, one can upscale or downscale the entire object.
假设现在已经成功处理了两个草图，则在检测到目标图像后，将如图4所示构造3D对象。然后，用户可以通过三种不同方式与对象进行交互：缩放，平移和旋转。 对于每次操作，用户都指定执行该操作的轴(X，Y或Z)。 在缩放的情况下，可以放大或缩小整个对象。
Also, you may have noticed the cube has a blue side! That is due to the annotation recognition system, a feature that will be described in Part 3. More on that later!
另外，您可能已经注意到立方体有蓝色的一面！ 这是由于注释识别系统的原因，该功能将在第3部分中进行描述。
与服务器通讯 (Communication with the Server)
Now that we have covered an example, let’s dive into parts of the client code. As previously mentioned, the client must be able to use a HTTP GET request to receive data from the server and a HTTP POST request to send image data to the server. The code snippet below captures an image of the screen of the AR application and sends it to the server through an HTTP POST.
现在，我们已经涵盖了一个示例，让我们深入研究部分客户端代码。 如前所述，客户端必须能够使用HTTP GET请求从服务器接收数据以及使用HTTP POST请求将图像数据发送到服务器。 下面的代码片段捕获了AR应用程序屏幕的图像，并通过HTTP POST将其发送到服务器。
IEnumerator TakePhoto(string type) // Start this Coroutine on some button click
{
// NOTE - you almost certainly have to do this here:
yield return new WaitForEndOfFrame();
Texture2D tex = new Texture2D(Screen.width, Screen.height,TextureFormat.RGB24, false);
tex.ReadPixels(new Rect(0, 0, Screen.width, Screen.height), 0, 0);
tex.Apply();
//Encode to a PNG
byte[] bytes = tex.EncodeToPNG();
WWWForm form = new WWWForm();
//form.AddField("frameCount", Time.frameCount.ToString());
// form.AddField("Width",Screen.width);
form.AddField("Type",type);
form.AddField("x",bboxX);
form.AddField("y",bboxY);
form.AddField("width",widthTarget);
form.AddField("height",heightTarget);
form.AddBinaryData("file", bytes);
print("sending picture");
// Upload to a cgi script
var w = UnityWebRequest.Post("http://SERVERP:PORT", form);
yield return w.SendWebRequest();
if (w.isNetworkError || w.isHttpError)
print(w.error);
else {
string response = w.downloadHandler.text;
if (response[0] == '1'){
sideCaptured = true;
}
if (response[1] == '1'){
frontCaptured = true;
}
}
yield return null;
// send to server
}The code snippet below performs an HTTP GET /data to receive the list of faces from the server and process the list into a usable format. In addition, if there is an error, then the Button.cs class is called to alert the user, as seen in Figure 2.
下面的代码段执行HTTP GET / data来从服务器接收面Kong列表，并将该列表处理为可用格式。 另外，如果存在错误，则调用Button.cs类以警告用户， 如图2所示。
UnityWebRequest www = UnityWebRequest.Get("http://SERVERIP:PORT/data");
yield return www.SendWebRequest();
Debug.Log(www.responseCode);
if(www.isNetworkError || www.isHttpError || www.responseCode == 500) {
GameObject myObject = GameObject.Find("ARCamera");
myObject.GetComponent().showError("Need to Capture Front and Side!");
yield return new WaitForSeconds(3);
myObject.GetComponent().clearError();
}
else {
// Show results as text
Debug.Log(www.downloadHandler.text);
string inputTextString = www.downloadHandler.text;
int next = 0;
var list = new List>();
while (next != -1){
int start = inputTextString.IndexOf('N',next);
int num_delimiter = inputTextString.IndexOf(',',start);
next = inputTextString.IndexOf('N',start+1);
int num_samples = Int16.Parse(inputTextString.Substring(start+1,num_delimiter-start-1));
string sub_input;
if (next != -1){
sub_input = inputTextString.Substring(num_delimiter+1,next-num_delimiter-1);
} else {
sub_input = inputTextString.Substring(num_delimiter+1,inputTextString.Length-num_delimiter-1);
}
sub_input = sub_input.Replace("[","");
sub_input = sub_input.Replace("]","");
var myList = sub_input.Split(',').Select(Convert.ToSingle).ToList();
myList.Add(Convert.ToSingle(num_samples));
list.Add(myList);
} 构造增强3D对象 (Constructing the Augmented 3D Object)
In general, the AR application performs the following operations to construct the AR object:
通常，AR应用程序执行以下操作来构造AR对象：
- Detection of the image target 检测图像目标
- An HTTP GET /data request HTTP GET /数据请求
- Decoding the list of corners to construct a mesh 解码角列表以构建网格
- Adjustment of the position of the 3D object 调整3D对象的位置
Vuforia is used to detect the target image by overriding the OnTrackableStateChanged() function and monitoring the Status variable as shown below.
Vuforia用于通过覆盖OnTrackableStateChanged()函数并监视Status变量来检测目标图像，如下所示。
public void OnTrackableStateChanged(
TrackableBehaviour.Status previousStatus,
TrackableBehaviour.Status newStatus)
{
if (newStatus == TrackableBehaviour.Status.DETECTED ||
newStatus == TrackableBehaviour.Status.TRACKED ||
newStatus == TrackableBehaviour.Status.EXTENDED_TRACKED)
{
Debug.Log("Detected");
StartCoroutine(GetData());
Debug.Log("Got Data");
StartCoroutine(CreateMesh());
Debug.Log("Got Mesh");
StartCoroutine(UpdatePos());
Debug.Log("Updated Position");
detected = true;
} else {
detected = false;
foreach (Transform child in gameObject.transform) {
GameObject.Destroy(child.gameObject);
}
}
}Once the image target is detected, the data is received using the GetData() function which invokes the HTTP GET /data request. The CreateMesh() and UpdatePos() function add the object to our mixed reality environment and adjust its position so that it is viewable near the image target. Once the image target is no longer detected, any 3D object that was constructed is then destroyed, allowing for the dynamic creation of a new virtual object.
一旦检测到图像目标，就可以使用调用HTTP GET / data请求的GetData()函数接收数据。 CreateMesh()和UpdatePos()函数将对象添加到我们的混合现实环境中，并调整其位置，以使其在图像目标附近可见。 一旦不再检测到图像目标，便会销毁构造的任何3D对象，从而可以动态创建新的虚拟对象。
The most important function arguably is theCreateMesh() function which creates our 3D object. In order to define a 3D face in Unity, 3 things must be specified: the vertices of the face, a set of triangles defining the details of the face, and the normals of the triangles which determine which direction the face is rendered. Luckily, the function Mesh.RecalculateNormals() can be used to calculate the normals after the vertices and triangle properties have been set of our new Mesh() object.
可以说，最重要的功能是创建3D对象的CreateMesh()函数。 为了在Unity中定义3D面部，必须指定3个方面：面部的顶点，一组定义面部细节的三角形以及确定法线渲染方向的三角形法线。 幸运的是，在设置了新的Mesh()对象的顶点和三角形属性后，可以使用函数Mesh.RecalculateNormals()计算法线。
If you remember, each element in our face list contains a set of points that define a particular face by defining its 3D vertices with clockwise ordering. However, to construct the mesh, we also have to specify the triangles and normals. For example, in the case of a square, our face element would have four vertices. We can have at a minimum 2 triangles, defined by splitting the square along one of its diagonal. Then, we have to construct a list of 6 vertices where each pair of 3 vertices is defined in clock-wise order. Each normal can then be inferred based on this ordering. If this sounds like a lot, it is! Let’s simplify it.
记住，面部列表中的每个元素都包含一组点，这些点通过按顺时针顺序定义其3D顶点来定义特定的面部。 但是，要构建网格，我们还必须指定三角形和法线。 例如，在正方形的情况下，我们的面部元素将具有四个顶点。 我们可以至少有2个三角形，这是通过将正方形沿对角线之一分裂而定义的。 然后，我们必须构造一个6个顶点的列表，其中每对3个顶点按顺时针顺序定义。 然后可以根据此顺序推断每个法线。 如果听起来很多，那就是！ 让我们简化一下。
To make life easier, we encapsulate the creation of triangles in a class called Triangulate() which is shown in the code snippet below.
为了使生活更轻松，我们将三角形的创建封装在一个称为Triangulate()的类中，该类在下面的代码片段中显示。
public class Triangulator
{
private List m_points = new List();
public Triangulator (Vector2[] points) {
m_points = new List(points);
}
public int[] Triangulate() {
List indices = new List();
int n = m_points.Count;
if (n < 3)
return indices.ToArray();
int[] V = new int[n];
if (Area() > 0) {
for (int v = 0; v < n; v++)
V[v] = v;
}
else {
for (int v = 0; v < n; v++)
V[v] = (n - 1) - v;
}
int nv = n;
int count = 2 * nv;
for (int v = nv - 1; nv > 2; ) {
if ((count--) <= 0)
return indices.ToArray();
int u = v;
if (nv <= u)
u = 0;
v = u + 1;
if (nv <= v)
v = 0;
int w = v + 1;
if (nv <= w)
w = 0;
if (Snip(u, v, w, nv, V)) {
int a, b, c, s, t;
a = V[u];
b = V[v];
c = V[w];
indices.Add(a);
indices.Add(b);
indices.Add(c);
for (s = v, t = v + 1; t < nv; s++, t++)
V[s] = V[t];
nv--;
count = 2 * nv;
}
}
indices.Reverse();
return indices.ToArray();
}
private float Area () {
int n = m_points.Count;
float A = 0.0f;
for (int p = n - 1, q = 0; q < n; p = q++) {
Vector2 pval = m_points[p];
Vector2 qval = m_points[q];
A += pval.x * qval.y - qval.x * pval.y;
}
return (A * 0.5f);
}
private bool Snip (int u, int v, int w, int n, int[] V) {
int p;
Vector2 A = m_points[V[u]];
Vector2 B = m_points[V[v]];
Vector2 C = m_points[V[w]];
if (Mathf.Epsilon > (((B.x - A.x) * (C.y - A.y)) - ((B.y - A.y) * (C.x - A.x))))
return false;
for (p = 0; p < n; p++) {
if ((p == u) || (p == v) || (p == w))
continue;
Vector2 P = m_points[V[p]];
if (InsideTriangle(A, B, C, P))
return false;
}
return true;
}
private bool InsideTriangle (Vector2 A, Vector2 B, Vector2 C, Vector2 P) {
float ax, ay, bx, by, cx, cy, apx, apy, bpx, bpy, cpx, cpy;
float cCROSSap, bCROSScp, aCROSSbp;
ax = C.x - B.x; ay = C.y - B.y;
bx = A.x - C.x; by = A.y - C.y;
cx = B.x - A.x; cy = B.y - A.y;
apx = P.x - A.x; apy = P.y - A.y;
bpx = P.x - B.x; bpy = P.y - B.y;
cpx = P.x - C.x; cpy = P.y - C.y;
aCROSSbp = ax * bpy - ay * bpx;
cCROSSap = cx * apy - cy * apx;
bCROSScp = bx * cpy - by * cpx;
return ((aCROSSbp >= 0.0f) && (bCROSScp >= 0.0f) && (cCROSSap >= 0.0f));
}
} Triangulate is used to extract the triangle list from a set of 2D points. For each 3D face, we can project it’s points onto two dimensions, extract the triangle list, and then re-project the points back into three dimensions. To do this, we can use createMesh(). Not going to lie, the createMesh() function is heavily involved. The code snippet is below, but we will go through it at a higher level.
Triangulate用于从一组2D点中提取三角形列表。 对于每个3D面，我们可以将其点投影到二维上，提取三角形列表，然后将点重新投影到三个维度上。 为此，我们可以使用createMesh()。 不会说谎，它会大量涉及createMesh()函数。 下面是代码段，但我们将在更高级别上进行介绍。
IEnumerator CreateMesh(){
while(inFirst)
yield return new WaitForSeconds(0.1f);
int meshID = 97;
int cur = 0;
var resources = Resources.FindObjectsOfTypeAll(typeof(Material));
foreach(var face in mList){
int num_of_vertices = Convert.ToInt32(face[face.Count-1]);
// should we make it hollow?
if (mMap["special"] == 'H' && cur < 4){
cur +=2 ;
continue;
}
Vector2[] vertices2D = new Vector2[num_of_vertices];
int j = 0;
Debug.Log(cur);
for (int i = 0; i < face.Count-1; i=i+3){
if (cur < 4){
vertices2D[j] = new Vector2(face[i],face[i+1]);
} else {
vertices2D[j] = new Vector2(face[i],face[i+2]);
}
Debug.Log(vertices2D[j]);
j = j +1;
}
Triangulator tr = new Triangulator(vertices2D);
int[] indices = tr.Triangulate();
Vector3[] vertices = new Vector3[vertices2D.Length];
j =0;
for (int i = 0; i < face.Count-1; i=i+3){
if (cur < 4){
vertices[j] = new Vector3(vertices2D[j].x, vertices2D[j].y,face[i+2]);
} else {
vertices[j] = new Vector3(vertices2D[j].x, face[i+1],vertices2D[j].y);
}
Debug.Log(vertices[j]);
j = j + 1;
}
Mesh msh = new Mesh();
msh.vertices = vertices;
msh.triangles = indices;
msh.RecalculateNormals();
msh.RecalculateBounds();
// Set up game object with mesh;
gameObject = GameObject.Find("newObject");
GameObject child = new GameObject(Char.ToString(Convert.ToChar(meshID)));
MeshRenderer render = child.AddComponent();
if (cur<4){ // get the color of the front
render.material = parseCharacterTexture(mMap["texturefront"]);
render.material.color = parseCharacterColor(mMap["colorfront"]);
} else {
render.material = parseCharacterTexture(mMap["textureside"]);
render.material.color = parseCharacterColor(mMap["colorside"]);
}
child.transform.parent = gameObject.transform;
MeshFilter filter = child.AddComponent();
filter.mesh = msh;
cur = cur + 1;
meshID = meshID + 1;
msh = new Mesh();
msh.vertices = vertices;
msh.triangles = indices;
msh.RecalculateNormals();
msh.RecalculateBounds();
child = new GameObject(Char.ToString(Convert.ToChar(meshID)));
render = child.AddComponent();
if (cur<4){ // get the color of the front
render.material = parseCharacterTexture(mMap["texturefront"]);
render.material.color = parseCharacterColor(mMap["colorfront"]);
} else {
render.material = parseCharacterTexture(mMap["textureside"]);
render.material.color = parseCharacterColor(mMap["colorside"]);
}
child.transform.parent = gameObject.transform;
filter = child.AddComponent();
filter.mesh = msh;
cur = cur + 1;
if (filter != null)
{
Mesh mesh = filter.mesh;
Vector3[] normals = mesh.normals;
for (int i=0;i In our face list, the first 2 elements define the front and back face of our 3D object. As an overhead, I stored the number of vertices as the last element in this list. It could also be calculated by dividing the original list of vertices by 3, but sometimes its hard to remember why you make the choices you do :)
在人脸列表中，前2个元素定义3D对象的正面和背面。 作为开销，我将顶点数存储为此列表中的最后一个元素。 也可以通过将原始顶点列表除以3来计算，但是有时很难记住为什么做出选择：)
The outer most foreach loop goes through each list of vertices in our list of faces. We then construct a list of 2D vertices because our triangulate method only works on 2D objects. The front and back face are projected on the X, Y axis and the side faces on the X, Z axis.
最外面的foreach循环遍历我们的面列表中的每个顶点列表。 然后，我们构建2D顶点列表，因为我们的三角剖分方法仅适用于2D对象。 正面和背面投影在X，Y轴上，侧面投影在X，Z轴上。
This list of 2D vertices is passed into the Triangulator class and the vertices are then re-projected back into 3D. This is done by simply adding the deleted dimension from the previous step.
该2D顶点列表将传递到Triangulator类中，然后将这些顶点重新投影回3D中。 只需添加上一步中已删除的维度即可完成此操作。
Next, an empty mesh is created and the vertices property is assigned to this list of 3D vertices, where the triangles properties is the one gathered from the triangulate() function of the Triangulator instance. We then call RecalculateNormals() and RecalculateBounds() to properly initialize the last parts of the mesh. The second function is to ensure that the bounding volume is correctly computed.
接下来，创建一个空网格，并将vertices属性分配给此3D顶点列表，其中triangles属性是从Triangulator实例的triangulate()函数中收集的一个。 然后，我们调用RecalculateNormals()和RecalculateBounds()来正确初始化网格的最后部分。 第二个功能是确保正确计算边界体积。
To add the 3D object to our virtual scene, we first find the empty mesh asset that was created in Part 1 using the command GameObject.find("newObject") and a new child object is added. A few properties of the child object, including its material and its color, can be changed, but most importantly we must specify this parent-child relationship using child.transform.parent = gameObject.transform.
要将3D对象添加到我们的虚拟场景中，我们首先使用命令GameObject.find("newObject")找到在第1部分中创建的空网格资产，并添加一个新的子对象。 可以更改子对象的一些属性，包括其材质和颜色，但是最重要的是，我们必须使用child.transform.parent = gameObject.transform.指定这种父子关系child.transform.parent = gameObject.transform.
You’ll see that some of the code in the lower-half of createMesh() is repeated, mainly making another mesh with the same coordinates and texture followed by manipulation of that new object’s normals and triangles. You may be wondering what the purpose of this is? In short, its a trick to sacrifice time for simplicity.
您将看到重复了createMesh()的下半部分的一些代码，主要是制作另一个具有相同坐标和纹理的网格，然后操纵该新对象的法线和三角形。 您可能想知道这样做的目的是什么？ 简而言之，这是为了简单而牺牲时间的技巧。
Remember the whole thing about specifying the winding order of vertices for you to be able to view an object in Unity? Well this part of the function makes a copy of the previously calculated mesh, inverts its normals, and reorders the triangles. Thus, we will see the face rendered whether we view it from the front or the back. This is necessary because one of the annotations that will later be incorporated into Sketch3D is the ability to make a hollow object by adding an “H” to the sketch. If we didn’t perform this step of adding an inverted mesh, then it might look like we are missing some of the faces when we look through the hollow object.
还记得关于指定顶点的缠绕顺序以使您能够在Unity中查看对象的全部内容吗？ 函数的这一部分很好地复制了先前计算的网格，反转了其法线，并对三角形进行了重新排序。 因此，无论从正面还是背面观看，我们都会看到渲染的脸部。 这是必要的，因为稍后将被合并到Sketch3D中的注释之一是能够通过向草图添加“ H”来制作空心对象的功能。 如果我们没有执行添加反面网格的步骤，那么当我们看空心物体时，看起来好像缺少了一些面。
And that’s it! In summary, we iterate through the list of faces, create two meshes to represent the front and back of each face, and add the meshes as a child of our otherwise empty 3D object!
就是这样！ 总而言之，我们遍历面列表，创建两个网格以表示每个面的正面和背面，然后将这些网格添加为否则为空的3D对象的子级！
第4章示例创作 (Section 4. Example Creations)
Now that we have it all working, we can deploy it on an Android application and start the server! The Github describes the exact steps for deployment, so I’m just going to show some sample results below.
既然我们一切正常，我们就可以将其部署在Android应用程序上并启动服务器！ Github描述了部署的确切步骤，因此我将在下面显示一些示例结果。
第5节。下一步 (Section 5. What’s Next)
In the final installment of the Sketch3D series, I will show how to add annotations to manipulate the objects that are created. Annotations are a great solution for sketch-based manipulation because they can easily be incorporated into a sketch and can be mapped to complex actions. For example, we can use annotations to manipulate the color of our object as shown in the green pyramid below.
在Sketch3D系列的最后一部分中，我将展示如何添加注释来操纵创建的对象。 注释是基于草图的操作的绝佳解决方案，因为它们可以轻松地合并到草图中并可以映射到复杂的动作。 例如，我们可以使用注释来操纵对象的颜色，如下面的绿色金字塔所示。
Annotations can also be mapped to more complex manipulations, such as hollowing out the object, like the hollowed-out, extruded pentagon below.
注释也可以映射到更复杂的操作，例如将对象挖空，如下面挖空的挤压五边形。
To add this feature, we have to perform annotation detection and classification. However, one apparent issue is that annotation may mess with our corner detection algorithm and affect the integrity of our constructed virtual object. To fix this issue, our mechanism to detect annotations should be highly precise such that the annotation can be removed. To accomplish simultaneous detection, classification, and removal of the annotations a segmentation network is trained to assign each pixel of the sketch a label that corresponds to a predicted annotation. Part 3 will discuss how this annotation segmentation network is trained and deployed in Sketch3D because ultimately, the benefit of sketch-based authoring will be in the ability to quickly customize the 3D virtual objects.
要添加此功能，我们必须执行注释检测和分类。 但是，一个明显的问题是注释可能会干扰我们的角点检测算法并影响我们构造的虚拟对象的完整性。 为了解决此问题，我们检测注解的机制应高度精确，以便可以删除注解。 为了完成对标注的同时检测，分类和删除，训练了一个分割网络，为草图的每个像素分配与预测的标注相对应的标签。 第3部分将讨论如何在Sketch3D中训练和部署此注释分割网络，因为最终基于草图的创作的好处在于能够快速自定义3D虚拟对象。
第六节结束语 (Section 6. Closing Words)
This article discussed the core of Sketch3D and the major components of the client/server architecture that allows for the dynamic creation of 3D virtual content from 2D sketches. If there are any comments, questions, or concerns, feel free to reach out in the comments below. Cheers!
本文讨论了Sketch3D的核心以及客户端/服务器体系结构的主要组件，这些组件允许从2D草图动态创建3D虚拟内容。 如果有任何评论，问题或疑虑，请随时与以下评论联系。 干杯!
翻译自: https://medium.com/swlh/part-2-sketch3d-designing-the-application-3da94f683ee5
小程序sketch