ARKit学习-7

转载请注明出处
Apple原文地址: https://developer.apple.com/documentation/arkit/displaying_an_ar_experience_with_metal

---

Displaying an AR Experience with Metal(使用 Metal 来展示 AR 场景)

Build a custom AR view by rendering camera images and using position-tracking information to display overlay content.
通过渲染相机图像，以及使用位置追踪信息来展示覆盖(overlay)物，从而来构建自定义的 AR 视图场景。

---

Overview

• ARKit includes view classes for easily displaying AR experiences with SceneKit or SpriteKit. However, if you instead build your own rendering engine (or integrate with a third-party engine), ARKit also provides all the support necessary to display an AR experience with a custom view.

• ARKit 中内置了一些视图类，从而能够轻松地用 SceneKit 或者 SpriteKit 来展示 AR 场景。然而，如果您使用的是自己的渲染引擎（或者集成了第三方引擎），那么 ARKit 还提供了自定义视图以及其他的支持环境，来展示 AR 场景。

image.png

• In any AR experience, the first step is to configure an ARSession
  object to manage camera capture and motion processing. A session defines and maintains a correspondence between the real-world space the device inhabits and a virtual space where you model AR content. To display your AR experience in a custom view, you’ll need to:

• 在所有的 AR 场景中，首先就是要配置一个 ARSession**对象，用来管理摄像头拍摄和对设备动作进行处理。Session 定义并维护现实空间和虚拟空间之间的关联关系，其中，现实空间是用户所处的世界，虚拟空间是可对可视化内容进行建模的世界。如果要在自定义视图当中展示您的 AR 场景的话，那么您需要：

• 1.Retrieve video frames and tracking information from the session.

• 1.从 Session 中检索视频帧和追踪信息

• 2.Render those frame images as the backdrop for your view.

• 2.将这些帧图像作为背景，渲染到自定义视图当中

• 3.Use the tracking information to position and draw AR content atop the camera image.

• 3.使用追踪信息，从而在相机图像上方定位并绘制 AR 内容

Note/注意
This article covers code found in Xcode project templates. For complete example code, create a new iOS application with the Augmented Reality template, and choose Metal from the Content Technology popup menu.
本文所涉及的代码均可以在 Xcode 项目模板当中找到。如果要获取完整的示例代码，请使用 “Augmented Reality” 模板来创建一个新的 iOS 应用，然后在弹出的 Content Technology 菜单当中选择 “Metal”

Get Video Frames and Tracking Data from the Session(从 Session 中获取视频帧和追踪数据)

• Create and maintain your own ARSession
  instance, and run it with a session configuration appropriate for the kind of AR experience you want to support. (To do this, see Building a Basic AR Experience.) The session captures video from the camera, tracks the device’s position and orientation in a modeled 3D space, and provides ARFrame
  objects. Each such object contains both an individual video frame image and position tracking information from the moment that frame was captured.

• 请自行创建并维护 ARSession实例，然后根据您所希望提供的 AR 场景类型，使用合适的 Session 配置来运行这个实例。（要实现这点的话，请参阅「构建基本的 AR 场景」。）Session 从摄像机当中捕获视频，然后在建模的 3D 空间中追踪设备的位置和方向，并提供 ARFrame对象。每个 ARFrame对象都包含有单独的视频帧 (frame) 图像和被捕获时的设备位置追踪信息。

• There are two ways to access ARFrame
  objects produced by an AR session, depending on whether your app favors a pull or a push design pattern.

• 要访问 AR Session 中生成的 ARFrame对象的话，有以下两种方法，使用何种方法取决于您应用的设计模式是偏好主动拉取 (pull) 还是被动推送 (push)。

• If you prefer to control frame timing (the pull design pattern), use the session’s currentFrame
  property to get the current frame image and tracking information each time you redraw your view’s contents. The ARKit Xcode template uses this approach:

• 如果您倾向于定时获取视频帧的话（也就是主动拉取设计模式），那么请使用 Session 的 currentFrame属性，这样就可以在每次重绘视图内容的时候，获取当前的帧图像和追踪信息。ARKit Xcode 模板使用了如下方法：

// in Renderer class, called from MTKViewDelegate.draw(in:) via Renderer.update()
func updateGameState() {        
    guard let currentFrame = session.currentFrame else {
        return
    }
    
    updateSharedUniforms(frame: currentFrame)
    updateAnchors(frame: currentFrame)
    updateCapturedImageTextures(frame: currentFrame)
    
    if viewportSizeDidChange {
        viewportSizeDidChange = false
        
        updateImagePlane(frame: currentFrame)
    }
}

• Alternatively, if your app design favors a push pattern, implement the session(_:didUpdate:)
  delegate method, and the session will call it once for each video frame it captures (at 60 frames per second by default).

• 相反，如果您的应用设计倾向于使用被动推送模式的话，那么请实现session(_:didUpdate:)代理方法，当每个视频帧被捕获之后，Session 就会调用这个代理方法（默认每秒捕获 60 帧）。

• Upon obtaining a frame, you’ll need to draw the camera image, and update and render any overlay content your AR experience includes.

• 获得一个视频帧之后，您就需要绘制相机图像了，然后将 AR 场景中包含的所有覆盖物进行更新和展示。

Draw the Camera Image(绘制相机图像)

• Each ARFrame
  object’s capturedImage
  property contains a pixel buffer captured from the device camera. To draw this image as the backdrop for your custom view, you’ll need to create textures from the image content and submit GPU rendering commands that use those textures.

• 每个 ARFrame对象的 capturedImage属性都包含了从设备相机中捕获的像素缓冲区 (pixel buffer)。要将这个图像作为背景绘制到自定义视图当中，您需要从图像内容中构建纹理 (texture)，然后提交使用这些纹理进行 GPU 渲染的命令。

• The pixel buffer’s contents are encoded in a biplanar YCbCr (also called YUV) data format; to render the image you’ll need to convert this pixel data to a drawable RGB format. For rendering with Metal, you can perform this conversion most efficiently in GPU shader code. Use CVMetalTextureCache APIs to create two Metal textures from the pixel buffer—one each for the buffer’s luma (Y) and chroma (CbCr) planes:

• 像素缓冲区的内容将被编码为双面 (biplanar) YCbCr 数据格式（也成为 YUV）；要渲染图像的话，您需要将这些像素数据转换为可绘制的 RGB 格式。对于 Metal 渲染而言，最高效的方法便是使用 GPU 着色代码 (shader code) 来执行这个转换了。借助CVMetalTextureCache**API，可以从像素缓冲区中生成两个 Metal 纹理——一个用于决定缓冲区的亮度 (Y)，一个用于决定缓冲区的色度 (CbCr) 面。

func updateCapturedImageTextures(frame: ARFrame) {
    // Create two textures (Y and CbCr) from the provided frame's captured image
    //从所提供的视频帧中，根据其中所捕获的图像，创建两个纹理 (Y and CbCr)
    let pixelBuffer = frame.capturedImage
    if (CVPixelBufferGetPlaneCount(pixelBuffer) < 2) {
        return
    }
    capturedImageTextureY = createTexture(fromPixelBuffer: pixelBuffer, pixelFormat:.r8Unorm, planeIndex:0)!
    capturedImageTextureCbCr = createTexture(fromPixelBuffer: pixelBuffer, pixelFormat:.rg8Unorm, planeIndex:1)!
}

func createTexture(fromPixelBuffer pixelBuffer: CVPixelBuffer, pixelFormat: MTLPixelFormat, planeIndex: Int) -> MTLTexture? {
    var mtlTexture: MTLTexture? = nil
    let width = CVPixelBufferGetWidthOfPlane(pixelBuffer, planeIndex)
    let height = CVPixelBufferGetHeightOfPlane(pixelBuffer, planeIndex)
    
    var texture: CVMetalTexture? = nil
    let status = CVMetalTextureCacheCreateTextureFromImage(nil, capturedImageTextureCache, pixelBuffer, nil, pixelFormat, width, height, planeIndex, &texture)
    if status == kCVReturnSuccess {
        mtlTexture = CVMetalTextureGetTexture(texture!)
    }
    
    return mtlTexture
}

• Next, encode render commands that draw those two textures using a fragment function that performs YCbCr to RGB conversion with a color transform matrix:

• 接下来，使用借助颜色变换矩阵将 YCbCr 转换为 RGB 的函数片段，完成这两个纹理的绘制，我们这里将整个渲染命令进行编码。

fragment float4 capturedImageFragmentShader(ImageColorInOut in [[stage_in]],
                                            texture2d capturedImageTextureY [[ texture(kTextureIndexY) ]],
                                            texture2d capturedImageTextureCbCr [[ texture(kTextureIndexCbCr) ]]) {
    
    constexpr sampler colorSampler(mip_filter::linear,
                                   mag_filter::linear,
                                   min_filter::linear);
    
    const float4x4 ycbcrToRGBTransform = float4x4(
        float4(+1.164380f, +1.164380f, +1.164380f, +0.000000f),
        float4(+0.000000f, -0.391762f, +2.017230f, +0.000000f),
        float4(+1.596030f, -0.812968f, +0.000000f, +0.000000f),
        float4(-0.874202f, +0.531668f, -1.085630f, +1.000000f)
    );
    
    // Sample Y and CbCr textures to get the YCbCr color at the given texture coordinate
    float4 ycbcr = float4(capturedImageTextureY.sample(colorSampler, in.texCoord).r,
                          capturedImageTextureCbCr.sample(colorSampler, in.texCoord).rg, 1.0);
    
    // Return converted RGB color
    return ycbcrToRGBTransform * ycbcr;
}

Note / 注意

• Use the displayTransform(withViewportSize:orientation:)
  method to make sure the camera image covers the entire view. For example use of this method, as well as complete Metal pipeline setup code, see the full Xcode template. (Create a new iOS application with the Augmented Reality template, and choose Metal from the Content Technology popup menu.)
• 请使用 displayTransform(withViewportSize:orientation:)方法来确保整个相机图像完全覆盖了整个视图。关于如何使用这个方法，以及完整的 Metal 管道配置代码，请参阅完整的 Xcode 模板。（请使用 “Augmented Reality” 模板来创建一个新的 iOS 应用，然后在弹出的 Content Technology 菜单当中选择 “Metal”。）

Track and Render Overlay Content(追踪并渲染覆盖内容)

• AR experiences typically focus on rendering 3D overlay content so that the content appears to be part of the real world seen in the camera image. To achieve this illusion, use the ARAnchor
  class to model the position and orientation of your own 3D content relative to real-world space. Anchors provide transforms that you can reference during rendering.

• AR 场景通常侧重于渲染 3D 覆盖物，使得这些内容似乎是从相机中所看到的真实世界的一部分。为了实现这种效果，我们使用 ARAnchor**类，来对 3D 内容相对于现实世界空间的位置和方向进行建模。锚点提供了变换 (transform) 属性，在渲染的时候可供参考。

• For example, the Xcode template creates an anchor located about 20 cm in front of the device whenever a user taps on the screen:

• 举个例子，当用户点击屏幕的时候，Xcode 模板会在设备前方大约 20 厘米处，创建一个锚点。

func handleTap(gestureRecognize: UITapGestureRecognizer) {
    // Create anchor using the camera's current position
    if let currentFrame = session.currentFrame {
        
        // Create a transform with a translation of 0.2 meters in front of the camera
        var translation = matrix_identity_float4x4
        translation.columns.3.z = -0.2
        let transform = simd_mul(currentFrame.camera.transform, translation)
        
        // Add a new anchor to the session
        let anchor = ARAnchor(transform: transform)
        session.add(anchor: anchor)
    }
}

• In your rendering engine, use the transform
  property of each ARAnchor
  object to place visual content. The Xcode template uses each of the anchors added to the session in its handleTap
  method to position a simple cube mesh:

• 在您的渲染引擎当中，使用每个 ARAnchor对象的 transform属性来放置虚拟内容。Xcode 模板在内部的 handleTap方法中，使用添加到 Session 当中每个锚点来定位一个简单的立方体网格 (cube mesh)：

func updateAnchors(frame: ARFrame) {
    // Update the anchor uniform buffer with transforms of the current frame's anchors
    anchorInstanceCount = min(frame.anchors.count, kMaxAnchorInstanceCount)
    
    var anchorOffset: Int = 0
    if anchorInstanceCount == kMaxAnchorInstanceCount {
        anchorOffset = max(frame.anchors.count - kMaxAnchorInstanceCount, 0)
    }
    
    for index in 0..

Note / 注意

• In a more complex AR experience, you can use hit testing or plane detection to find the positions of real-world surfaces. For details, see the planeDetection
  property and the hitTest(_:types:)
  method. In both cases, ARKit provides results as ARAnchor
  objects, so you still use anchor transforms to place visual content.
• 在更为复杂的 AR 场景中，您可以使用点击测试或者水平面检测，来寻找真实世界当中曲面的位置。要了解关于此内容的详细信息，请参阅 planeDetection属性和hitTest(_:types:)方法。对于这两者而言，ARKit 都会生成 ARAnchor对象作为结果，因此您仍然需要使用锚点的 transform 属性来放置虚拟内容。

Render with Realistic Lighting(根据实际光照度进行渲染)

• When you configure shaders for drawing 3D content in your scene, use the estimated lighting information in each ARFrame
  object to produce more realistic shading:

• 当您在场景中配置用于绘制 3D 内容的着色器时，请使用每个 ARFrame**对象当中的预计光照度信息，来产生更为逼真的阴影：

// in Renderer.updateSharedUniforms(frame:):
// Set up lighting for the scene using the ambient intensity if provided
var ambientIntensity: Float = 1.0
if let lightEstimate = frame.lightEstimate {
    ambientIntensity = Float(lightEstimate.ambientIntensity) / 1000.0
}
let ambientLightColor: vector_float3 = vector3(0.5, 0.5, 0.5)
uniforms.pointee.ambientLightColor = ambientLightColor * ambientIntensity

Note / 注意

• For the complete set of Metal setup and rendering commands that go with this example, see the see the full Xcode template. (Create a new iOS application with the Augmented Reality template, and choose Metal from the Content Technology popup menu.)
• 要了解该示例中的全部 Metal 配置，以及所使用的渲染命令，请参见完整的 Xcode 模板。（请使用 “Augmented Reality” 模板来创建一个新的 iOS 应用，然后在弹出的 Content Technology 菜单当中选择 “Metal”。