python-ue4-metahuman-nerf:我创造了一个数字人!!

原文:https://zhuanlan.zhihu.com/p/561152870

目录

收起

1. 准备工作:制作 MetaHuman 角色

1.1 创建 MetaHuman 角色

1.2 Quixel Bridge 下载 MetaHuman

1.3 导出 MetaHuman 到 UE4

2. UE 渲染 MetaHuman 的多视点图片

2.1 如何在 UE 中手动渲染视频?

2.2 Python 自动化渲染

2.3 全部代码

3. 渲染结果

随着 ECCV 2020 NeRF 的出现,在计算机视觉、计算机图形学、三维重建等领域砸开了无数的坑位,一篇 NeRF 养活了多少科研工作者、大厂员工。但是要想训练这样一个 Neural Radiance Filed 却不是一件容易的事情,首先从数据来看,训练一个 NeRF 需要该场景的多视点图片,同时还需要知道每张图片所对应的相机参数,要想得到这样的数据,可谓是耗时耗力。那么,有没有一种方式,可以虚拟的仿真出一组 NeRF 的训练数据?

答案当然是有的,而且不止一种方式。下面本文将分享一种使用 UE4 来渲染带有相机参数的多视点图片的方法。在正式开始之前,首先来介绍一下 MetaHuman,它就是我们要渲染的对象。在计算机视觉、三维重建领域,人脸方向的研究一直是一个热点,而 MetaHuman 则是一个高保真的数字人开源框架,我们可以利用 MetaHuman 来设计出自己想要的数字人形象,然后渲染出其多视点图片,用作进一步的研究。

1. 准备工作:制作 MetaHuman 角色

在 UE4 中渲染 MetaHuman 之前,我们首先需要创造一个 Metahuman 角色,然后并将其导入到 UE4 中进行渲染,整体可分为以下几个步骤

  1. 使用 MetaHuman Creator 创建 MetaHuman 角色;也可以不创建新角色,直接使用自带的即可。
  2. 借助 Quixel Bridge 工具将创建的 MetaHuman 角色下载下来。
  3. 在 Quixel Bridge 中将 MetaHuman 角色导出到 UE4中。

下面分别对这几步进行说明:

1.1 创建 MetaHuman 角色

登入 MetaHuman Creator 官网:https://metahuman.unrealengine.com/,会看到如下界面,左侧的 CREATE METAHUMAN 是系统默认给出的 Metahuman 角色,可以直接在其基础上进行修改,创作一个新的 Metahuman,这会存在左侧的 MY METAHUMAN 目录中。

python-ue4-metahuman-nerf:我创造了一个数字人!!_第1张图片

具体的操作界面如下,整体分为三部分:脸部、毛发、身体。我们可以对这三部分中的各个细节进行操作,大家可以自行尝试~

python-ue4-metahuman-nerf:我创造了一个数字人!!_第2张图片

1.2 Quixel Bridge 下载 MetaHuman

我们刚刚创作好的 MetaHuman 角色是存在于官方的服务器上的,在对它进行操作之前,还需要借助 Quixel Bridge 工具将其下载下来。

Quixel Bridge 下载链接:https://quixel.com/bridge

在下载时候,我们使用 Epic Games 账号,后续用的也一直是这个账号。下载完成并登录之后,我们会看到如下界面

python-ue4-metahuman-nerf:我创造了一个数字人!!_第3张图片

左侧导航栏中的 MetaHuman Presets 和 My MetaHuman 分别表示默认角色和我们前一步创建的角色。点击右上角绿色的下载按钮即可下载到本地。

1.3 导出 MetaHuman 到 UE4

在导出 MetaHuman 角色到 UE4 之前,我们需要配置:导出的目标引擎是什么、引擎的版本号、引擎的位置。如下图所示

python-ue4-metahuman-nerf:我创造了一个数字人!!_第4张图片

之后,点击对应 MetaHuman 角色右上角的 Export 按钮即可将其导出到 UE 引擎中(注意:在导出时,要保证 UE 是打开的)。

2. UE 渲染 MetaHuman 的多视点图片

在利用 Python 自动化渲染符合要求的多视点图片之前,我们首先要搞明白的一件事情是:如何手动的在 UE4 中渲染视频序列(视频和图片本质上是一样的,图片就是视频一帧一帧截出来的)?

2.1 如何在 UE 中手动渲染视频?

step1:将 MetaHuman 导入到 UE4 的场景中,并将其定位到合适的位置。在场景中添加电影摄像机 CineCameraActor, 如下:

python-ue4-metahuman-nerf:我创造了一个数字人!!_第5张图片

step2:创建过场动画,并添加关卡序列,如下

python-ue4-metahuman-nerf:我创造了一个数字人!!_第6张图片

step3:依次点击,轨道 — Actor到Sequencer — CineCameraActor,创建 CineCameraActor 的轨道序列,如下

python-ue4-metahuman-nerf:我创造了一个数字人!!_第7张图片

step4:在创建好的 CineCameraActor 轨道序列中添加关键帧(下图添加了 5 个关键帧,分别指定了相机的位置和旋转),使得相机按照既定方式运动,如下

python-ue4-metahuman-nerf:我创造了一个数字人!!_第8张图片

step5:至此,视频序列创建完毕,点击 将影片渲染为视频序列或图片序列,完成导出

python-ue4-metahuman-nerf:我创造了一个数字人!!_第9张图片

上述就是在 UE4 中渲染 MetaHuman 的整个过程,只需要鼠标点点就行了,比较简单。但是有一个严重的问题是:如果要渲染一个多视点 MetaHuman 数据集用作 NeRF 训练的话,我们是必须要知道相机的姿态的;而在上述过程中,我们只能准确的知道关键帧的相机姿态,其他帧的相机姿态都是被插值出来的。如果要手动定义数百上千个关键帧,又是相当繁琐的一项工作,因此我们下面考虑利用 python 来自动化的执行上述过程,自动化的定义关键帧,指定相机的运动。

2.2 Python 自动化渲染

首先,总结一下前一章节手动渲染的过程,整体上可以分为如下 4 个步骤:

  1. 导入 MetaHuman 角色,并将其定位的合适位置
  2. 创建关卡序列,并创建 CineCameraActor 轨道序列
  3. 设置相机参数,并指定相机的位置和旋转,定义相机的运动方式
  4. 导出渲染的视频、图片序列

上述 4 个步骤中,除了步骤一还需要我们手动操作以外,其余步骤全部都可以利用 python 脚本来实现。在 UE4 中使用 python 接口还需要进行一些简单的配置,具体操作可以参考:https://www.bilibili.com/video/BV1PE411d7z8

步骤一:角色导入以及人脸定位

由于我只想渲染 MetaHuman 的人脸区域,因此我需要将其余身体部分移动到相机视野之外。需要移动的身体部位有:Torso、Legs、Feet。

接着进行 MetaHuman 的定位,为了方便,我们可以将其 Face 位置定位在坐标原点(0,0,0),这是为了方便后面指定相机的位置与旋转。

步骤二:指定相机运动方式:位置和旋转

在这一步,我希望相机能够以 MetaHuman 的人脸(也就是坐标原点)为中心,做球形旋转运动。关于相机的外参,做如下设置:

  • 相机到坐标原点的距离(即:球半径)为35
  • 相机的俯仰角设置为如下角度:[-30, -15, 0, 15, 30]
  • 在每个俯仰角下,偏航角区间设置为:[-180, 180], 每间隔角度 10 定义一个关键点(即:拍摄一张照片)

渲染的相关的设置如下:

  • 相机的视场角设置为:100
  • 所渲染图片的分辨率:512
  • 帧与帧之间的延迟:1.5

相机内参的设置见:步骤三具体的渲染代码。

现在,我们将相机的运动数据(相机位置和旋转)、渲染的相关设置,存为一个 .json 文件,在步骤三渲染的时候直接读取这些数据即可。代码如下:

import json
import numpy as np

# 得到相机的运动数据(位置、旋转)
def get_camera_transform():
    radius = 35
    camara_transform = []

    for y_pitch in range(-30, 31, 15):
        for z_yaw in range(-180, 181, 10):
            location_x = radius * np.cos(z_yaw / 180 * np.pi) * np.cos(y_pitch / 180 * np.pi)
            location_y = radius * np.sin(z_yaw / 180 * np.pi) * np.cos(y_pitch / 180 * np.pi)
            location_z = -radius * np.sin(y_pitch / 180 * np.pi)

            print([-location_x, -location_y, location_z, 0, y_pitch, z_yaw])
            camara_transform.append([-location_x, -location_y, location_z, 0, y_pitch, z_yaw])

    return camara_transform

# 将基础数据存为一个 .json 文件
def make_json():
    camara_transform = get_camera_transform()

    data_dict = {
        f"film_fov": 100,
        "film_resolution": 512,
        "delay_every_frame": 1.5,
        "output_image_path": "F:/projects/Unreal projects/MetaHumanRender/RenderImages/Danielle/face_hair",
        "camera_data": {
            "camera_qRrq9DqY": {
                "type": "Camera",
                "camera_transform": camara_transform
            }
        }
    }

    data_json = json.dumps(data_dict, indent=4)
    with open("camera_data.json", 'w') as json_file:
        json_file.write(data_json)


if __name__ == '__main__':
    make_json()

步骤三:读取相机运动数据,创建关卡序列添加关键帧

首先,读取上一节生成的.json文件,获取相机运动数据

def read_json_file(path):
    file = open(path, "rb")
    file_json = json.load(file)

    export_path = file_json['output_image_path']  # image export path
    film_fov = file_json.get('film_fov', None)  # camera film fov
    film_resolution = file_json.get("film_resolution", 512)  # camera file resolution
    delay_every_frame = file_json.get("delay_every_frame", 3.0)  # sequence run delay every frame
    camera_json = file_json['camera_data']  # camera data

    camera_transform_array = []
    camera_rotation_array = []
    camera_name_array = []
    for camera in camera_json:
        camera_transform = camera_json[camera]["camera_transform"]
        for index in range(len(camera_transform)):

            camera_transform_array.append(unreal.Transform(
                location=[camera_transform[index][0], camera_transform[index][1], camera_transform[index][2]],
                rotation=[camera_transform[index][3], camera_transform[index][4], camera_transform[index][5]],
                scale=[1.0, 1.0, 1.0]
            ))
            camera_rotation_array.append(unreal.Rotator(
                roll=camera_transform[index][3],
                pitch=camera_transform[index][4],
                yaw=camera_transform[index][5]
            ))
        camera_name_array.append(camera)

    return camera_transform_array, camera_rotation_array, camera_name_array, export_path, film_fov, film_resolution, delay_every_frame

接着创建关卡序列,并添加关键帧,每一个相机姿态对应一个关键帧。在这一部分,做要做了如下几个事情,具体细节可参考代码

  • 在场景中创建一个电影摄像机 CineCameraActor,并指定其位置和旋转。
  • 设置胶片或数字传感器的尺寸。
  • 相机的镜头的参数设置:最小焦距、最大焦距等。
  • 聚焦设置:焦距、光圈、视场角等。
  • 相机的绑定,相机的关键帧添加,对每一个相机姿态添加一个关键帧。

代码

def create_sequence(asset_name, camera_transform_array, camera_rotation_array, camera_name_array, film_fov,
                    package_path='/Game/'):
    # create a sequence
    sequence = unreal.AssetToolsHelpers.get_asset_tools().create_asset(asset_name, package_path, unreal.LevelSequence,
                                                                       unreal.LevelSequenceFactoryNew())
    sequence.set_display_rate(unreal.FrameRate(numerator=1, denominator=1))
    sequence.set_playback_start_seconds(0)
    sequence.set_playback_end_seconds(len(camera_transform_array))

    # add a camera cut track
    camera_cut_track = sequence.add_master_track(unreal.MovieSceneCameraCutTrack)
    print(camera_transform_array[0], '=======================================================================')
    print(camera_rotation_array[0].get_editor_property("roll"))

    # 添加关键帧
    for index in range(0, len(camera_name_array)):
        camera_cut_section = camera_cut_track.add_section()
        camera_cut_section.set_start_frame_bounded(6 * index)
        camera_cut_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
        camera_cut_section.set_start_frame_seconds(6 * index)
        camera_cut_section.set_end_frame_seconds(6 * index + len(camera_transform_array))

        # Create a cine camera actor
        # 在世界编辑器中创建一个电影摄像机CineCameraActor,并指定位置和旋转
        camera_actor = unreal.EditorLevelLibrary().spawn_actor_from_class(unreal.CineCameraActor,
                                                                          unreal.Vector(0, 0, 0),
                                                                          unreal.Rotator(0, 0, 0))
        # 指定电影摄像机CineCameraActor的名称
        camera_actor.set_actor_label(camera_name_array[index])
        # 返回CineCamera组件
        camera_component = camera_actor.get_cine_camera_component()
        ratio = math.tan(film_fov / 360.0 * math.pi) * 2

        # step1:胶片背板设置
        filmback = camera_component.get_editor_property("filmback")
        # sensor_height:胶片或数字传感器的垂直尺寸,以毫米为单位
        filmback.set_editor_property("sensor_height", 60)
        # sensor_width:胶片或数字传感器的水平尺寸,以毫米为单位
        filmback.set_editor_property("sensor_width", 60)

        # step2:镜头设置
        lens_settings = camera_component.get_editor_property("lens_settings")
        lens_settings.set_editor_property("min_focal_length", 4.0)
        lens_settings.set_editor_property("max_focal_length", 1000)
        lens_settings.set_editor_property("min_f_stop", 1.2)
        lens_settings.set_editor_property("max_f_stop", 500)
        print(lens_settings)

        # step3:聚焦设置
        focal_length = 45  # 焦距
        current_aperture = 500  # 光圈
        camera_component.set_editor_property("current_focal_length", focal_length)  # 设置当前焦距
        camera_component.set_editor_property("current_aperture", current_aperture)  # 设置当前光圈
        fov = camera_component.get_editor_property("field_of_view")
        print(camera_component.get_editor_property("current_focal_length"))
        print(camera_component.get_editor_property("current_aperture"))
        print(f"fov - {fov}")

        # add a binding for the camera
        camera_binding = sequence.add_possessable(camera_actor)
        transform_track = camera_binding.add_track(unreal.MovieScene3DTransformTrack)
        transform_section = transform_track.add_section()
        transform_section.set_start_frame_bounded(6 * index)
        transform_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
        transform_section.set_start_frame_seconds(6 * index)
        transform_section.set_end_frame_seconds(6 * index + len(camera_transform_array))

        # get channel for location_x location_y location_z
        channel_location_x = transform_section.get_channels()[0]
        channel_location_y = transform_section.get_channels()[1]
        channel_location_z = transform_section.get_channels()[2]

        # get key for rotation_y rotation_z
        channel_rotation_x = transform_section.get_channels()[3]
        channel_rotation_y = transform_section.get_channels()[4]
        channel_rotation_z = transform_section.get_channels()[5]

        # camera_transform = camera_transform_array[index]
        # camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
        # camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
        # camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")

        for i in range(len(camera_transform_array)):
            camera_transform = camera_transform_array[i]
            camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
            camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
            camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")
            camera_rotation = camera_rotation_array[i]
            camera_rotate_roll = camera_rotation.get_editor_property("roll")
            camera_rotate_pitch = camera_rotation.get_editor_property("pitch")
            camera_rotate_yaw = camera_rotation.get_editor_property("yaw")

            new_time = unreal.FrameNumber(value=i)
            channel_location_x.add_key(new_time, camera_location_x, 0.0)
            channel_location_y.add_key(new_time, camera_location_y, 0.0)
            channel_location_z.add_key(new_time, camera_location_z, 0.0)
            channel_rotation_x.add_key(new_time, camera_rotate_roll, 0.0)
            channel_rotation_y.add_key(new_time, camera_rotate_pitch, 0.0)
            channel_rotation_z.add_key(new_time, camera_rotate_yaw, 0.0)

            # if i % 5 == 0:
            #     print(new_time, [camera_location_x, camera_location_y, camera_location_z, camera_rotate_roll, camera_rotate_pitch, camera_rotate_yaw])


        # add the binding for the camera cut section
        camera_binding_id = sequence.make_binding_id(camera_binding, unreal.MovieSceneObjectBindingSpace.LOCAL)
        camera_cut_section.set_camera_binding_id(camera_binding_id)

    # save sequence asset
    unreal.EditorAssetLibrary.save_loaded_asset(sequence, False)

    return sequence

步骤四:将关卡序列渲染为视频、图片序列

将创建的关卡序列渲染为电影序列,这部分主要就是设置一些渲染参数,对应手动渲染的 step5,具体参数的含义可参考 unreal 官方文档。代码如下:

# sequence movie
def render_sequence_to_movie(export_path, film_resolution, delay_every_frame, on_finished_callback):
    # 1) Create an instance of our UAutomatedLevelSequenceCapture and override all of the settings on it. This class is currently
    # set as a config class so settings will leak between the Unreal Sequencer Render-to-Movie UI and this object. To work around
    # this, we set every setting via the script so that no changes the user has made via the UI will affect the script version.
    # The users UI settings will be reset as an unfortunate side effect of this.
    capture_settings = unreal.AutomatedLevelSequenceCapture()

    # Set all POD settings on the UMovieSceneCapture
    # capture_settings.settings.output_directory = unreal.DirectoryPath("../../../unreal-render-release/Saved/VideoCaptures1/")
    capture_settings.settings.output_directory = unreal.DirectoryPath(export_path)

    # If you game mode is implemented in Blueprint, load_asset(...) is going to return you the C++ type ('Blueprint') and not what the BP says it inherits from.
    # Instead, because game_mode_override is a TSubclassOf we can use unreal.load_class to get the UClass which is implicitly convertable.
    # ie: capture_settings.settings.game_mode_override = unreal.load_class(None, "/Game/AI/TestingSupport/AITestingGameMode.AITestingGameMode_C")
    capture_settings.settings.game_mode_override = None
    capture_settings.settings.output_format = "{camera}" + "_{frame}"
    capture_settings.settings.overwrite_existing = False
    capture_settings.settings.use_relative_frame_numbers = False
    capture_settings.settings.handle_frames = 0
    capture_settings.settings.zero_pad_frame_numbers = 4
    # If you wish to override the output framerate you can use these two lines, otherwise the framerate will be derived from the sequence being rendered
    capture_settings.settings.use_custom_frame_rate = False
    # capture_settings.settings.custom_frame_rate = unreal.FrameRate(24,1)
    capture_settings.settings.resolution.res_x = film_resolution
    capture_settings.settings.resolution.res_y = film_resolution
    capture_settings.settings.enable_texture_streaming = False
    capture_settings.settings.cinematic_engine_scalability = True
    capture_settings.settings.cinematic_mode = True
    capture_settings.settings.allow_movement = False  # Requires cinematic_mode = True
    capture_settings.settings.allow_turning = False  # Requires cinematic_mode = True
    capture_settings.settings.show_player = False  # Requires cinematic_mode = True
    capture_settings.settings.show_hud = False  # Requires cinematic_mode = True
    capture_settings.use_separate_process = False
    capture_settings.close_editor_when_capture_starts = False  # Requires use_separate_process = True
    capture_settings.additional_command_line_arguments = "-NOSCREENMESSAGES"  # Requires use_separate_process = True
    capture_settings.inherited_command_line_arguments = ""  # Requires use_separate_process = True

    # Set all the POD settings on UAutomatedLevelSequenceCapture
    capture_settings.use_custom_start_frame = False  # If False, the system will automatically calculate the start based on sequence content
    capture_settings.use_custom_end_frame = False  # If False, the system will automatically calculate the end based on sequence content
    capture_settings.custom_start_frame = unreal.FrameNumber(0)  # Requires use_custom_start_frame = True
    capture_settings.custom_end_frame = unreal.FrameNumber(0)  # Requires use_custom_end_frame = True
    capture_settings.warm_up_frame_count = 0
    capture_settings.delay_before_warm_up = 0
    capture_settings.delay_before_shot_warm_up = 0
    capture_settings.delay_every_frame = delay_every_frame
    capture_settings.write_edit_decision_list = True

    # Tell the capture settings which level sequence to render with these settings. The asset does not need to be loaded,
    # as we're only capturing the path to it and when the PIE instance is created it will load the specified asset.
    # If you only had a reference to the level sequence, you could use "unreal.SoftObjectPath(mysequence.get_path_name())"
    capture_settings.level_sequence_asset = unreal.SoftObjectPath(sequence_asset_path)

    # To configure the video output we need to tell the capture settings which capture protocol to use. The various supported
    # capture protocols can be found by setting the Unreal Content Browser to "Engine C++ Classes" and filtering for "Protocol"
    # ie: CompositionGraphCaptureProtocol, ImageSequenceProtocol_PNG, etc. Do note that some of the listed protocols are not intended
    # to be used directly.
    # Right click on a Protocol and use "Copy Reference" and then remove the extra formatting around it. ie:
    # Class'/Script/MovieSceneCapture.ImageSequenceProtocol_PNG' gets transformed into "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"
    capture_settings.set_image_capture_protocol_type(
        unreal.load_class(None, "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"))

    # After we have set the capture protocol to a soft class path we can start editing the settings for the instance of the protocol that is internallyc reated.
    capture_settings.get_image_capture_protocol().compression_quality = 85

    # Finally invoke Sequencer's Render to Movie functionality. This will examine the specified settings object and either construct a new PIE instance to render in,
    # or create and launch a new process (optionally shutting down your editor).
    unreal.SequencerTools.render_movie(capture_settings, on_finished_callback)

2.3 全部代码

import sys

import unreal, os, json, math

# sequence asset path
sequence_asset_path = '/Game/Render_Sequence.Render_Sequence'


# read json file
def read_json_file(path):
    file = open(path, "rb")
    file_json = json.load(file)

    export_path = file_json['output_image_path']  # image export path
    film_fov = file_json.get('film_fov', None)  # camera film fov
    film_resolution = file_json.get("film_resolution", 512)  # camera file resolution
    delay_every_frame = file_json.get("delay_every_frame", 3.0)  # sequence run delay every frame
    camera_json = file_json['camera_data']  # camera data

    camera_transform_array = []
    camera_rotation_array = []
    camera_name_array = []
    for camera in camera_json:
        camera_transform = camera_json[camera]["camera_transform"]
        for index in range(len(camera_transform)):

            camera_transform_array.append(unreal.Transform(
                location=[camera_transform[index][0], camera_transform[index][1], camera_transform[index][2]],
                rotation=[camera_transform[index][3], camera_transform[index][4], camera_transform[index][5]],
                scale=[1.0, 1.0, 1.0]
            ))
            camera_rotation_array.append(unreal.Rotator(
                roll=camera_transform[index][3],
                pitch=camera_transform[index][4],
                yaw=camera_transform[index][5]
            ))
        camera_name_array.append(camera)

    return camera_transform_array, camera_rotation_array, camera_name_array, export_path, film_fov, film_resolution, delay_every_frame


def create_sequence(asset_name, camera_transform_array, camera_rotation_array, camera_name_array, film_fov,
                    package_path='/Game/'):
    # create a sequence
    sequence = unreal.AssetToolsHelpers.get_asset_tools().create_asset(asset_name, package_path, unreal.LevelSequence,
                                                                       unreal.LevelSequenceFactoryNew())
    sequence.set_display_rate(unreal.FrameRate(numerator=1, denominator=1))
    sequence.set_playback_start_seconds(0)
    sequence.set_playback_end_seconds(len(camera_transform_array))

    # add a camera cut track
    camera_cut_track = sequence.add_master_track(unreal.MovieSceneCameraCutTrack)
    print(camera_transform_array[0], '=======================================================================')
    print(camera_rotation_array[0].get_editor_property("roll"))

    # 添加关键帧
    for index in range(0, len(camera_name_array)):
        camera_cut_section = camera_cut_track.add_section()
        camera_cut_section.set_start_frame_bounded(6 * index)
        camera_cut_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
        camera_cut_section.set_start_frame_seconds(6 * index)
        camera_cut_section.set_end_frame_seconds(6 * index + len(camera_transform_array))

        # Create a cine camera actor
        # 在世界编辑器中创建一个电影摄像机CineCameraActor,并指定位置和旋转
        camera_actor = unreal.EditorLevelLibrary().spawn_actor_from_class(unreal.CineCameraActor,
                                                                          unreal.Vector(0, 0, 0),
                                                                          unreal.Rotator(0, 0, 0))
        # 指定电影摄像机CineCameraActor的名称
        camera_actor.set_actor_label(camera_name_array[index])
        # 返回CineCamera组件
        camera_component = camera_actor.get_cine_camera_component()
        ratio = math.tan(film_fov / 360.0 * math.pi) * 2

        # step1:胶片背板设置
        filmback = camera_component.get_editor_property("filmback")
        # sensor_height:胶片或数字传感器的垂直尺寸,以毫米为单位
        filmback.set_editor_property("sensor_height", 60)
        # sensor_width:胶片或数字传感器的水平尺寸,以毫米为单位
        filmback.set_editor_property("sensor_width", 60)

        # step2:镜头设置
        lens_settings = camera_component.get_editor_property("lens_settings")
        lens_settings.set_editor_property("min_focal_length", 4.0)
        lens_settings.set_editor_property("max_focal_length", 1000)
        lens_settings.set_editor_property("min_f_stop", 1.2)
        lens_settings.set_editor_property("max_f_stop", 500)
        print(lens_settings)

        # step3:聚焦设置
        focal_length = 45  # 焦距
        current_aperture = 500  # 光圈
        camera_component.set_editor_property("current_focal_length", focal_length)  # 设置当前焦距
        camera_component.set_editor_property("current_aperture", current_aperture)  # 设置当前光圈
        fov = camera_component.get_editor_property("field_of_view")
        print(camera_component.get_editor_property("current_focal_length"))
        print(camera_component.get_editor_property("current_aperture"))
        print(f"fov - {fov}")

        # add a binding for the camera
        camera_binding = sequence.add_possessable(camera_actor)
        transform_track = camera_binding.add_track(unreal.MovieScene3DTransformTrack)
        transform_section = transform_track.add_section()
        transform_section.set_start_frame_bounded(6 * index)
        transform_section.set_end_frame_bounded(6 * index + len(camera_transform_array))
        transform_section.set_start_frame_seconds(6 * index)
        transform_section.set_end_frame_seconds(6 * index + len(camera_transform_array))

        # get channel for location_x location_y location_z
        channel_location_x = transform_section.get_channels()[0]
        channel_location_y = transform_section.get_channels()[1]
        channel_location_z = transform_section.get_channels()[2]

        # get key for rotation_y rotation_z
        channel_rotation_x = transform_section.get_channels()[3]
        channel_rotation_y = transform_section.get_channels()[4]
        channel_rotation_z = transform_section.get_channels()[5]

        # camera_transform = camera_transform_array[index]
        # camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
        # camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
        # camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")

        for i in range(len(camera_transform_array)):
            camera_transform = camera_transform_array[i]
            camera_location_x = camera_transform.get_editor_property("translation").get_editor_property("x")
            camera_location_y = camera_transform.get_editor_property("translation").get_editor_property("y")
            camera_location_z = camera_transform.get_editor_property("translation").get_editor_property("z")
            camera_rotation = camera_rotation_array[i]
            camera_rotate_roll = camera_rotation.get_editor_property("roll")
            camera_rotate_pitch = camera_rotation.get_editor_property("pitch")
            camera_rotate_yaw = camera_rotation.get_editor_property("yaw")

            new_time = unreal.FrameNumber(value=i)
            channel_location_x.add_key(new_time, camera_location_x, 0.0)
            channel_location_y.add_key(new_time, camera_location_y, 0.0)
            channel_location_z.add_key(new_time, camera_location_z, 0.0)
            channel_rotation_x.add_key(new_time, camera_rotate_roll, 0.0)
            channel_rotation_y.add_key(new_time, camera_rotate_pitch, 0.0)
            channel_rotation_z.add_key(new_time, camera_rotate_yaw, 0.0)

            # if i % 5 == 0:
            #     print(new_time, [camera_location_x, camera_location_y, camera_location_z, camera_rotate_roll, camera_rotate_pitch, camera_rotate_yaw])


        # add the binding for the camera cut section
        camera_binding_id = sequence.make_binding_id(camera_binding, unreal.MovieSceneObjectBindingSpace.LOCAL)
        camera_cut_section.set_camera_binding_id(camera_binding_id)

    # save sequence asset
    unreal.EditorAssetLibrary.save_loaded_asset(sequence, False)

    return sequence


# sequence movie
def render_sequence_to_movie(export_path, film_resolution, delay_every_frame, on_finished_callback):
    # 1) Create an instance of our UAutomatedLevelSequenceCapture and override all of the settings on it. This class is currently
    # set as a config class so settings will leak between the Unreal Sequencer Render-to-Movie UI and this object. To work around
    # this, we set every setting via the script so that no changes the user has made via the UI will affect the script version.
    # The users UI settings will be reset as an unfortunate side effect of this.
    capture_settings = unreal.AutomatedLevelSequenceCapture()

    # Set all POD settings on the UMovieSceneCapture
    # capture_settings.settings.output_directory = unreal.DirectoryPath("../../../unreal-render-release/Saved/VideoCaptures1/")
    capture_settings.settings.output_directory = unreal.DirectoryPath(export_path)

    # If you game mode is implemented in Blueprint, load_asset(...) is going to return you the C++ type ('Blueprint') and not what the BP says it inherits from.
    # Instead, because game_mode_override is a TSubclassOf we can use unreal.load_class to get the UClass which is implicitly convertable.
    # ie: capture_settings.settings.game_mode_override = unreal.load_class(None, "/Game/AI/TestingSupport/AITestingGameMode.AITestingGameMode_C")
    capture_settings.settings.game_mode_override = None
    capture_settings.settings.output_format = "{camera}" + "_{frame}"
    capture_settings.settings.overwrite_existing = False
    capture_settings.settings.use_relative_frame_numbers = False
    capture_settings.settings.handle_frames = 0
    capture_settings.settings.zero_pad_frame_numbers = 4
    # If you wish to override the output framerate you can use these two lines, otherwise the framerate will be derived from the sequence being rendered
    capture_settings.settings.use_custom_frame_rate = False
    # capture_settings.settings.custom_frame_rate = unreal.FrameRate(24,1)
    capture_settings.settings.resolution.res_x = film_resolution
    capture_settings.settings.resolution.res_y = film_resolution
    capture_settings.settings.enable_texture_streaming = False
    capture_settings.settings.cinematic_engine_scalability = True
    capture_settings.settings.cinematic_mode = True
    capture_settings.settings.allow_movement = False  # Requires cinematic_mode = True
    capture_settings.settings.allow_turning = False  # Requires cinematic_mode = True
    capture_settings.settings.show_player = False  # Requires cinematic_mode = True
    capture_settings.settings.show_hud = False  # Requires cinematic_mode = True
    capture_settings.use_separate_process = False
    capture_settings.close_editor_when_capture_starts = False  # Requires use_separate_process = True
    capture_settings.additional_command_line_arguments = "-NOSCREENMESSAGES"  # Requires use_separate_process = True
    capture_settings.inherited_command_line_arguments = ""  # Requires use_separate_process = True

    # Set all the POD settings on UAutomatedLevelSequenceCapture
    capture_settings.use_custom_start_frame = False  # If False, the system will automatically calculate the start based on sequence content
    capture_settings.use_custom_end_frame = False  # If False, the system will automatically calculate the end based on sequence content
    capture_settings.custom_start_frame = unreal.FrameNumber(0)  # Requires use_custom_start_frame = True
    capture_settings.custom_end_frame = unreal.FrameNumber(0)  # Requires use_custom_end_frame = True
    capture_settings.warm_up_frame_count = 0
    capture_settings.delay_before_warm_up = 0
    capture_settings.delay_before_shot_warm_up = 0
    capture_settings.delay_every_frame = delay_every_frame
    capture_settings.write_edit_decision_list = True

    # Tell the capture settings which level sequence to render with these settings. The asset does not need to be loaded,
    # as we're only capturing the path to it and when the PIE instance is created it will load the specified asset.
    # If you only had a reference to the level sequence, you could use "unreal.SoftObjectPath(mysequence.get_path_name())"
    capture_settings.level_sequence_asset = unreal.SoftObjectPath(sequence_asset_path)

    # To configure the video output we need to tell the capture settings which capture protocol to use. The various supported
    # capture protocols can be found by setting the Unreal Content Browser to "Engine C++ Classes" and filtering for "Protocol"
    # ie: CompositionGraphCaptureProtocol, ImageSequenceProtocol_PNG, etc. Do note that some of the listed protocols are not intended
    # to be used directly.
    # Right click on a Protocol and use "Copy Reference" and then remove the extra formatting around it. ie:
    # Class'/Script/MovieSceneCapture.ImageSequenceProtocol_PNG' gets transformed into "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"
    capture_settings.set_image_capture_protocol_type(
        unreal.load_class(None, "/Script/MovieSceneCapture.ImageSequenceProtocol_PNG"))

    # After we have set the capture protocol to a soft class path we can start editing the settings for the instance of the protocol that is internallyc reated.
    capture_settings.get_image_capture_protocol().compression_quality = 85

    # Finally invoke Sequencer's Render to Movie functionality. This will examine the specified settings object and either construct a new PIE instance to render in,
    # or create and launch a new process (optionally shutting down your editor).
    unreal.SequencerTools.render_movie(capture_settings, on_finished_callback)


# sequence end call back
def on_render_movie_finished(success):
    unreal.log('on_render_movie_finished is called')
    if success:
        unreal.log('---end LevelSequenceTask')


# check asset is exist
def check_sequence_asset_exist(root_dir):
    sequence_path = os.path.join(root_dir, 'Content', 'Render_Sequence.uasset')
    if (os.path.exists(sequence_path)):
        unreal.log('---sequence is exist os path---')
        os.remove(sequence_path)
    else:
        unreal.log('---sequence is not exist os path---')

    if unreal.EditorAssetLibrary.does_asset_exist(sequence_asset_path):
        unreal.log('---sequence dose asset exist---')
        unreal.EditorAssetLibrary.delete_asset(sequence_asset_path)
    else:
        unreal.log('---sequence does not asset exist---')

    sequence_asset_data = unreal.AssetRegistryHelpers.get_asset_registry().get_asset_by_object_path(sequence_asset_path)
    if sequence_asset_data:
        unreal.log('---sequence is exist on content---')
        sequence_asset = unreal.AssetData.get_asset(sequence_asset_data)
        unreal.EditorAssetLibrary.delete_loaded_asset(sequence_asset)
    else:
        unreal.log('---sequence is not exit on content---')


def main():
    # bind event
    on_finished_callback = unreal.OnRenderMovieStopped()
    on_finished_callback.bind_callable(on_render_movie_finished)

    # get json file
    root_dir = unreal.SystemLibrary.get_project_directory()
    json_path = os.path.join(root_dir, 'RawData', 'camera_data.json')

    # get json data
    camera_transform_array, camera_rotation_array, camera_name_array, export_path, film_fov, film_resolution, delay_every_frame = read_json_file(
        json_path)

    # check sequence is asset
    check_sequence_asset_exist(root_dir)

    # create sequence
    create_sequence('Render_Sequence', camera_transform_array, camera_rotation_array, camera_name_array, film_fov)

    # run sequence
    render_sequence_to_movie(export_path, film_resolution, delay_every_frame, on_finished_callback)

3. 渲染结果

python-ue4-metahuman-nerf:我创造了一个数字人!!_第10张图片

上图即为渲染的多视点图片,每张图片对应的相机姿态也是确定的。

至此,在 UE4 中利用 python 实现自动化渲染的整个过程就结束了。在这里留下一个小问题:

UE4 中的相机姿态数据我们是无法直接使用的,如何将其转化为 NeRF 中的相机姿态数据呢?大家可以自行探索一下,后续文章再对这一部分的内容进行讲解。

你可能感兴趣的:(深度学习,图象处理,python,人工智能,深度学习)