视频I/O硬件加速

Video I/O hardware acceleration

视频I/O硬件加速

• Introduction
• Software prerequisites
• Hardware prerequisites
  • Intel hardware
  • AMD hardware
  • NVIDIA hardware
• Installation BKC
  • Installation BKC on Ubuntu 20.04 (Intel CPU with HD Graphics)
  • Installation BKC on Windows
• Samples and benchmarks
• Troubleshooting
  • FFmpeg
  • GStreamer

Introduction

介绍

Since OpenCV 4.5.2 new properties are added to control H/W acceleration modes for video decoding and encoding tasks. New builtin properties brings easy to use API for OpenCV Users.

由于OpenCV 4.5.2增加了新的属性来控制视频解码和编码任务的H/W加速模式。新的内置属性为OpenCV用户带来了易于使用的API。

VideoCapture capture(filename, CAP_FFMPEG,
    {
        CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY,
    }
);

Hardware-accelerated API is described in OpenCV API documentation.

​OpenCV API文档中描述了硬件加速API。

 This feature currently is in preview mode, so feel free to try it with your configuration and provide feedback through OpenCV issues.

​该功能目前处于预览模式，因此可以在配置中尝试，并通过OpenCV问题提供反馈。

Software prerequisites

软件先决条件

OpenCV uses external Media I/O libraries and/or OS-provided APIs under unified VideoCapture and VideoWriter APIs. Wrapper code in OpenCV over some external framework is called backend.

​OpenCV在统一的VideoCapture和VideoWriter API下使用外部媒体I/O库或操作系统提供的API。OpenCV中在外部框架上的包装代码称为后端。

H/W accelerated processing is supported through these libraries:

通过以下库支持H/W加速处理：

• FFmpeg 4.0+ with enabled hwaccels support: HWAccelIntro – FFmpeg

• ​支持hwaccels的FFmpeg 4.0+：HWAccelIntro–FFmpeg

• GStreamer 1.x+ with installed VAAPI plugin and others.

• ​安装了VAAPI插件和其他插件的GStreamer 1.x+。

• (Windows) Microsoft Media Foundation (MSMF)

• （Windows）微软媒体基金会（MSMF）

To properly utilize hardware capabilities there are several low-level API/SDKs used:

为了正确利用硬件功能，使用了几种低级API/SDK：

• (Linux) VA-API (Video Acceleration API) is an open-source library and API specification, which provides access to graphics hardware acceleration capabilities for video processing.

• ​（Linux）VA-API（视频加速API）是一个开源库和API规范，它提供了对视频处理的图形硬件加速功能的访问。

• (Windows) DirectX Video Acceleration (DXVA/DXVA2) is a Microsoft API that allows video decoding/encoding to be hardware-accelerated.

• ​（Windows）DirectX视频加速（DXVA/DXVA2）是一种Microsoft API，允许对视频解码/编码进行硬件加速。

• Intel® Media SDK provides an API to access hardware-accelerated video decode, encode and filtering on Intel® platforms with integrated graphics. oneVPL (oneAPI Video Processing Library) is an evolution of Intel Media SDK.

• “英特尔®媒体SDK”提供了一个API，可在具有集成图形的英特尔®平台上访问硬件加速的视频解码、编码和过滤。oneVPL（oneAPI视频处理库）是Intel Media SDK的一种演变。

• NVIDIA Video Codec SDK is a NVIDIA proprietary library for hardware-accelerated video decode/encode on CUDA-compatible GPUs.

• NVIDIA Video Codec SDK是一个NVIDIA专有库，用于在兼容CUDA的GPU上进行硬件加速视频解码/编码。

• Video Decode and Presentation API for Unix (VDPAU) is an open source library and API to offload portions of the video decoding process and video post-processing to the GPU video-hardware, developed by NVIDIA.

• Unix视频解码和演示API（VDPAU）是一个开源库和API，用于将部分视频解码过程和视频后处理转移到由NVIDIA开发的GPU视频硬件。

• AMD AMF

• others APIs/SDK from hardware vendors

• 其他来自硬件供应商的API/SDK

Installation guidelines for some Best Known Configurations (BKC) are described below.

下面介绍一些最知名配置（BKC）的安装指南。

Current HW acceleration types support matrix, in priority order:

当前硬件加速类型支持列表，按优先级顺序：

OS	Backend	VideoCapture	VideoWriter
Linux	FFMPEG	VAAPI	MFX, VAAPI
	GStreamer	VAAPI (and others HW plugins)	VAAPI (and others HW plugins)
Windows	FFMPEG	D3D11	MFX
	MSMF	D3D11	-

Hardware prerequisites

硬件先决条件

Hardware-accelerated decoding/encoding requires capable hardware.

硬件加速解码/编码需要有能力的硬件。

Intel hardware

英特尔硬件

You can check H/W support matrix on these resources:

您可以查看以下资源的H/W支持列表：

• Intel Media Driver page which provides VAAPI support on Linux
• 在Linux上提供VAAPI支持的英特尔媒体驱动程序页
• Wiki page about Intel Quick Sync Video.
• 关于英特尔快速同步视频的Wiki页面。

AMD hardware

AMD硬件

You can check H/W support matrix on these resources:

您可以查看以下资源的H/W支持列表：

• Wiki page about AMD Video Core Next
• 关于AMD Video Core的Wiki页面下一页

NVIDIA hardware

英伟达硬件

You can check H/W support matrix on these resources:

您可以查看以下资源的H/W支持列表：

• Wiki page about NVIDIA NVDEC
• 关于NVIDIA NVDEC的维基页面
• Wiki page about NVIDIA NVENC
• 关于NVIDIA NVENC的维基页面
• NVIDIA Video Codec SDK
• NVIDIA SDK视频编解码器

Installation BKC

安装BKC

Check these resources about installation of media libraries:

查看以下有关媒体库安装的资源：

• FFmpeg
• GStreamer
• Microsoft Media Foundation (MSMF) runtime is usually already preinstalled on Windows (except some "Base" editions, which are widely used in Docker Windows images)
• ​Microsoft Media Foundation（MSMF）运行时通常已经预装在Windows上（除了一些在Docker Windows映像中广泛使用的“基本”版本）

Installation BKC on Ubuntu 20.04 (Intel CPU with HD Graphics)

在Ubuntu 20.04上安装BKC（带高清图形的英特尔CPU）

Install these packages:

安装以下软件包：

• VAAPI: apt-get install libva-dev vainfo
• FFmpeg: apt-get install ffmpeg libavcodec-dev libavformat-dev libswscale-dev
• GStreamer and its plugins:

  apt-get install --no-install-recommends \
    libgstreamer1.0-0 libgstreamer1.0-dev \
    libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev \
    gstreamer1.0-plugins-base gstreamer1.0-plugins-bad gstreamer1.0-libav gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-ugly gstreamer1.0-vaapi gstreamer1.0-tools
  

• Media SDK packages: apt-get install libmfx-dev libmfx-tools

After installation of the packages above you need to rebuild OpenCV from scratch (clean build directory). You should see these entries in CMake summary log:

​安装上述软件包后，您需要从头开始重建OpenCV（clean构建目录）。您应该在CMake摘要日志中看到以下条目：

--   Video I/O:
...
--     FFMPEG:                      YES
--       avcodec:                   YES (58.54.100)
--       avformat:                  YES (58.29.100)
--       avutil:                    YES (56.31.100)
--       swscale:                   YES (5.5.100)
--       avresample:                YES (4.0.0)
--     GStreamer:                   YES (1.16.2)

Install full-feature VAAPI driver for Intel hardware:

为英特尔硬件安装全功能VAAPI驱动程序：

apt-get install intel-media-va-driver-non-free

This package installs VAAPI driver with support for both HW decode and encode, and automatically uninstalls package 'intel-media-va-driver' (which supports HW decode only) if was installed previously as dependency of other packages.

此软件包安装支持硬件解码和编码的VAAPI驱动程序，如果以前作为其他软件包的依赖项安装了“intel-media-va-driver”（仅支持硬件解码），则会自动卸载该软件包。

Correct installation should output something like this for vainfo call (CPU: Intel i5-6600 (Skylake)):

正确的安装应该为vainfo call（CPU:Intel i5-6600（Skylake））输出如下内容：

libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   :	VAEntrypointVideoProc
      VAProfileNone                   :	VAEntrypointStats
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Simple            :	VAEntrypointEncSlice
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointFEI
      VAProfileH264Main               :	VAEntrypointEncSliceLP
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointFEI
      VAProfileH264High               :	VAEntrypointEncSliceLP
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:	VAEntrypointFEI
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointFEI

Note: There are several VAAPI drivers for Intel hardware: i965 and iHD. There is strong recommendation to use iHD version (mandatory for modern hardware).

注意：英特尔硬件有几个VAAPI驱动程序：i965和iHD。强烈建议使用iHD版本（现代硬件必须使用）。

Installation BKC on Windows

在Windows上安装BKC

Media decoders/encoders runtimes are usually a part of Graphics Drivers Software on Windows.

媒体解码器/编码器运行时通常是Windows上图形驱动程序软件的一部分。

Dedicated SDKs may be required if you want to rebuild customized versions of FFmpeg/GStreamer.

如果要重建自定义版本的FFmpeg/GStreamer，可能需要专用SDK。

This section is not complete

本节不完整

Environment variables

环境变量

Environment variable OPENCV_FFMPEG_CAPTURE_OPTIONS allows to experiment with acceleration types other than D3D11VA/VAAPI/MFX in VideoCapture/VideoWriter APIs with FFMPEG backend implementation. For example, to use VAAPI and VDPAU acceleration (in priority order) in VideoCapture, open VideoCapture with parameters '{ CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY }' and set environment variable

环境变量OPENCV_FFMPEG_CAPTURE_OPTIONS允许在视频捕获/视频编写器API中使用除D3D11VA/VAAPI/MFX之外的加速类型，并使用FFMPEG后端实现。例如，要在视频捕获中使用VAAPI和VDPAU加速度（按优先级顺序），请使用参数“{CAP_PROP_HW_ACCELERATION，VIDEO_ACCELERATION_ANY}”打开VideoCapture并设置环境变量

OPENCV_FFMPEG_CAPTURE_OPTIONS="hw_decoders_any;vaapi,vdpau"

To use NVENC/CUDA acceleration in VideoWriter, open VideoWriter with parameters '{ VIDEOWRITER_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY }' and set environment variable

要在VideoWriter中使用NVENC/CUDA加速，请使用参数“{VIDEOWRITER_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_ANY }”打开VideoWriter并设置环境变量

OPENCV_FFMPEG_WRITER_OPTIONS="hw_encoders_any;cuda"

Acceleration naming in these environment variables follows FFMpeg convension.

​这些环境变量中的加速命名遵循FFMpeg约定。

Samples and benchmarks

示例和基准

1. samples/tapi/video_acceleration.cpp
2. samples/cpp/videocapture_gstreamer_pipeline.cpp Wiki page
3. Media SDK backend Wiki page

Troubleshooting

疑难解答

Before reporting the problem please collect information about:

在报告问题之前，请收集以下信息：

• used video stream, including information about used codec (see below, use FFmpeg or GStreamer native tools)
• 使用过的视频流，包括有关使用过的编解码器的信息（请参阅下文，使用FFmpeg或GStreamer本机工具）
• installed packages: apt list --installed | grep -e va-driver -e mfx -e ffmpeg -e libva -e opencl -e intel-media -e gstreamer -e i965 (use yum list installed on Fedora/CentOS/RedHat)
• 已安装的软件包：apt list --installed | grep -e va-driver -e mfx -e ffmpeg -e libva -e opencl -e intel-media -e gstreamer -e i965（在Fedora/CentOS/RedHat上使用yum list installed）
• dump of vainfo command
• vainfo命令的转储

FFmpeg

Run ffplay / ffmpeg on the media stream with issues. Try to run with or without the -hwaccel option.

在有问题的媒体流上运行ffplay/ffmpeg。尝试使用或不使用-hwaccel选项运行。

Use ffprobe -show_streams  to extract information about the video stream

使用ffprobe-show_streams提取有关视频流的信息

GStreamer

Use gst-launch utility to check media stream with issues.

​使用gst launch实用程序检查有问题的媒体流。

Use GST_DEBUG environment variable to see extra messages from GStreamer.

使用GST_DEBUG环境变量查看来自GStreamer的额外消息。

© Copyright 2022, OpenCV team

©版权所有2022，OpenCV团队