FFmpeg+QSV+SDL2 格式流转说明
最近在做ffmpeg qsv硬解码并使用sdl2显示服务，但是对于创建硬解码器之后，解码及显示流程中数据格式流转还是不清晰。本文可能存在误区，欢迎大家批评指正
1.创建硬解码器
解码第一步需要指定硬解码器
av_hwdevice_ctx_create负责创建一个特殊类型的硬解码器并创建上下文
该函数注释：
/**
* Open a device of the specified type and create an AVHWDeviceContext for it.
*
* This is a convenience function intended to cover the simple cases. Callers
* who need to fine-tune device creation/management should open the device
* manually and then wrap it in an AVHWDeviceContext using
* av_hwdevice_ctx_alloc()/av_hwdevice_ctx_init().
*
* The returned context is already initialized and ready for use, the caller
* should not call av_hwdevice_ctx_init() on it. The user_opaque/free fields of
* the created AVHWDeviceContext are set by this function and should not be
* touched by the caller.
*
* @param device_ctx On success, a reference to the newly-created device context
* will be written here. The reference is owned by the caller
* and must be released with av_buffer_unref() when no longer
* needed. On failure, NULL will be written to this pointer.
* @param type The type of the device to create.
* @param device A type-specific string identifying the device to open.
* @param opts A dictionary of additional (type-specific) options to use in
* opening the device. The dictionary remains owned by the caller.
* @param flags currently unused
*
* @return 0 on success, a negative AVERROR code on failure.
*/
int av_hwdevice_ctx_create(AVBufferRef **device_ctx, enum AVHWDeviceType type,
const char *device, AVDictionary *opts, int flags);第二个参数type可选：
enum AVHWDeviceType {
AV_HWDEVICE_TYPE_NONE,
AV_HWDEVICE_TYPE_VDPAU,
AV_HWDEVICE_TYPE_CUDA,
AV_HWDEVICE_TYPE_VAAPI,
AV_HWDEVICE_TYPE_DXVA2,
AV_HWDEVICE_TYPE_QSV,
AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
AV_HWDEVICE_TYPE_D3D11VA,
AV_HWDEVICE_TYPE_DRM,
AV_HWDEVICE_TYPE_OPENCL,
AV_HWDEVICE_TYPE_MEDIACODEC,
AV_HWDEVICE_TYPE_VULKAN,
};可以看到该值表示硬解码器的类型，与解码后的目标像素格式完全没有关系
这里我们选择AV_HWDEVICE_TYPE_QSV
创建完成后用av_hwdevice_get_hwframe_constraints函数输出硬解支持输出格式
//硬解支持格式
av_hwdevice_get_hwframe_constraints valid_hw_formats support:116 (AV_PIX_FMT_QSV)
//软解支持格式
av_hwdevice_get_hwframe_constraints valid_sw_formats support:23 (AV_PIX_FMT_NV12)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:28 (AV_PIX_FMT_BGRA)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:161 (AV_PIX_FMT_P010LE)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:11 (AV_PIX_FMT_PAL8)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:1 (AV_PIX_FMT_YUYV422)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:195 (AV_PIX_FMT_Y210LE)我们使用avcodec_find_decoder查找解码器时，传入的codeid是27(AV_CODEC_ID_H264)，如果是硬解的话，必须使用avcodec_find_decoder_by_name，因为没有硬解的codeid
enum AVPixelFormat (*get_format)(struct AVCodecContext *s, const enum AVPixelFormat * fmt);
这是一个回调函数，它的作用就是告诉解码器codec自己的目标像素格式是什么。获取到了硬解码器codec可以支持的目标格式之后，就通过这个回调函数告知给codec，示例代码：
引用一段源码:
static enum AVPixelFormat get_format_hwdec(struct AVCodecContext *avctx,
const enum AVPixelFormat *fmt)
{
struct mp_filter *vd = avctx->opaque;
vd_ffmpeg_ctx *ctx = vd->priv;
MP_VERBOSE(vd, "Pixel formats supported by decoder:");
for (int i = 0; fmt[i] != AV_PIX_FMT_NONE; i++)
MP_VERBOSE(vd, " %s", av_get_pix_fmt_name(fmt[i]));
MP_VERBOSE(vd, "\n");
const char *profile = avcodec_profile_name(avctx->codec_id, avctx->profile);
MP_VERBOSE(vd, "Codec profile: %s (0x%x), avctx->codec_id:%d\n", profile ? profile : "unknown",
avctx->profile, avctx->codec_id);
assert(ctx->use_hwdec);
ctx->hwdec_request_reinit |= ctx->hwdec_failed;
ctx->hwdec_failed = false;
//选择目标像素格式
enum AVPixelFormat select = AV_PIX_FMT_NONE;
for (int i = 0; fmt[i] != AV_PIX_FMT_NONE; i++) {
if (ctx->hwdec.pix_fmt == fmt[i]) {
if (init_generic_hwaccel(vd, fmt[i]) < 0)
break;
select = fmt[i];
break;
}
}
if (select == AV_PIX_FMT_NONE) {
ctx->hwdec_failed = true;
select = avcodec_default_get_format(avctx, fmt);
}
const char *name = av_get_pix_fmt_name(select);
MP_VERBOSE(vd, "Requesting pixfmt '%s' from decoder. select:%d\n", name ? name : "-", select);
return select;
}
fmt是这个解码器codec支持的像素格式，且按照质量优劣进行排序.我的环境中像素格式支持情况：
codec pixfmt[0]:23
codec pixfmt[1]:161
codec pixfmt[2]:116
如果没有特别的需要，这个步骤是可以省略的。内部默认会使用“native”的格式
2.创建SDL Texture
SDL_Texture *p_texture = SDL_CreateTexture(p_renderer,SDL_PIXELFORMAT_NV12,SDL_TEXTUREACCESS_STREAMING,pCodecCtx->width,pCodecCtx->height);
SDL_PIXELFORMAT_NV12此处format格式最好使用和frame的format对应的格式。免去了格式转换的工作，效率会提升很多
示例代码为ffmpeg上解码模块和SDL显示模块format对照表
static const struct TextureFormatEntry {
enum AVPixelFormat format;
int texture_fmt;
} sdl_texture_format_map[] = {
{ AV_PIX_FMT_RGB8, SDL_PIXELFORMAT_RGB332 },
{ AV_PIX_FMT_RGB444, SDL_PIXELFORMAT_RGB444 },
{ AV_PIX_FMT_RGB555, SDL_PIXELFORMAT_RGB555 },
{ AV_PIX_FMT_BGR555, SDL_PIXELFORMAT_BGR555 },
{ AV_PIX_FMT_RGB565, SDL_PIXELFORMAT_RGB565 },
{ AV_PIX_FMT_BGR565, SDL_PIXELFORMAT_BGR565 },
{ AV_PIX_FMT_RGB24, SDL_PIXELFORMAT_RGB24 },
{ AV_PIX_FMT_BGR24, SDL_PIXELFORMAT_BGR24 },
{ AV_PIX_FMT_0RGB32, SDL_PIXELFORMAT_RGB888 },
{ AV_PIX_FMT_0BGR32, SDL_PIXELFORMAT_BGR888 },
{ AV_PIX_FMT_NE(RGB0, 0BGR), SDL_PIXELFORMAT_RGBX8888 },
{ AV_PIX_FMT_NE(BGR0, 0RGB), SDL_PIXELFORMAT_BGRX8888 },
{ AV_PIX_FMT_RGB32, SDL_PIXELFORMAT_ARGB8888 },
{ AV_PIX_FMT_RGB32_1, SDL_PIXELFORMAT_RGBA8888 },
{ AV_PIX_FMT_BGR32, SDL_PIXELFORMAT_ABGR8888 },
{ AV_PIX_FMT_BGR32_1, SDL_PIXELFORMAT_BGRA8888 },
{ AV_PIX_FMT_YUV420P, SDL_PIXELFORMAT_IYUV },
{ AV_PIX_FMT_YUYV422, SDL_PIXELFORMAT_YUY2 },
{ AV_PIX_FMT_UYVY422, SDL_PIXELFORMAT_UYVY },
{ AV_PIX_FMT_NONE, SDL_PIXELFORMAT_UNKNOWN },
3.小结
如果创建解码器时不指定get_format的格式，系统一般会默认选取支持的pix_fmt第一个作为frame的format。创建显示模块时可以根据frame的format进行动态创建texture