FFmpeg+QSV+SDL2 格式流转说明

最近在做ffmpeg qsv硬解码并使用sdl2显示服务，但是对于创建硬解码器之后，解码及显示流程中数据格式流转还是不清晰。本文可能存在误区，欢迎大家批评指正

1.创建硬解码器

解码第一步需要指定硬解码器

av_hwdevice_ctx_create负责创建一个特殊类型的硬解码器并创建上下文

该函数注释：

/**
 * Open a device of the specified type and create an AVHWDeviceContext for it.
 *
 * This is a convenience function intended to cover the simple cases. Callers
 * who need to fine-tune device creation/management should open the device
 * manually and then wrap it in an AVHWDeviceContext using
 * av_hwdevice_ctx_alloc()/av_hwdevice_ctx_init().
 *
 * The returned context is already initialized and ready for use, the caller
 * should not call av_hwdevice_ctx_init() on it. The user_opaque/free fields of
 * the created AVHWDeviceContext are set by this function and should not be
 * touched by the caller.
 *
 * @param device_ctx On success, a reference to the newly-created device context
 *                   will be written here. The reference is owned by the caller
 *                   and must be released with av_buffer_unref() when no longer
 *                   needed. On failure, NULL will be written to this pointer.
 * @param type The type of the device to create.
 * @param device A type-specific string identifying the device to open.
 * @param opts A dictionary of additional (type-specific) options to use in
 *             opening the device. The dictionary remains owned by the caller.
 * @param flags currently unused
 *
 * @return 0 on success, a negative AVERROR code on failure.
 */
int av_hwdevice_ctx_create(AVBufferRef **device_ctx, enum AVHWDeviceType type,
                           const char *device, AVDictionary *opts, int flags);

第二个参数type可选：

enum AVHWDeviceType {
    AV_HWDEVICE_TYPE_NONE,
    AV_HWDEVICE_TYPE_VDPAU,
    AV_HWDEVICE_TYPE_CUDA,
    AV_HWDEVICE_TYPE_VAAPI,
    AV_HWDEVICE_TYPE_DXVA2,
    AV_HWDEVICE_TYPE_QSV,
    AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
    AV_HWDEVICE_TYPE_D3D11VA,
    AV_HWDEVICE_TYPE_DRM,
    AV_HWDEVICE_TYPE_OPENCL,
    AV_HWDEVICE_TYPE_MEDIACODEC,
    AV_HWDEVICE_TYPE_VULKAN,
};

可以看到该值表示硬解码器的类型，与解码后的目标像素格式完全没有关系

这里我们选择AV_HWDEVICE_TYPE_QSV

创建完成后用av_hwdevice_get_hwframe_constraints函数输出硬解支持输出格式

//硬解支持格式
av_hwdevice_get_hwframe_constraints valid_hw_formats support:116 (AV_PIX_FMT_QSV)
//软解支持格式
av_hwdevice_get_hwframe_constraints valid_sw_formats support:23 (AV_PIX_FMT_NV12)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:28 (AV_PIX_FMT_BGRA)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:161 (AV_PIX_FMT_P010LE)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:11 (AV_PIX_FMT_PAL8)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:1 (AV_PIX_FMT_YUYV422)
av_hwdevice_get_hwframe_constraints valid_sw_formats support:195 (AV_PIX_FMT_Y210LE)

 

我们使用avcodec_find_decoder查找解码器时，传入的codeid是27(AV_CODEC_ID_H264)，如果是硬解的话，必须使用avcodec_find_decoder_by_name，因为没有硬解的codeid

enum AVPixelFormat (*get_format)(struct AVCodecContext *s, const enum AVPixelFormat * fmt);
这是一个回调函数，它的作用就是告诉解码器codec自己的目标像素格式是什么。获取到了硬解码器codec可以支持的目标格式之后，就通过这个回调函数告知给codec，示例代码：

引用一段源码:
static enum AVPixelFormat get_format_hwdec(struct AVCodecContext *avctx,
                                           const enum AVPixelFormat *fmt)
{
    struct mp_filter *vd = avctx->opaque;
    vd_ffmpeg_ctx *ctx = vd->priv;

    MP_VERBOSE(vd, "Pixel formats supported by decoder:");
    for (int i = 0; fmt[i] != AV_PIX_FMT_NONE; i++)
        MP_VERBOSE(vd, " %s", av_get_pix_fmt_name(fmt[i]));
    MP_VERBOSE(vd, "\n");

    const char *profile = avcodec_profile_name(avctx->codec_id, avctx->profile);
    MP_VERBOSE(vd, "Codec profile: %s (0x%x), avctx->codec_id:%d\n", profile ? profile : "unknown",
               avctx->profile, avctx->codec_id);

    assert(ctx->use_hwdec);

    ctx->hwdec_request_reinit |= ctx->hwdec_failed;
    ctx->hwdec_failed = false;

    //选择目标像素格式
    enum AVPixelFormat select = AV_PIX_FMT_NONE;
    for (int i = 0; fmt[i] != AV_PIX_FMT_NONE; i++) {
        if (ctx->hwdec.pix_fmt == fmt[i]) {
            if (init_generic_hwaccel(vd, fmt[i]) < 0)
                break;
            select = fmt[i];
            break;
        }
    }

    if (select == AV_PIX_FMT_NONE) {       
        ctx->hwdec_failed = true;
        select = avcodec_default_get_format(avctx, fmt);
    }

    const char *name = av_get_pix_fmt_name(select);
    MP_VERBOSE(vd, "Requesting pixfmt '%s' from decoder. select:%d\n", name ? name : "-", select);
    return select;
}

fmt是这个解码器codec支持的像素格式，且按照质量优劣进行排序.我的环境中像素格式支持情况：

codec pixfmt[0]:23

codec pixfmt[1]:161

codec pixfmt[2]:116

如果没有特别的需要，这个步骤是可以省略的。内部默认会使用“native”的格式

2.创建SDL Texture

SDL_Texture *p_texture = SDL_CreateTexture(p_renderer,SDL_PIXELFORMAT_NV12,SDL_TEXTUREACCESS_STREAMING,pCodecCtx->width,pCodecCtx->height);

SDL_PIXELFORMAT_NV12此处format格式最好使用和frame的format对应的格式。免去了格式转换的工作，效率会提升很多

示例代码为ffmpeg上解码模块和SDL显示模块format对照表

static const struct TextureFormatEntry {
    enum AVPixelFormat format;
    int texture_fmt;
} sdl_texture_format_map[] = {
    { AV_PIX_FMT_RGB8,           SDL_PIXELFORMAT_RGB332 },
    { AV_PIX_FMT_RGB444,         SDL_PIXELFORMAT_RGB444 },
    { AV_PIX_FMT_RGB555,         SDL_PIXELFORMAT_RGB555 },
    { AV_PIX_FMT_BGR555,         SDL_PIXELFORMAT_BGR555 },
    { AV_PIX_FMT_RGB565,         SDL_PIXELFORMAT_RGB565 },
    { AV_PIX_FMT_BGR565,         SDL_PIXELFORMAT_BGR565 },
    { AV_PIX_FMT_RGB24,          SDL_PIXELFORMAT_RGB24 },
    { AV_PIX_FMT_BGR24,          SDL_PIXELFORMAT_BGR24 },
    { AV_PIX_FMT_0RGB32,         SDL_PIXELFORMAT_RGB888 },
    { AV_PIX_FMT_0BGR32,         SDL_PIXELFORMAT_BGR888 },
    { AV_PIX_FMT_NE(RGB0, 0BGR), SDL_PIXELFORMAT_RGBX8888 },
    { AV_PIX_FMT_NE(BGR0, 0RGB), SDL_PIXELFORMAT_BGRX8888 },
    { AV_PIX_FMT_RGB32,          SDL_PIXELFORMAT_ARGB8888 },
    { AV_PIX_FMT_RGB32_1,        SDL_PIXELFORMAT_RGBA8888 },
    { AV_PIX_FMT_BGR32,          SDL_PIXELFORMAT_ABGR8888 },
    { AV_PIX_FMT_BGR32_1,        SDL_PIXELFORMAT_BGRA8888 },
    { AV_PIX_FMT_YUV420P,        SDL_PIXELFORMAT_IYUV },
    { AV_PIX_FMT_YUYV422,        SDL_PIXELFORMAT_YUY2 },
    { AV_PIX_FMT_UYVY422,        SDL_PIXELFORMAT_UYVY },
    { AV_PIX_FMT_NONE,           SDL_PIXELFORMAT_UNKNOWN },

 3.小结

如果创建解码器时不指定get_format的格式，系统一般会默认选取支持的pix_fmt第一个作为frame的format。创建显示模块时可以根据frame的format进行动态创建texture