android中多媒体解码openmax的实现

每个 AwesomePlayer 只有一个OMX服务的入口，但是AwesomePlayer不一定就只需要1种解码器。有可能音视频都有，或者有很多种。这个时候这些解码器都需要OMX的服务，也就是OMX那头需要建立不同的解码器的组件来对应着AwesomePlayer中不同的code。OMX中非常重要的2个成员就是 OMXMaster 和 OMXNodeInstance。OMX通过这俩个成员来创建和维护不同的openmax 解码器组件，为AwesomePlayer中不同解码提供服务。让我们看看他们是怎么实现这些工作的。

1. OMX中 OMXNodeInstance 负责创建并维护不同的实例，这些实例是根据上面需求创建的，以node作为唯一标识。这样播放器中每个OMXCodec在OMX服务端都对应有了自己的OMXNodeInstance实例。

2.OMXMaster 维护底层软硬件解码库，根据OMXNodeInstance中想要的解码器来创建解码实体组件。

接下来我们假设视频解码器需要的是AVC，来看看解码器创建的流程。

(默认走软解码)

1.准备工作初始化OMXMaster

OMX构造函数中会进行初始化。

OMXMaster *mMaster;

OMXMaster *mMaster;

OMX::OMX()
    : mMaster(new OMXMaster),
      mNodeCounter(0) {
}

OMX::OMX()
    : mMaster(new OMXMaster),
      mNodeCounter(0) {
}

OMXMaster::OMXMaster()
    : mVendorLibHandle(NULL) {
    addVendorPlugin();
    addPlugin(new SoftOMXPlugin);
}

OMXMaster::OMXMaster()
    : mVendorLibHandle(NULL) {
    addVendorPlugin();
    addPlugin(new SoftOMXPlugin);
}

OMXMaster 负责OMX中编解码器插件管理，软件解码和硬件解码都是 使用OMX标准，挂载plugins的方式来进行管理。

软解通过 addPlugin(new SoftOMXPlugin);会把这些编解码器的名字都放在mPluginByComponentName中。

android 默认会提供一系列的软件解码器。目前支持这些格式的软编解码。

kComponents[] = {
    { "OMX.google.aac.decoder", "aacdec", "audio_decoder.aac" },
    { "OMX.google.aac.encoder", "aacenc", "audio_encoder.aac" },
    { "OMX.google.amrnb.decoder", "amrdec", "audio_decoder.amrnb" },
    { "OMX.google.amrnb.encoder", "amrnbenc", "audio_encoder.amrnb" },
    { "OMX.google.amrwb.decoder", "amrdec", "audio_decoder.amrwb" },
    { "OMX.google.amrwb.encoder", "amrwbenc", "audio_encoder.amrwb" },
    { "OMX.google.h264.decoder", "h264dec", "video_decoder.avc" },
    { "OMX.google.h264.encoder", "h264enc", "video_encoder.avc" },
    { "OMX.google.g711.alaw.decoder", "g711dec", "audio_decoder.g711alaw" },
    { "OMX.google.g711.mlaw.decoder", "g711dec", "audio_decoder.g711mlaw" },
    { "OMX.google.h263.decoder", "mpeg4dec", "video_decoder.h263" },
    { "OMX.google.h263.encoder", "mpeg4enc", "video_encoder.h263" },
    { "OMX.google.mpeg4.decoder", "mpeg4dec", "video_decoder.mpeg4" },
    { "OMX.google.mpeg4.encoder", "mpeg4enc", "video_encoder.mpeg4" },
    { "OMX.google.mp3.decoder", "mp3dec", "audio_decoder.mp3" },
    { "OMX.google.vorbis.decoder", "vorbisdec", "audio_decoder.vorbis" },
    { "OMX.google.vpx.decoder", "vpxdec", "video_decoder.vpx" },
    { "OMX.google.raw.decoder", "rawdec", "audio_decoder.raw" },
    { "OMX.google.flac.encoder", "flacenc", "audio_encoder.flac" },
};

kComponents[] = {
    { "OMX.google.aac.decoder", "aacdec", "audio_decoder.aac" },
    { "OMX.google.aac.encoder", "aacenc", "audio_encoder.aac" },
    { "OMX.google.amrnb.decoder", "amrdec", "audio_decoder.amrnb" },
    { "OMX.google.amrnb.encoder", "amrnbenc", "audio_encoder.amrnb" },
    { "OMX.google.amrwb.decoder", "amrdec", "audio_decoder.amrwb" },
    { "OMX.google.amrwb.encoder", "amrwbenc", "audio_encoder.amrwb" },
    { "OMX.google.h264.decoder", "h264dec", "video_decoder.avc" },
    { "OMX.google.h264.encoder", "h264enc", "video_encoder.avc" },
    { "OMX.google.g711.alaw.decoder", "g711dec", "audio_decoder.g711alaw" },
    { "OMX.google.g711.mlaw.decoder", "g711dec", "audio_decoder.g711mlaw" },
    { "OMX.google.h263.decoder", "mpeg4dec", "video_decoder.h263" },
    { "OMX.google.h263.encoder", "mpeg4enc", "video_encoder.h263" },
    { "OMX.google.mpeg4.decoder", "mpeg4dec", "video_decoder.mpeg4" },
    { "OMX.google.mpeg4.encoder", "mpeg4enc", "video_encoder.mpeg4" },
    { "OMX.google.mp3.decoder", "mp3dec", "audio_decoder.mp3" },
    { "OMX.google.vorbis.decoder", "vorbisdec", "audio_decoder.vorbis" },
    { "OMX.google.vpx.decoder", "vpxdec", "video_decoder.vpx" },
    { "OMX.google.raw.decoder", "rawdec", "audio_decoder.raw" },
    { "OMX.google.flac.encoder", "flacenc", "audio_encoder.flac" },
};

硬件编解码是通过 addVendorPlugin(); 加载 libstagefrighthw.so.各个芯片平台可以遵循openmax 标准,生成libstagefrighthw.so的库来提供android应用。

void OMXMaster::addVendorPlugin() {
    addPlugin("libstagefrighthw.so");
}

void OMXMaster::addVendorPlugin() {
    addPlugin("libstagefrighthw.so");
}

然后通过dlopen、dlsym来调用库中的函数。

这部分准备工作是在AwesomePlayer的构造函数中

CHECK_EQ(mClient.connect(), (status_t)OK); 已经完成了。

2.创建mVideoSource

有了上面的OMX，接下来会在AwesomePlayer::initVideoDecoder中创建mVideoSource 实例，下面代码只保留的主要部分：

status_t AwesomePlayer::initVideoDecoder(uint32_t flags) {
    ATRACE_CALL();
    mVideoSource = OMXCodec::Create(
            mClient.interface(), mVideoTrack->getFormat(),
            false, // createEncoder
            mVideoTrack,
            NULL, flags, USE_SURFACE_ALLOC ? mNativeWindow : NULL);
    status_t err = mVideoSource->start();
    return mVideoSource != NULL ? OK : UNKNOWN_ERROR;
}

status_t AwesomePlayer::initVideoDecoder(uint32_t flags) {
    ATRACE_CALL();
    mVideoSource = OMXCodec::Create(
            mClient.interface(), mVideoTrack->getFormat(),
            false, // createEncoder
            mVideoTrack,
            NULL, flags, USE_SURFACE_ALLOC ? mNativeWindow : NULL);
    status_t err = mVideoSource->start();
    return mVideoSource != NULL ? OK : UNKNOWN_ERROR;
}

保留主要部分，去除编码相关

sp OMXCodec::Create(
        const sp &omx,
        const sp &meta, bool createEncoder,
        const sp &source,
        const char *matchComponentName,
        uint32_t flags,
        const sp &nativeWindow) {
    int32_t requiresSecureBuffers;

    const char *mime;
    bool success = meta->findCString(kKeyMIMEType, &mime);
    CHECK(success);

    Vector matchingCodecs;
    Vector matchingCodecQuirks;
    findMatchingCodecs(
            mime, createEncoder, matchComponentName, flags,
            &matchingCodecs, &matchingCodecQuirks);

    sp observer = new OMXCodecObserver;
    IOMX::node_id node = 0;

    for (size_t i = 0; i < matchingCodecs.size(); ++i) {
        const char *componentNameBase = matchingCodecs[i].string();
        uint32_t quirks = matchingCodecQuirks[i];
        const char *componentName = componentNameBase;

        AString tmp;

        status_t err = omx->allocateNode(componentName, observer, &node);
        if (err == OK) {
            ALOGV("Successfully allocated OMX node '%s'", componentName);

            sp codec = new OMXCodec(
                    omx, node, quirks, flags,
                    createEncoder, mime, componentName,
                    source, nativeWindow);

            observer->setCodec(codec);

            err = codec->configureCodec(meta);

            if (err == OK) {
                if (!strcmp("OMX.Nvidia.mpeg2v.decode", componentName)) {
                    codec->mFlags |= kOnlySubmitOneInputBufferAtOneTime;
                }

                return codec;
            }

            ALOGV("Failed to configure codec '%s'", componentName);
        }
    }

    return NULL;
}

sp OMXCodec::Create(
        const sp &omx,
        const sp &meta, bool createEncoder,
        const sp &source,
        const char *matchComponentName,
        uint32_t flags,
        const sp &nativeWindow) {
    int32_t requiresSecureBuffers;

    const char *mime;
    bool success = meta->findCString(kKeyMIMEType, &mime);
    CHECK(success);

    Vector matchingCodecs;
    Vector matchingCodecQuirks;
    findMatchingCodecs(
            mime, createEncoder, matchComponentName, flags,
            &matchingCodecs, &matchingCodecQuirks);

    sp observer = new OMXCodecObserver;
    IOMX::node_id node = 0;

    for (size_t i = 0; i < matchingCodecs.size(); ++i) {
        const char *componentNameBase = matchingCodecs[i].string();
        uint32_t quirks = matchingCodecQuirks[i];
        const char *componentName = componentNameBase;

        AString tmp;

        status_t err = omx->allocateNode(componentName, observer, &node);
        if (err == OK) {
            ALOGV("Successfully allocated OMX node '%s'", componentName);

            sp codec = new OMXCodec(
                    omx, node, quirks, flags,
                    createEncoder, mime, componentName,
                    source, nativeWindow);

            observer->setCodec(codec);

            err = codec->configureCodec(meta);

            if (err == OK) {
                if (!strcmp("OMX.Nvidia.mpeg2v.decode", componentName)) {
                    codec->mFlags |= kOnlySubmitOneInputBufferAtOneTime;
                }

                return codec;
            }

            ALOGV("Failed to configure codec '%s'", componentName);
        }
    }

    return NULL;
}

1.根据mVideoTrack传进来的视频信息，查找相匹配的解码器。

bool success = meta->findCString(kKeyMIMEType, &mime);
findMatchingCodecs(
           mime, createEncoder, matchComponentName, flags,
           &matchingCodecs, &matchingCodecQuirks);

bool success = meta->findCString(kKeyMIMEType, &mime);
findMatchingCodecs(
           mime, createEncoder, matchComponentName, flags,
           &matchingCodecs, &matchingCodecQuirks);

2. 创建OMXCodecObserver 实例，OMXCodecObserver功能后续会详细介绍。创建一个node 并初始化为0.

sp observer = new OMXCodecObserver;
    IOMX::node_id node = 0;

sp observer = new OMXCodecObserver;
    IOMX::node_id node = 0;

3. 通过omx入口 依靠binder 机制调用OMX服务中的allocateNode()，这一步把匹配得到的解码器组件名、OMXCodecObserver实例和初始化为0的node一并传入。

status_t err = omx->allocateNode(componentName, observer, &node);

status_t err = omx->allocateNode(componentName, observer, &node);

这个allocateNode 就是文章最开始讲的，在OMX那头创建一个和mVideoSource相匹配的解码实例。用node值作为唯一标识。
让我们来看看真正的omx中allocateNode做了啥？

status_t OMX::allocateNode(
        const char *name, const sp &observer, node_id *node) {
    Mutex::Autolock autoLock(mLock);

    *node = 0;

    OMXNodeInstance *instance = new OMXNodeInstance(this, observer);

    OMX_COMPONENTTYPE *handle;
    OMX_ERRORTYPE err = mMaster->makeComponentInstance(
            name, &OMXNodeInstance::kCallbacks,
            instance, &handle);

    if (err != OMX_ErrorNone) {
        ALOGV("FAILED to allocate omx component '%s'", name);

        instance->onGetHandleFailed();

        return UNKNOWN_ERROR;
    }

    *node = makeNodeID(instance);
    mDispatchers.add(*node, new CallbackDispatcher(instance));

    instance->setHandle(*node, handle);

    mLiveNodes.add(observer->asBinder(), instance);
    observer->asBinder()->linkToDeath(this);

    return OK;
}

status_t OMX::allocateNode(
        const char *name, const sp &observer, node_id *node) {
    Mutex::Autolock autoLock(mLock);

    *node = 0;

    OMXNodeInstance *instance = new OMXNodeInstance(this, observer);

    OMX_COMPONENTTYPE *handle;
    OMX_ERRORTYPE err = mMaster->makeComponentInstance(
            name, &OMXNodeInstance::kCallbacks,
            instance, &handle);

    if (err != OMX_ErrorNone) {
        ALOGV("FAILED to allocate omx component '%s'", name);

        instance->onGetHandleFailed();

        return UNKNOWN_ERROR;
    }

    *node = makeNodeID(instance);
    mDispatchers.add(*node, new CallbackDispatcher(instance));

    instance->setHandle(*node, handle);

    mLiveNodes.add(observer->asBinder(), instance);
    observer->asBinder()->linkToDeath(this);

    return OK;
}

创建一个OMXNodeInstance实例。

通过mMaster->makeComponentInstance创建真正解码器的组件，并通过handle与OMXNodeInstance关联。

所以说mMaster->makeComponentInstance这里是建立解码器组件的核心。会把 mVideoSource需要的解码器name一直传递下去。

OMX_ERRORTYPE OMXMaster::makeComponentInstance(
        const char *name,
        const OMX_CALLBACKTYPE *callbacks,
        OMX_PTR appData,
        OMX_COMPONENTTYPE **component) {
    Mutex::Autolock autoLock(mLock);

    *component = NULL;

    ssize_t index = mPluginByComponentName.indexOfKey(String8(name));

    if (index < 0) {
        return OMX_ErrorInvalidComponentName;
    }

    OMXPluginBase *plugin = mPluginByComponentName.valueAt(index);
    OMX_ERRORTYPE err =
        plugin->makeComponentInstance(name, callbacks, appData, component);

    if (err != OMX_ErrorNone) {
        return err;
    }

    mPluginByInstance.add(*component, plugin);

    return err;
}

OMX_ERRORTYPE OMXMaster::makeComponentInstance(
        const char *name,
        const OMX_CALLBACKTYPE *callbacks,
        OMX_PTR appData,
        OMX_COMPONENTTYPE **component) {
    Mutex::Autolock autoLock(mLock);

    *component = NULL;

    ssize_t index = mPluginByComponentName.indexOfKey(String8(name));

    if (index < 0) {
        return OMX_ErrorInvalidComponentName;
    }

    OMXPluginBase *plugin = mPluginByComponentName.valueAt(index);
    OMX_ERRORTYPE err =
        plugin->makeComponentInstance(name, callbacks, appData, component);

    if (err != OMX_ErrorNone) {
        return err;
    }

    mPluginByInstance.add(*component, plugin);

    return err;
}

最开始OMXMaster通过 addPlugin(new SoftOMXPlugin);把支持的软解码放在mPluginByComponentName中，在makeComponentInstance中通过上面传下来的解码器的name值从 mPluginByComponentName找到相对应的plugin，然后调用  plugin->makeComponentInstance(name, callbacks, appData, component);

这里的plugin 值得就是软解SoftOMXPlugin 也就是调用了

OMX_ERRORTYPE SoftOMXPlugin::makeComponentInstance(
        const char *name,
        const OMX_CALLBACKTYPE *callbacks,
        OMX_PTR appData,
        OMX_COMPONENTTYPE **component) {
    ALOGV("makeComponentInstance '%s'", name);

    for (size_t i = 0; i < kNumComponents; ++i) {
        if (strcmp(name, kComponents[i].mName)) {
            continue;
        }

        AString libName = "libstagefright_soft_";
        libName.append(kComponents[i].mLibNameSuffix);
        libName.append(".so");

        void *libHandle = dlopen(libName.c_str(), RTLD_NOW);

        if (libHandle == NULL) {
            ALOGE("unable to dlopen %s", libName.c_str());

            return OMX_ErrorComponentNotFound;
        }

        typedef SoftOMXComponent *(*CreateSoftOMXComponentFunc)(
                const char *, const OMX_CALLBACKTYPE *,
                OMX_PTR, OMX_COMPONENTTYPE **);

        CreateSoftOMXComponentFunc createSoftOMXComponent =
            (CreateSoftOMXComponentFunc)dlsym(
                    libHandle,
                    "_Z22createSoftOMXComponentPKcPK16OMX_CALLBACKTYPE"
                    "PvPP17OMX_COMPONENTTYPE");

        if (createSoftOMXComponent == NULL) {
            dlclose(libHandle);
            libHandle = NULL;

            return OMX_ErrorComponentNotFound;
        }

        sp codec =
            (*createSoftOMXComponent)(name, callbacks, appData, component);

        if (codec == NULL) {
            dlclose(libHandle);
            libHandle = NULL;

            return OMX_ErrorInsufficientResources;
        }

        OMX_ERRORTYPE err = codec->initCheck();
        if (err != OMX_ErrorNone) {
            dlclose(libHandle);
            libHandle = NULL;

            return err;
        }

        codec->incStrong(this);
        codec->setLibHandle(libHandle);

        return OMX_ErrorNone;
    }

    return OMX_ErrorInvalidComponentName;
}

OMX_ERRORTYPE SoftOMXPlugin::makeComponentInstance(
        const char *name,
        const OMX_CALLBACKTYPE *callbacks,
        OMX_PTR appData,
        OMX_COMPONENTTYPE **component) {
    ALOGV("makeComponentInstance '%s'", name);

    for (size_t i = 0; i < kNumComponents; ++i) {
        if (strcmp(name, kComponents[i].mName)) {
            continue;
        }

        AString libName = "libstagefright_soft_";
        libName.append(kComponents[i].mLibNameSuffix);
        libName.append(".so");

        void *libHandle = dlopen(libName.c_str(), RTLD_NOW);

        if (libHandle == NULL) {
            ALOGE("unable to dlopen %s", libName.c_str());

            return OMX_ErrorComponentNotFound;
        }

        typedef SoftOMXComponent *(*CreateSoftOMXComponentFunc)(
                const char *, const OMX_CALLBACKTYPE *,
                OMX_PTR, OMX_COMPONENTTYPE **);

        CreateSoftOMXComponentFunc createSoftOMXComponent =
            (CreateSoftOMXComponentFunc)dlsym(
                    libHandle,
                    "_Z22createSoftOMXComponentPKcPK16OMX_CALLBACKTYPE"
                    "PvPP17OMX_COMPONENTTYPE");

        if (createSoftOMXComponent == NULL) {
            dlclose(libHandle);
            libHandle = NULL;

            return OMX_ErrorComponentNotFound;
        }

        sp codec =
            (*createSoftOMXComponent)(name, callbacks, appData, component);

        if (codec == NULL) {
            dlclose(libHandle);
            libHandle = NULL;

            return OMX_ErrorInsufficientResources;
        }

        OMX_ERRORTYPE err = codec->initCheck();
        if (err != OMX_ErrorNone) {
            dlclose(libHandle);
            libHandle = NULL;

            return err;
        }

        codec->incStrong(this);
        codec->setLibHandle(libHandle);

        return OMX_ErrorNone;
    }

    return OMX_ErrorInvalidComponentName;
}

通过上面传下来的解码器的name，找到对应库的名字。假如是264的话，要加载的库就是 libstagefright_soft_h264dec.so,也就是对应上层264解码的话，omx解码组件会加载对应的 libstagefright_soft_ h264dec.so库。相对应的软解代码在 Android4.1.1\frameworks\av\media\libstagefright\codecs\on2\h264dec 中。

加载完264解码库后 通过dlopen、dlsym来调用库中函数。

通过调用 SoftAVC 中的 createSoftOMXComponent 来创建真正264解码器实例SoftOMXComponent。以后真正视频解码的工作都是通过avc 这个SoftAVC实例完成的

android::SoftOMXComponent *createSoftOMXComponent(
        const char *name, const OMX_CALLBACKTYPE *callbacks,
        OMX_PTR appData, OMX_COMPONENTTYPE **component) {
    return new android::SoftAVC(name, callbacks, appData, component);
}

android::SoftOMXComponent *createSoftOMXComponent(
        const char *name, const OMX_CALLBACKTYPE *callbacks,
        OMX_PTR appData, OMX_COMPONENTTYPE **component) {
    return new android::SoftAVC(name, callbacks, appData, component);
}

经过这一路下来，终于完成了解码器的创建工作。简单总结一下。

1.AwesomePlayer中通过initVideoDecoder 来创建video解码器mVideoSource

2.mVideoSource 中通过上部分demux后的视频流 mVideoTrack来获得解码器的类型，通过类型调用omx->allocateNode 创建omx node实例与自己对应。以后都是通过node实例来操作解码器。

3.在 omx->allocateNode中 通过mMaster->makeComponentInstance 来创建真正对应的解码器组件。这个解码器组件是完成之后解码实际工作的。

4.在创建mMaster->makeComponentInstance过程中，也是通过上面mVideoTrack 过来的解码器类型名，找到相对应的解码器的库，然后实例化。