MediaCodec框架剖析

MediaCodec 是Android 提供的编解码库，MediaCodec 可以访问底层的媒体编解码器框架（StageFright 或者 OpenOMX），就是编解码组件，它与MediaExtractor、MediaMuxer、Surface、MediaSync、MediaCrypto、MediaDrm、Image、AudioTrack 一起构成Android 多媒体模块开发者相关的基本工具，MediaCodec本身通过调用底层的编解码组件获得了Codec的能力；

1. MediaCodec 源码调用流程剖析

创建MediaCodec实例：
MediaCodec->createDecoderByType(String type)：创建解码器；
MediaCodec->createEncoderByType(String type)：创建编码器；
这个type就是解析后的mimeType，mimeType 例如："video/avc"

MediaCodec->createByCodecName(String name)：根据codec名字创建的MediaCodec实例；
name是codec的名字：例如OMX.google.h264.decoder 是软解码；OMX.MTK.VIDEO.DECODER.AVC是硬解码；

1727    /**
1728     * Instantiate the preferred decoder supporting input data of the given mime type.
1729     *
1730     * The following is a partial list of defined mime types and their semantics:
1731     * 
1732     * "video/x-vnd.on2.vp8" - VP8 video (i.e. video in .webm)
1733     * 
"video/x-vnd.on2.vp9" - VP9 video (i.e. video in .webm)
1734     * 
"video/avc" - H.264/AVC video
1735     * 
"video/hevc" - H.265/HEVC video
1736     * 
"video/mp4v-es" - MPEG4 video
1737     * 
"video/3gpp" - H.263 video
1738     * 
"audio/3gpp" - AMR narrowband audio
1739     * 
"audio/amr-wb" - AMR wideband audio
1740     * 
"audio/mpeg" - MPEG1/2 audio layer III
1741     * 
"audio/mp4a-latm" - AAC audio (note, this is raw AAC packets, not packaged in LATM!)
1742     * 
"audio/vorbis" - vorbis audio
1743     * 
"audio/g711-alaw" - G.711 alaw audio
1744     * 
"audio/g711-mlaw" - G.711 ulaw audio
1745     * 
1746     *
1747     * Note: It is preferred to use {@link MediaCodecList#findDecoderForFormat}
1748     * and {@link #createByCodecName} to ensure that the resulting codec can handle a
1749     * given format.
1750     *
1751     * @param type The mime type of the input data.
1752     * @throws IOException if the codec cannot be created.
1753     * @throws IllegalArgumentException if type is not a valid mime type.
1754     * @throws NullPointerException if type is null.
1755     */
1756    @NonNull
1757    public static MediaCodec createDecoderByType(@NonNull String type)
1758            throws IOException {
1759        return new MediaCodec(type, true /* nameIsType */, false /* encoder */);
1760    }

MediaCodec 文件中已经枚举出各个type 类型了，一般情况下我们针对视频使用 "video/avc"，针对音频使用 "audio/mp4a-latm"

/frameworks/base/media/java/android/media/MediaCodec.java
/frameworks/base/media/jni/android_media_MediaCodec.cpp
/frameworks/av/media/libstagefright/MediaCodec.cpp
/frameworks/av/media/libstagefright/MediaCodecList.cpp
/frameworks/av/media/libmedia/MediaCodecInfo.cpp

JMediaCodec 是android_media_MediaCodec.cpp 文件中定义的类；

上层执行MediaCodec->createCodecByType 会调用到 /frameworks/av/media/libstagefright/MediaCodecList.cpp
中来检索对应的codec，检索的代码如下：
/frameworks/av/media/libstagefright/MediaCodecList.cpp
中定义的是初始化就写好的手机支持的编解码类型，这时候只需要去本地检索一下就可以的；

321//static
322void MediaCodecList::findMatchingCodecs(
323        const char *mime, bool encoder, uint32_t flags,
324        Vector *matches) {
325    matches->clear();
326
327    const sp list = getInstance();
328    if (list == nullptr) {
329        return;
330    }
331
332    size_t index = 0;
333    for (;;) {
334        ssize_t matchIndex =
335            list->findCodecByType(mime, encoder, index);
336
337        if (matchIndex < 0) {
338            break;
339        }
340
341        index = matchIndex + 1;
342
343        const sp info = list->getCodecInfo(matchIndex);
344        CHECK(info != nullptr);
345        AString componentName = info->getCodecName();
346
347        if ((flags & kHardwareCodecsOnly) && isSoftwareCodec(componentName)) {
348            ALOGV("skipping SW codec '%s'", componentName.c_str());
349        } else {
350            matches->push(componentName);
351            ALOGV("matching '%s'", componentName.c_str());
352        }
353    }
354
355    if (flags & kPreferSoftwareCodecs ||
356            property_get_bool("debug.stagefright.swcodec", false)) {
357        matches->sort(compareSoftwareCodecsFirst);
358    }
359}

源码中那些codec是软编码的codec：

OMX.google. 开头的codec name；

c2.android. 开头的codec name；
例如上面的OMX.google.h264.decoder 是软解码， OMX.MTK.VIDEO.DECODER.AVC是硬解码

293bool MediaCodecList::isSoftwareCodec(const AString &componentName) {
294    return componentName.startsWithIgnoreCase("OMX.google.")
295            || componentName.startsWithIgnoreCase("c2.android.")
296            || (!componentName.startsWithIgnoreCase("OMX.")
297                    && !componentName.startsWithIgnoreCase("c2."));
298}

获取了MediaCodec实例，底层也初始化好了对应的codec实例，接下来开始执行MediaCodec->configure操作；

format参数是MediaFormat信息；
使用MediaExtractor 可以分离出所有的track信息，有video和audio；

surface 参数主要为video 显示准备，我们将解码完的数据写入surface中，然后使用SurfaceFlinger合成一个个surface，最终展示出来，MediaCodec 可以脱离MediaPlayer 播放视频；

1005status_t MediaCodec::configure(
1006        const sp &format,
1007        const sp &surface,
1008        const sp &crypto,
1009        const sp &descrambler,
1010        uint32_t flags) {
1011    sp msg = new AMessage(kWhatConfigure, this);
1012
1013    if (mAnalyticsItem != NULL) {
1014        int32_t profile = 0;
1015        if (format->findInt32("profile", &profile)) {
1016            mAnalyticsItem->setInt32(kCodecProfile, profile);
1017        }
1018        int32_t level = 0;
1019        if (format->findInt32("level", &level)) {
1020            mAnalyticsItem->setInt32(kCodecLevel, level);
1021        }
1022        mAnalyticsItem->setInt32(kCodecEncoder, (flags & CONFIGURE_FLAG_ENCODE) ? 1 : 0);
1023    }
1024
1025    if (mIsVideo) {
1026        format->findInt32("width", &mVideoWidth);
1027        format->findInt32("height", &mVideoHeight);
1028        if (!format->findInt32("rotation-degrees", &mRotationDegrees)) {
1029            mRotationDegrees = 0;
1030        }
1031
1032        if (mAnalyticsItem != NULL) {
1033            mAnalyticsItem->setInt32(kCodecWidth, mVideoWidth);
1034            mAnalyticsItem->setInt32(kCodecHeight, mVideoHeight);
1035            mAnalyticsItem->setInt32(kCodecRotation, mRotationDegrees);
1036            int32_t maxWidth = 0;
1037            if (format->findInt32("max-width", &maxWidth)) {
1038                mAnalyticsItem->setInt32(kCodecMaxWidth, maxWidth);
1039            }
1040            int32_t maxHeight = 0;
1041            if (format->findInt32("max-height", &maxHeight)) {
1042                mAnalyticsItem->setInt32(kCodecMaxHeight, maxHeight);
1043            }
1044        }
1045
1046        // Prevent possible integer overflow in downstream code.
1047        if ((uint64_t)mVideoWidth * mVideoHeight > (uint64_t)INT32_MAX / 4) {
1048            ALOGE("buffer size is too big, width=%d, height=%d", mVideoWidth, mVideoHeight);
1049            return BAD_VALUE;
1050        }
1051    }
1052
1053    msg->setMessage("format", format);
1054    msg->setInt32("flags", flags);
1055    msg->setObject("surface", surface);
1056
1057    if (crypto != NULL || descrambler != NULL) {
1058        if (crypto != NULL) {
1059            msg->setPointer("crypto", crypto.get());
1060        } else {
1061            msg->setPointer("descrambler", descrambler.get());
1062        }
1063        if (mAnalyticsItem != NULL) {
1064            mAnalyticsItem->setInt32(kCodecCrypto, 1);
1065        }
1066    } else if (mFlags & kFlagIsSecure) {
1067        ALOGW("Crypto or descrambler should be given for secure codec");
1068    }
1069
1070    // save msg for reset
1071    mConfigureMsg = msg;
1072
1073    status_t err;
1074    Vector resources;
1075    MediaResource::Type type = (mFlags & kFlagIsSecure) ?
1076            MediaResource::kSecureCodec : MediaResource::kNonSecureCodec;
1077    MediaResource::SubType subtype =
1078            mIsVideo ? MediaResource::kVideoCodec : MediaResource::kAudioCodec;
1079    resources.push_back(MediaResource(type, subtype, 1));
1080    // Don't know the buffer size at this point, but it's fine to use 1 because
1081    // the reclaimResource call doesn't consider the requester's buffer size for now.
1082    resources.push_back(MediaResource(MediaResource::kGraphicMemory, 1));
1083    for (int i = 0; i <= kMaxRetry; ++i) {
1084        if (i > 0) {
1085            // Don't try to reclaim resource for the first time.
1086            if (!mResourceManagerService->reclaimResource(resources)) {
1087                break;
1088            }
1089        }
1090
1091        sp response;
1092        err = PostAndAwaitResponse(msg, &response);
1093        if (err != OK && err != INVALID_OPERATION) {
1094            // MediaCodec now set state to UNINITIALIZED upon any fatal error.
1095            // To maintain backward-compatibility, do a reset() to put codec
1096            // back into INITIALIZED state.
1097            // But don't reset if the err is INVALID_OPERATION, which means
1098            // the configure failure is due to wrong state.
1099
1100            ALOGE("configure failed with err 0x%08x, resetting...", err);
1101            reset();
1102        }
1103        if (!isResourceError(err)) {
1104            break;
1105        }
1106    }
1107    return err;
1108}

执行MediaCode->start ，PostAndAwaitResponse 开启一个Looper循环，MediaCodec开始工作；

如果一个输入缓冲区准备好，读取部分数据，复制到缓冲区；

如果一个输出缓冲区准备好，复制到缓冲区；

一个MediaCodec对象可以对特定的数据进行编码或者解码，MediaCodec只负责读入缓冲区数据，再输出到缓冲区，不负责播放音频或者视频；

1214status_t MediaCodec::start() {
1215    sp msg = new AMessage(kWhatStart, this);
1216
1217    status_t err;
1218    Vector resources;
1219    MediaResource::Type type = (mFlags & kFlagIsSecure) ?
1220            MediaResource::kSecureCodec : MediaResource::kNonSecureCodec;
1221    MediaResource::SubType subtype =
1222            mIsVideo ? MediaResource::kVideoCodec : MediaResource::kAudioCodec;
1223    resources.push_back(MediaResource(type, subtype, 1));
1224    // Don't know the buffer size at this point, but it's fine to use 1 because
1225    // the reclaimResource call doesn't consider the requester's buffer size for now.
1226    resources.push_back(MediaResource(MediaResource::kGraphicMemory, 1));
1227    for (int i = 0; i <= kMaxRetry; ++i) {
1228        if (i > 0) {
1229            // Don't try to reclaim resource for the first time.
1230            if (!mResourceManagerService->reclaimResource(resources)) {
1231                break;
1232            }
1233            // Recover codec from previous error before retry start.
1234            err = reset();
1235            if (err != OK) {
1236                ALOGE("retrying start: failed to reset codec");
1237                break;
1238            }
1239            sp response;
1240            err = PostAndAwaitResponse(mConfigureMsg, &response);
1241            if (err != OK) {
1242                ALOGE("retrying start: failed to configure codec");
1243                break;
1244            }
1245        }
1246
1247        sp response;
1248        err = PostAndAwaitResponse(msg, &response);
1249        if (!isResourceError(err)) {
1250            break;
1251        }
1252    }
1253    return err;
1254}

    ByteBuffer[] inputBuffers = mCodec.getInputBuffers();
    ByteBuffer[] outputBuffers = mCodec.getOutputBuffers();

上面是获取MediaCodec 输入缓冲区和输出缓冲区的方法，
我们读取输入缓冲区的数据，然后复制到共同缓冲区供surface消费；

MediaCodec.BufferInfo 就是MediaCodec中定义的共同缓冲区的结构；

1487    public final static class BufferInfo {
1488        /**
1489         * Update the buffer metadata information.
1490         *
1491         * @param newOffset the start-offset of the data in the buffer.
1492         * @param newSize   the amount of data (in bytes) in the buffer.
1493         * @param newTimeUs the presentation timestamp in microseconds.
1494         * @param newFlags  buffer flags associated with the buffer.  This
1495         * should be a combination of  {@link #BUFFER_FLAG_KEY_FRAME} and
1496         * {@link #BUFFER_FLAG_END_OF_STREAM}.
1497         */
1498        public void set(
1499                int newOffset, int newSize, long newTimeUs, @BufferFlag int newFlags) {
1500            offset = newOffset;
1501            size = newSize;
1502            presentationTimeUs = newTimeUs;
1503            flags = newFlags;
1504        }
1505
1506        /**
1507         * The start-offset of the data in the buffer.
1508         */
1509        public int offset;
1510
1511        /**
1512         * The amount of data (in bytes) in the buffer.  If this is {@code 0},
1513         * the buffer has no data in it and can be discarded.  The only
1514         * use of a 0-size buffer is to carry the end-of-stream marker.
1515         */
1516        public int size;
1517
1518        /**
1519         * The presentation timestamp in microseconds for the buffer.
1520         * This is derived from the presentation timestamp passed in
1521         * with the corresponding input buffer.  This should be ignored for
1522         * a 0-sized buffer.
1523         */
1524        public long presentationTimeUs;
1525
1526        /**
1527         * Buffer flags associated with the buffer.  A combination of
1528         * {@link #BUFFER_FLAG_KEY_FRAME} and {@link #BUFFER_FLAG_END_OF_STREAM}.
1529         *
1530         * Encoded buffers that are key frames are marked with
1531         * {@link #BUFFER_FLAG_KEY_FRAME}.
1532         *
1533         * The last output buffer corresponding to the input buffer
1534         * marked with {@link #BUFFER_FLAG_END_OF_STREAM} will also be marked
1535         * with {@link #BUFFER_FLAG_END_OF_STREAM}. In some cases this could
1536         * be an empty buffer, whose sole purpose is to carry the end-of-stream
1537         * marker.
1538         */
1539        @BufferFlag
1540        public int flags;
1541
1542        /** @hide */
1543        @NonNull
1544        public BufferInfo dup() {
1545            BufferInfo copy = new BufferInfo();
1546            copy.set(offset, size, presentationTimeUs, flags);
1547            return copy;
1548        }
1549    };

presentationTimeUs 表示数据帧的时间戳；

flags 数据帧的标记，BUFFER_FLAG_KEY_FRAME 表示关键帧；BUFFER_FLAG_END_OF_STREAM 表示最后输出的帧；

size 表示帧数据块的大小；

offset表示帧的起始偏移量；

2.MediaCodec 和OMX框架

MediaPlayer 播放器全面剖析(一)
MediaPlayer 播放器全面剖析(二)
MediaPlayer 全面剖析的文章中分析过，NuPlayer 中创建解码器部分会直接创建MediaCodec实例，并调用到 OMX框架中；

上一章在初始化MediaCodec的时候会调用到 MediaCodec::init 方法中；
/frameworks/av/media/libstagefright/MediaCodec.cpp
在它的init函数中：

940    mCodec->setCallback(
941            std::unique_ptr(
942                    new CodecCallback(new AMessage(kWhatCodecNotify, this))));
943    mBufferChannel = mCodec->getBufferChannel();
944    mBufferChannel->setCallback(
945            std::unique_ptr(
946                    new BufferCallback(new AMessage(kWhatCodecNotify, this))));
947
948    sp msg = new AMessage(kWhatInit, this);
949    msg->setObject("codecInfo", mCodecInfo);
950    // name may be different from mCodecInfo->getCodecName() if we stripped
951    // ".secure"
952    msg->setString("name", name);

发送一个 kWhatInit 消息，

2307        case kWhatInit:
2308        {
2309            sp replyID;
2310            CHECK(msg->senderAwaitsResponse(&replyID));
2311
2312            if (mState != UNINITIALIZED) {
2313                PostReplyWithError(replyID, INVALID_OPERATION);
2314                break;
2315            }
2316
2317            mReplyID = replyID;
2318            setState(INITIALIZING);
2319
2320            sp codecInfo;
2321            CHECK(msg->findObject("codecInfo", &codecInfo));
2322            AString name;
2323            CHECK(msg->findString("name", &name));
2324
2325            sp format = new AMessage;
2326            format->setObject("codecInfo", codecInfo);
2327            format->setString("componentName", name);
2328
2329            mCodec->initiateAllocateComponent(format);
2330            break;
2331        }

/frameworks/av/media/libstagefright/ACodec.cpp

631void ACodec::initiateAllocateComponent(const sp &msg) {
632    msg->setWhat(kWhatAllocateComponent);
633    msg->setTarget(this);
634    msg->post();
635}

发送kWhatAllocateComponent 消息，执行到ACodec::UninitializedState::onAllocateComponent 方法；说明node节点分配成功；

执行MediaCodec->configure 的时候会执行到/frameworks/av/media/libstagefright/ACodec.cpp
ACodec::configureCodec 函数，

2034        if (encoder) {
2035            err = setupVideoEncoder(mime, msg, outputFormat, inputFormat);
2036        } else {
2037            err = setupVideoDecoder(mime, msg, haveNativeWindow, usingSwRenderer, outputFormat);
2038        }

ACodec.cpp文件中创建了很多格式的解码实例，分别负责解析这类编码格式的文件，篇幅有限，不一一介绍；

MediaCodec框架剖析

1. MediaCodec 源码调用流程剖析

2.MediaCodec 和OMX框架

你可能感兴趣的:(MediaCodec框架剖析)