OMX标准接口OMX_FillThisBuffer机制详解

一、引言：
OMX组件的标准接口（OMX_Core.h）中，OMX_FillThisBuffer和OMX_EmptyThisBuffer共同完成了OMX的buffer运转。OMX_FillThisBuffer是操作解码完后数据（PCM/YUV）的，OMX_EmptyThisBuffer是操作解码前（es）数据的。本博客将分析OMX_FillThisBuffer，下篇博客再分析OMX_EmptyThisBuffer。
整个buffer机制的分析需要从两个方面进行，一方面是OMX回调通知ACodec和MediaCodec，另一方面是MediaCodec使用完buffer后通知OMX这个过程，两个过程组合起来实际上就是生产者与消费者的关系。对于OMX_FillThisBuffer而言，MediaCodec是消费者，OMX组件是生产者。
之前讲OMX消息机制的时候，说过OMXNodeInstance会实例化一个OMX_CALLBACKTYPE的结构体，指向了三个函数：

OMX_CALLBACKTYPE OMXNodeInstance::kCallbacks = {
    &OnEvent, &OnEmptyBufferDone, &OnFillBufferDone
};

OMX_CALLBACKTYPE就是OMX_Core.h中的标准接口：

typedef struct OMX_CALLBACKTYPE
{
   OMX_ERRORTYPE (*EventHandler)(
        OMX_IN OMX_HANDLETYPE hComponent,
        OMX_IN OMX_PTR pAppData,
        OMX_IN OMX_EVENTTYPE eEvent,
        OMX_IN OMX_U32 nData1,
        OMX_IN OMX_U32 nData2,
        OMX_IN OMX_PTR pEventData);

    OMX_ERRORTYPE (*EmptyBufferDone)(
        OMX_IN OMX_HANDLETYPE hComponent,
        OMX_IN OMX_PTR pAppData,
        OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);

    OMX_ERRORTYPE (*FillBufferDone)(
        OMX_OUT OMX_HANDLETYPE hComponent,
        OMX_OUT OMX_PTR pAppData,
        OMX_OUT OMX_BUFFERHEADERTYPE* pBuffer);

} OMX_CALLBACKTYPE;

EmptyBufferDone和FillBufferDone就是OMX通知给ACodec回调接口。

二、ACodecBufferChannel介绍：
ACodecBufferChannel这个类是供ACodec用于更新buffer信息的，类的实例化操作是在MediaCodec的init函数中完成：

status_t MediaCodec::init(const AString &name) {
	...
	/* 1.实例化ACodecBufferChannel */
    mBufferChannel = mCodec->getBufferChannel();
    /* 2.设置ACodecBufferChannel的回调函数 */
    mBufferChannel->setCallback(
            std::unique_ptr<CodecBase::BufferCallback>(
                    new BufferCallback(new AMessage(kWhatCodecNotify, this))));	
	...
}

看一下ACodec中getBufferChannel函数的操作：

std::shared_ptr<BufferChannelBase> ACodec::getBufferChannel() {
    if (!mBufferChannel) {
        mBufferChannel = std::make_shared<ACodecBufferChannel>(
                new AMessage(kWhatInputBufferFilled, this),
                new AMessage(kWhatOutputBufferDrained, this));
    }
    return mBufferChannel;
}

构造函数中实例化了两个msg，分别和OMX组件的标准回调消息EmptyBufferDone及FillBufferDone配合使用。
回到前面看回调的设置，当ACodecBufferChannel实例化完成后，MediaCodec为了能和ACodec同步获知OMX对buffer的更新情况，会设置一个BufferCallback（主要是一个msg）到ACodecBufferChannel中：

BufferCallback::BufferCallback(const sp<AMessage> &notify)
    : mNotify(notify) {}

notify就是上面的kWhatCodecNotify，有了这个msg之后，ACodecBufferChannel中的消息就可以由MediaCodec来处理了。

三、OMX通过FillBufferDone通知至MediaCodec：
不论是vdec组件还是adec组件，当某个buffer的数据填充完之后，OMX组件会调用callbacks.FillBufferDone通知至上层，下面看下芯片平台的具体实现：

pMSComponent->callbacks.FillBufferDone(pOMXComponent, pMSComponent->callbackData, bufferHeader);

参数中bufferHeader包括了解码完后buffer的所有信息。下面看下OMXNodeInstance中OnFillBufferDone的实现：

OMX_ERRORTYPE OMXNodeInstance::OnFillBufferDone(
        OMX_IN OMX_HANDLETYPE /* hComponent */,
        OMX_IN OMX_PTR pAppData,
        OMX_IN OMX_BUFFERHEADERTYPE* pBuffer) {
    if (pAppData == NULL) {
        ALOGE("b/25884056");
        return OMX_ErrorBadParameter;
    }
    OMXNodeInstance *instance = static_cast<OMXNodeInstance *>(pAppData);
    if (instance->mDying) {
        return OMX_ErrorNone;
    }
    int fenceFd = instance->retrieveFenceFromMeta_l(pBuffer, kPortIndexOutput);

    omx_message msg;
    msg.type = omx_message::FILL_BUFFER_DONE;
    msg.fenceFd = fenceFd;
    msg.u.extended_buffer_data.buffer = instance->findBufferID(pBuffer);
    msg.u.extended_buffer_data.range_offset = pBuffer->nOffset;
    msg.u.extended_buffer_data.range_length = pBuffer->nFilledLen;
    msg.u.extended_buffer_data.flags = pBuffer->nFlags;
    msg.u.extended_buffer_data.timestamp = pBuffer->nTimeStamp;
    instance->mDispatcher->post(msg);

    return OMX_ErrorNone;
}

OnFillBufferDone的函数实现比较简单，就是将buffer的相关信息填充到omx_message结构体中并发送出去。下面看下omx_message::FILL_BUFFER_DONE的消息处理：

void OMXNodeInstance::onMessages(std::list<omx_message> &messages) {
    for (std::list<omx_message>::iterator it = messages.begin(); it != messages.end(); ) {
    	/* OMXNodeInstance的消息处理 */
        if (handleMessage(*it)) {
            messages.erase(it++);
        } else {
            ++it;
        }
    }
	/* ACodec的消息处理 */
    if (!messages.empty()) {
        mObserver->onMessages(messages);
    }
}

先看handleMessage：

if (msg.type == omx_message::FILL_BUFFER_DONE) {
	/* 1.获取buffer信息 */
    OMX_BUFFERHEADERTYPE *buffer =
        findBufferHeader(msg.u.extended_buffer_data.buffer, kPortIndexOutput);
    if (buffer == NULL) {
        ALOGE("b/25884056");
        return false;
    }

    {
        Mutex::Autolock _l(mDebugLock);
        mOutputBuffersWithCodec.remove(buffer);

        CLOG_BUMPED_BUFFER(
                FBD, WITH_STATS(FULL_BUFFER(
                        msg.u.extended_buffer_data.buffer, buffer, msg.fenceFd)));

        unbumpDebugLevel_l(kPortIndexOutput);
    }

    BufferMeta *buffer_meta =
        static_cast<BufferMeta *>(buffer->pAppPrivate);

    if (buffer->nOffset + buffer->nFilledLen < buffer->nOffset
            || buffer->nOffset + buffer->nFilledLen > buffer->nAllocLen) {
        CLOG_ERROR(onFillBufferDone, OMX_ErrorBadParameter,
                FULL_BUFFER(NULL, buffer, msg.fenceFd));
    }
    /* 2.将OMX的buffer拷贝到framework层共享buffer中 */
    buffer_meta->CopyFromOMX(buffer);

    // fix up the buffer info (especially timestamp) if needed
    codecBufferFilled(msg);
}

之前分析消息机制的时候，已经说过，这里的mObserver就是ACodec传递下来的，中间的消息传递过程就不再赘述：

 case omx_message::FILL_BUFFER_DONE:
 {
     IOMX::buffer_id bufferID;
     CHECK(msg->findInt32("buffer", (int32_t*)&bufferID));

     int32_t rangeOffset, rangeLength, flags, fenceFd;
     int64_t timeUs;

     CHECK(msg->findInt32("range_offset", &rangeOffset));
     CHECK(msg->findInt32("range_length", &rangeLength));
     CHECK(msg->findInt32("flags", &flags));
     CHECK(msg->findInt64("timestamp", &timeUs));
     CHECK(msg->findInt32("fence_fd", &fenceFd));
	
     return onOMXFillBufferDone(
             bufferID,
             (size_t)rangeOffset, (size_t)rangeLength,
             (OMX_U32)flags,
             timeUs,
             fenceFd);
 }

继续看BaseState::onOMXFillBufferDone：

bool ACodec::BaseState::onOMXFillBufferDone(
        IOMX::buffer_id bufferID,
        size_t rangeOffset, size_t rangeLength,
        OMX_U32 flags,
        int64_t timeUs,
        int fenceFd) {

	...
	PortMode mode = getPortMode(kPortIndexOutput);
	switch (mode) {
		...
        case RESUBMIT_BUFFERS:
        {	
        	...
        	/* 更新buffer池 */
        	mCodec->mBufferChannel->drainThisBuffer(info->mBufferID, flags);
        	...
        }
        ...
   }	
	...
}

这里就需要用到ACodecBufferChannel了，看一下drainThisBuffer的实现：

void ACodecBufferChannel::drainThisBuffer(
        IOMX::buffer_id bufferId,
        OMX_U32 omxFlags) {
    ALOGV("drainThisBuffer #%d", bufferId);
    std::shared_ptr<const std::vector<const BufferInfo>> array(
            std::atomic_load(&mOutputBuffers));
    BufferInfoIterator it = findBufferId(array, bufferId);

    if (it == array->end()) {
        ALOGE("drainThisBuffer: unrecognized buffer #%d", bufferId);
        return;
    }
    if (it->mClientBuffer != it->mCodecBuffer) {
        it->mClientBuffer->setFormat(it->mCodecBuffer->format());
    }

    uint32_t flags = 0;
    if (omxFlags & OMX_BUFFERFLAG_SYNCFRAME) {
        flags |= MediaCodec::BUFFER_FLAG_SYNCFRAME;
    }
    if (omxFlags & OMX_BUFFERFLAG_CODECCONFIG) {
        flags |= MediaCodec::BUFFER_FLAG_CODECCONFIG;
    }
    if (omxFlags & OMX_BUFFERFLAG_EOS) {
        flags |= MediaCodec::BUFFER_FLAG_EOS;
    }
    it->mClientBuffer->meta()->setInt32("flags", flags);
	/* 调用BufferCallback的BufferCallback函数 */
    mCallback->onOutputBufferAvailable(
            std::distance(array->begin(), it),
            it->mClientBuffer);
}

可以看到该函数的重点是回调通知到onOutputBufferAvailable：

void BufferCallback::onOutputBufferAvailable(
        size_t index, const sp<MediaCodecBuffer> &buffer) {
    sp<AMessage> notify(mNotify->dup());
    notify->setInt32("what", kWhatDrainThisBuffer);
    notify->setSize("index", index);
    notify->setObject("buffer", buffer);
    notify->post();
}

消息处理中，最重要的就是下面这句：

case kWhatDrainThisBuffer:
{
	/* size_t index = */updateBuffers(kPortIndexOutput, msg);
	...
}

updateBuffers实现：

size_t MediaCodec::updateBuffers(
        int32_t portIndex, const sp<AMessage> &msg) {
    CHECK(portIndex == kPortIndexInput || portIndex == kPortIndexOutput);
    size_t index;
    CHECK(msg->findSize("index", &index));
    sp<RefBase> obj;
    CHECK(msg->findObject("buffer", &obj));
    sp<MediaCodecBuffer> buffer = static_cast<MediaCodecBuffer *>(obj.get());

    {
        Mutex::Autolock al(mBufferLock);
        if (mPortBuffers[portIndex].size() <= index) {
            mPortBuffers[portIndex].resize(align(index + 1, kNumBuffersAlign));
        }
        mPortBuffers[portIndex][index].mData = buffer;
    }
    /* 将可用buffer的index入队 */
    mAvailPortBuffers[portIndex].push_back(index);

    return index;
}

mAvailPortBuffers这个队列存储了目前可用的buffer数量，实际上OMX组件与MediaCodec的最终交互就体现在这个队列中。到这里，整个OMX->MediaCodec的buffer通知机制就分析完了，总结来说，就是当底层OMX组件有解码完后的数据时，会将buffer信息层层回调，通知至MediaCodec并更新到可用buffer队列中，进而供MediaCodec查询使用。

四、dequeueOutputBuffer分析：
在上一个章节中讲到当OMX底层完成了解码并将buffer信息更新到mAvailPortBuffers后，此时，上层应用就可以使用这个buffer中的数据进行播放或者送显，而首先需要做的，就是调用dequeueOutputBuffer获得可用buffer的index：

status_t MediaCodec::dequeueOutputBuffer(
        size_t *index,
        size_t *offset,
        size_t *size,
        int64_t *presentationTimeUs,
        uint32_t *flags,
        int64_t timeoutUs) {
	...
	/* 发送消息查询index */
    sp<AMessage> msg = new AMessage(kWhatDequeueOutputBuffer, this);
    msg->setInt64("timeoutUs", timeoutUs);

    sp<AMessage> response;
    status_t err;
    if ((err = PostAndAwaitResponse(msg, &response)) != OK) {
        return err;
    }

    CHECK(response->findSize("index", index));
    CHECK(response->findSize("offset", offset));
    CHECK(response->findSize("size", size));
    CHECK(response->findInt64("timeUs", presentationTimeUs));
    CHECK(response->findInt32("flags", (int32_t *)flags));
	...

    return OK;
}

看一下kWhatDequeueOutputBuffer的做法：

case kWhatDequeueOutputBuffer:
{
	...
	/* 获取输出index */
    if (handleDequeueOutputBuffer(replyID, true /* new request */)) {
        break;
    }
	...
}

可以看到函数是通过handleDequeueOutputBuffer拿到index的：

bool MediaCodec::handleDequeueOutputBuffer(const sp<AReplyToken> &replyID, bool newRequest) {
	...
	else {
        sp<AMessage> response = new AMessage;
        ssize_t index = dequeuePortBuffer(kPortIndexOutput);

        if (index < 0) {
            CHECK_EQ(index, -EAGAIN);
            return false;
        }

        const sp<MediaCodecBuffer> &buffer =
            mPortBuffers[kPortIndexOutput][index].mData;

        response->setSize("index", index);
        response->setSize("offset", buffer->offset());
        response->setSize("size", buffer->size());

        int64_t timeUs;
        CHECK(buffer->meta()->findInt64("timeUs", &timeUs));

        statsBufferReceived(timeUs);

        response->setInt64("timeUs", timeUs);

        int32_t flags;
        CHECK(buffer->meta()->findInt32("flags", &flags));

        response->setInt32("flags", flags);
        response->postReply(replyID);
    }

    return true;
}

再次跟进到dequeuePortBuffer：

ssize_t MediaCodec::dequeuePortBuffer(int32_t portIndex) {
    CHECK(portIndex == kPortIndexInput || portIndex == kPortIndexOutput);
	/* 1.获取buffer队列的临时指针对象  */
    List<size_t> *availBuffers = &mAvailPortBuffers[portIndex];

    if (availBuffers->empty()) {
        return -EAGAIN;
    }
	/* 2.获得队首index */
    size_t index = *availBuffers->begin();
    /* 3.将此index从队列中删除 */
    availBuffers->erase(availBuffers->begin());
	
	/* 4.通过index拿到对应的buffer信息并填充 */
    BufferInfo *info = &mPortBuffers[portIndex][index];
    CHECK(!info->mOwnedByClient);
    {
        Mutex::Autolock al(mBufferLock);
        info->mOwnedByClient = true;

        // set image-data
        if (info->mData->format() != NULL) {
            sp<ABuffer> imageData;
            if (info->mData->format()->findBuffer("image-data", &imageData)) {
                info->mData->meta()->setBuffer("image-data", imageData);
            }
            int32_t left, top, right, bottom;
            if (info->mData->format()->findRect("crop", &left, &top, &right, &bottom)) {
                info->mData->meta()->setRect("crop-rect", left, top, right, bottom);
            }
        }
    }

    return index;
}

dequeuePortBuffer函数的目的很简单，就是从mAvailPortBuffers中获取可用buffer的index，如果没有的话，会返回-EAGAIN。

五、releaseOutputBuffer分析：
当输出buffer使用完之后，需要OMX进行继续填充，这个过程就是由releaseOutputBuffer来完成的：

status_t MediaCodec::releaseOutputBuffer(size_t index) {
    sp<AMessage> msg = new AMessage(kWhatReleaseOutputBuffer, this);
    msg->setSize("index", index);

    sp<AMessage> response;
    return PostAndAwaitResponse(msg, &response);
}

跟进kWhatReleaseOutputBuffer：

case kWhatReleaseOutputBuffer:
{
    sp<AReplyToken> replyID;
    CHECK(msg->senderAwaitsResponse(&replyID));
/* 检查当前状态是否正常 */
    if (!isExecuting()) {
        PostReplyWithError(replyID, INVALID_OPERATION);
        break;
    } else if (mFlags & kFlagStickyError) {
        PostReplyWithError(replyID, getStickyError());
        break;
    }
/* 继续往下调用 */
    status_t err = onReleaseOutputBuffer(msg);

    PostReplyWithError(replyID, err);
    break;
}

继续onReleaseOutputBuffer：

status_t MediaCodec::onReleaseOutputBuffer(const sp<AMessage> &msg) {
	...
	/* 是否需要渲染后再释放 */
	if (render && buffer->size() != 0) {
		...
        mBufferChannel->renderOutputBuffer(buffer, renderTimeNs);
    } else {
    	/* 直接释放 */
        mBufferChannel->discardBuffer(buffer);
    }
	return OK;
}

renderOutputBuffer和discardBuffer的区别为是否需要渲染后再去释放，通常，视频数据是借助MediaCodec绑定的Surface去渲染的，所以会调用renderOutputBuffer，而音频数据此时已经copy到audiotrack中了，故直接释放作废即可。
接下来看一下mBufferChannel（ACodecBufferChannel.cpp）中的具体实现：

status_t ACodecBufferChannel::discardBuffer(const sp<MediaCodecBuffer> &buffer) {
    std::shared_ptr<const std::vector<const BufferInfo>> array(
            std::atomic_load(&mInputBuffers));
    bool input = true;
    BufferInfoIterator it = findClientBuffer(array, buffer);
    if (it == array->end()) {
        array = std::atomic_load(&mOutputBuffers);
        input = false;
        it = findClientBuffer(array, buffer);
        if (it == array->end()) {
            return -ENOENT;
        }
    }
    ALOGV("discardBuffer #%d", it->mBufferId);
    /* 获取kWhatOutputBufferDrained消息 */
    sp<AMessage> msg = input ? mInputBufferFilled->dup() : mOutputBufferDrained->dup();
    msg->setObject("buffer", it->mCodecBuffer);
    msg->setInt32("buffer-id", it->mBufferId);
    /* 打上废弃标志 */
    msg->setInt32("discarded", true);
    msg->post();
    return OK;
}

mInputBufferFilled->dup()对应的是kWhatInputBufferFilled消息实体，mOutputBufferDrained->dup()对应的是kWhatOutputBufferDrained消息实体。如果调用renderOutputBuffer的话会往msg中设置render为true的键值对，调用discardBuffer的话会往msg中设置discarded为true的键值对。
继续往下，看下kWhatOutputBufferDrained的消息处理：

case kWhatOutputBufferDrained:
{
    onOutputBufferDrained(msg);
    break;
}

onOutputBufferDrained函数比较长，我们只关注重点部分：

void ACodec::BaseState::onOutputBufferDrained(const sp<AMessage> &msg) {
	...
	PortMode mode = getPortMode(kPortIndexOutput);
	switch (mode) {
		...
		case RESUBMIT_BUFFERS:
        {
            if (!mCodec->mPortEOS[kPortIndexOutput]) {
                if (info->mStatus == BufferInfo::OWNED_BY_NATIVE_WINDOW) {
                    // We cannot resubmit the buffer we just rendered, dequeue
                    // the spare instead.

                    info = mCodec->dequeueBufferFromNativeWindow();
                }

                if (info != NULL) {
                    ALOGV("[%s] calling fillBuffer %u",
                         mCodec->mComponentName.c_str(), info->mBufferID);
                    info->checkWriteFence("onOutputBufferDrained::RESUBMIT_BUFFERS");
                    /* 调入到ACodec中 */
                    status_t err = mCodec->fillBuffer(info);
                    if (err != OK) {
                        mCodec->signalError(OMX_ErrorUndefined, makeNoSideEffectStatus(err));
                    }
                }
            }
            break;
        }
        ...
	}
	...
}

因为此时ACodec的状态机为ExecutingState，所以会进入RESUBMIT_BUFFERS的case中去，最终会调用到fillBuffer通知到OMX组件进行buffer的填充：

status_t ACodec::fillBuffer(BufferInfo *info) {
    status_t err;
    // Even in dynamic ANW buffer mode, if the graphic buffer is not changing,
    // send sPreset instead of the same graphic buffer, so that OMX server
    // side doesn't update the meta. In theory it should make no difference,
    // however when the same buffer is parcelled again, a new handle could be
    // created on server side, and some decoder doesn't recognize the handle
    // even if it's the same buffer.
    if (!storingMetadataInDecodedBuffers() || !info->mNewGraphicBuffer) {
        err = mOMXNode->fillBuffer(
            info->mBufferID, OMXBuffer::sPreset, info->mFenceFd);
    } else {
        err = mOMXNode->fillBuffer(
            info->mBufferID, info->mGraphicBuffer, info->mFenceFd);
    }

    info->mNewGraphicBuffer = false;
    info->mFenceFd = -1;
    if (err == OK) {
        info->mStatus = BufferInfo::OWNED_BY_COMPONENT;
    }
    return err;
}

不管是视频还是音频数据填充，都会走入到OMXNodeInstance中：

status_t OMXNodeInstance::fillBuffer(
        IOMX::buffer_id buffer, const OMXBuffer &omxBuffer, int fenceFd) {
    Mutex::Autolock autoLock(mLock);

    OMX_BUFFERHEADERTYPE *header = findBufferHeader(buffer, kPortIndexOutput);
    if (header == NULL) {
        ALOGE("b/25884056");
        return BAD_VALUE;
    }
    ...
    OMX_ERRORTYPE err = OMX_FillThisBuffer(mHandle, header);
}

最终，在这里就会调入到OMX的标准接口OMX_FillThisBuffer中去了。再看一下芯片平台对这个接口的实现：

OMX_ERRORTYPE MS_OMX_FillThisBuffer(
    OMX_IN OMX_HANDLETYPE        hComponent,
    OMX_IN OMX_BUFFERHEADERTYPE *pBuffer)
{
	...
	/* 1.上面传下来的pBuffer确认是否匹配底层已申请的buffer */
	for (i = 0; i < pMSPort->portDefinition.nBufferCountActual; i++) {
        if (pBuffer == pMSPort->bufferHeader[i]) {
            findBuffer = OMX_TRUE;
            break;
        }
    }

	...
	/* 将buffer消息写入到msg中并入列 */
	message = MS_OSAL_Malloc(sizeof(MS_OMX_MESSAGE));
    if (message == NULL) {
        ret = OMX_ErrorInsufficientResources;
        goto EXIT;
    }
    message->messageType = MS_OMX_CommandFillBuffer;
    message->messageParam = (OMX_U32) i;
    message->pCmdData = (OMX_PTR)pBuffer;

    if (pMSComponent->u8DebugLevel) MS_OSAL_Log(MS_LOG_DEBUG, "%s FTB: %lld", pMSComponent->componentName, pBuffer->nTimeStamp);

    MS_OSAL_Queue(&pMSPort->bufferQ, (void *)message);
    MS_OSAL_SemaphorePost(pMSPort->bufferSemID);
}

芯片平台底层是通过消息机制来进行buffer传递的，IL层中只需要将可用buffer的信息写入msg中，并推入到IL层自己实现的消息队列即可，至于buffer出现线程那边，会不停地读取可用msg，然后从msg中读取出可用的buffer进行新一轮的数据填充即可。

六、总结：
一张图剖析OMX_FillThisBuffer和FillBufferDone机制：