Android Camera原理之camera service与camera provider session会话与capture request轮转

《Android Camera架构》
《Android Camera进程间通信类总结》
《Android Camera模块解析之拍照》
《Android Camera模块解析之视频录制》
《Android Camera原理之CameraDeviceCallbacks回调模块》
《Android Camera原理之openCamera模块(一)》
《Android Camera原理之openCamera模块(二)》
《Android Camera原理之createCaptureSession模块》
《Android Camera原理之setRepeatingRequest与capture模块》
《Android Camera原理之编译》
《Android Camera原理之camera provider启动》
《Android Camera原理之cameraserver与cameraprovider是怎样联系的》
《Android Camera原理之camera service与camera provider session会话与capture request轮转》
《Android Camera原理之camera HAL底层数据结构与类总结》
《Android Camera原理之camera service类与接口关系》

上层调用CameraManager.openCamera的时候，会触发底层的一系列反应，之前我们分享过camera framework到camera service之间的调用，但是光看这一块还不够深入，接下来我们讨论一下camera service与camera provider之间在openCamera调用的时候做了什么事情。

status_t Camera3Device::initialize(sp manager, const String8& monitorTags) {
    sp session;
    status_t res = manager->openSession(mId.string(), this,
            /*out*/ &session);
    //......
    res = manager->getCameraCharacteristics(mId.string(), &mDeviceInfo);
    //......
    std::shared_ptr queue;
    auto requestQueueRet = session->getCaptureRequestMetadataQueue(
        [&queue](const auto& descriptor) {
            queue = std::make_shared(descriptor);
            if (!queue->isValid() || queue->availableToWrite() <= 0) {
                ALOGE("HAL returns empty request metadata fmq, not use it");
                queue = nullptr;
                // don't use the queue onwards.
            }
        });
    //......
    std::unique_ptr& resQueue = mResultMetadataQueue;
    auto resultQueueRet = session->getCaptureResultMetadataQueue(
        [&resQueue](const auto& descriptor) {
            resQueue = std::make_unique(descriptor);
            if (!resQueue->isValid() || resQueue->availableToWrite() <= 0) {
                ALOGE("HAL returns empty result metadata fmq, not use it");
                resQueue = nullptr;
                // Don't use the resQueue onwards.
            }
        });
    //......
 
    mInterface = new HalInterface(session, queue);
    std::string providerType;
    mVendorTagId = manager->getProviderTagIdLocked(mId.string());
    mTagMonitor.initialize(mVendorTagId);
    if (!monitorTags.isEmpty()) {
        mTagMonitor.parseTagsToMonitor(String8(monitorTags));
    }
 
    return initializeCommonLocked();
}

上面camera service中执行openCamera 中的核心步骤，可以看出，第一步执行的就是 manager->openSession(mId.string(), this, /out/ &session);

本文就是通过剖析openSession的执行流程来还原 camera service 与camera provider 的执行过程。

Android Camera原理之camera service与camera provider session会话与capture request轮转_第1张图片

CameraProviderManager--openSession流程.jpg

为了让大家看得更加清楚，列出各个文件的位置：
CameraProviderManager : frameworks/av/services/camera/libcameraservice/common/CameraServiceProvider.cpp

CameraDevice : hardware/interfaces/camera/device/3.2/default/CameraDevice.cpp

CameraModule : hardware/interfaces/camera/common/1.0/default/CameraModule.cpp

CameraHAL : hardware/libhardware/modules/camera/3_0/CameraHAL.cpp

Camera : hardware/libhardware/modules/camera/3_0/Camera.cpp

CameraDeviceSession : hardware/interfaces/camera/device/3.2/default/CameraDeviceSession.cpp

中间涉及到一些指针函数的映射，如果看不明白，可以参考：《Android Camera原理之底层数据结构总结》，具体的调用流程就不说了，按照上面的时序图走，都能看明白的。

一些回调关系还是值得说下的。我们看下CameraProviderManager::openSession调用的地方：

status_t CameraProviderManager::openSession(const std::string &id,
        const sp& callback,
        /*out*/
        sp *session) {
 
    std::lock_guard lock(mInterfaceMutex);
 
    auto deviceInfo = findDeviceInfoLocked(id,
            /*minVersion*/ {3,0}, /*maxVersion*/ {4,0});
    if (deviceInfo == nullptr) return NAME_NOT_FOUND;
 
    auto *deviceInfo3 = static_cast(deviceInfo);
 
    Status status;
    hardware::Return ret;
    ret = deviceInfo3->mInterface->open(callback, [&status, &session]
            (Status s, const sp& cameraSession) {
                status = s;
                if (status == Status::OK) {
                    *session = cameraSession;
                }
            });
    if (!ret.isOk()) {
        ALOGE("%s: Transaction error opening a session for camera device %s: %s",
                __FUNCTION__, id.c_str(), ret.description().c_str());
        return DEAD_OBJECT;
    }
    return mapToStatusT(status);
}

我们看下IPC调用的地方：

ret = deviceInfo3->mInterface->open(callback, [&status, &session]
            (Status s, const sp& cameraSession) {
                status = s;
                if (status == Status::OK) {
                    *session = cameraSession;
                }
            });

传入两个参数，一个是 const sp& callback，另一个是open_cb _hidl_cb

callback 提供了 camera HAL层到 camera service的回调。

open_cb _hidl_cb 是硬件抽象层提供了一种IPC间回传数据的方式。就本段代码而言，需要传回两个数据，一个status：表示当前openSession是否成功；另一个是session：表示camera session会话创建成功之后返回的session数据。

CameraDevice::open(...)函数

{
        session = createSession(
                device, info.static_camera_characteristics, callback);
        if (session == nullptr) {
            ALOGE("%s: camera device session allocation failed", __FUNCTION__);
            mLock.unlock();
            _hidl_cb(Status::INTERNAL_ERROR, nullptr);
            return Void();
        }
        if (session->isInitFailed()) {
            ALOGE("%s: camera device session init failed", __FUNCTION__);
            session = nullptr;
            mLock.unlock();
            _hidl_cb(Status::INTERNAL_ERROR, nullptr);
            return Void();
        }
        mSession = session;
 
        IF_ALOGV() {
            session->getInterface()->interfaceChain([](
                ::android::hardware::hidl_vec<::android::hardware::hidl_string> interfaceChain) {
                    ALOGV("Session interface chain:");
                    for (auto iface : interfaceChain) {
                        ALOGV("  %s", iface.c_str());
                    }
                });
        }
        mLock.unlock();
}
_hidl_cb(status, session->getInterface());

最后执行的代码 _hidl_cb(status, session→getInterface()); 当前session创建成功之后，回调到 camera service 中。

const sp& callback 设置到什么地方？这个问题非常重要的，camera 上层很依赖底层的回调，所以我们要搞清楚底层的回调被设置到什么地方，然后在搞清楚在合适的时机触发这些回调。

执行CameraDeviceSession构造函数的时候，传入了这个callback。

CameraDeviceSession::CameraDeviceSession(
    camera3_device_t* device,
    const camera_metadata_t* deviceInfo,
    const sp& callback) :
        camera3_callback_ops({&sProcessCaptureResult, &sNotify}),
        mDevice(device),
        mDeviceVersion(device->common.version),
        mIsAELockAvailable(false),
        mDerivePostRawSensKey(false),
        mNumPartialResults(1),
        mResultBatcher(callback) {
    mDeviceInfo = deviceInfo;
    camera_metadata_entry partialResultsCount =
            mDeviceInfo.find(ANDROID_REQUEST_PARTIAL_RESULT_COUNT);
    if (partialResultsCount.count > 0) {
        mNumPartialResults = partialResultsCount.data.i32[0];
    }
    mResultBatcher.setNumPartialResults(mNumPartialResults);
 
    camera_metadata_entry aeLockAvailableEntry = mDeviceInfo.find(
            ANDROID_CONTROL_AE_LOCK_AVAILABLE);
    if (aeLockAvailableEntry.count > 0) {
        mIsAELockAvailable = (aeLockAvailableEntry.data.u8[0] ==
                ANDROID_CONTROL_AE_LOCK_AVAILABLE_TRUE);
    }
 
    // Determine whether we need to derive sensitivity boost values for older devices.
    // If post-RAW sensitivity boost range is listed, so should post-raw sensitivity control
    // be listed (as the default value 100)
    if (mDeviceInfo.exists(ANDROID_CONTROL_POST_RAW_SENSITIVITY_BOOST_RANGE)) {
        mDerivePostRawSensKey = true;
    }
 
    mInitFail = initialize();
}

CameraDeviceSession 中的 mResultBatcher 类构造中传入了这个 callback，现在由 CameraDeviceSession::ResultBatcher 来持有 callback了。看下ResultBatcher全局代码，在CameraDeviceSession.h中。

那以后底层要回调到上层必定要经过 CameraDeviceSession::ResultBatcher的mCallback来完成了。

class ResultBatcher {
public:
    ResultBatcher(const sp& callback);
    void setNumPartialResults(uint32_t n);
    void setBatchedStreams(const std::vector& streamsToBatch);
    void setResultMetadataQueue(std::shared_ptr q);
 
    void registerBatch(uint32_t frameNumber, uint32_t batchSize);
    void notify(NotifyMsg& msg);
    void processCaptureResult(CaptureResult& result);
 
protected:
    struct InflightBatch {
        // Protect access to entire struct. Acquire this lock before read/write any data or
        // calling any methods. processCaptureResult and notify will compete for this lock
        // HIDL IPCs might be issued while the lock is held
        Mutex mLock;
 
        bool allDelivered() const;
 
        uint32_t mFirstFrame;
        uint32_t mLastFrame;
        uint32_t mBatchSize;
 
        bool mShutterDelivered = false;
        std::vector mShutterMsgs;
 
        struct BufferBatch {
            BufferBatch(uint32_t batchSize) {
                mBuffers.reserve(batchSize);
            }
            bool mDelivered = false;
            // This currently assumes every batched request will output to the batched stream
            // and since HAL must always send buffers in order, no frameNumber tracking is
            // needed
            std::vector mBuffers;
        };
        // Stream ID -> VideoBatch
        std::unordered_map mBatchBufs;
 
        struct MetadataBatch {
            //                   (frameNumber, metadata)
            std::vector> mMds;
        };
        // Partial result IDs that has been delivered to framework
        uint32_t mNumPartialResults;
        uint32_t mPartialResultProgress = 0;
        // partialResult -> MetadataBatch
        std::map mResultMds;
 
        // Set to true when batch is removed from mInflightBatches
        // processCaptureResult and notify must check this flag after acquiring mLock to make
        // sure this batch isn't removed while waiting for mLock
        bool mRemoved = false;
    };
 
 
    // Get the batch index and pointer to InflightBatch (nullptrt if the frame is not batched)
    // Caller must acquire the InflightBatch::mLock before accessing the InflightBatch
    // It's possible that the InflightBatch is removed from mInflightBatches before the
    // InflightBatch::mLock is acquired (most likely caused by an error notification), so
    // caller must check InflightBatch::mRemoved flag after the lock is acquried.
    // This method will hold ResultBatcher::mLock briefly
    std::pair> getBatch(uint32_t frameNumber);
 
    static const int NOT_BATCHED = -1;
 
    // move/push function avoids "hidl_handle& operator=(hidl_handle&)", which clones native
    // handle
    void moveStreamBuffer(StreamBuffer&& src, StreamBuffer& dst);
    void pushStreamBuffer(StreamBuffer&& src, std::vector& dst);
 
    void sendBatchMetadataLocked(
            std::shared_ptr batch, uint32_t lastPartialResultIdx);
 
    // Check if the first batch in mInflightBatches is ready to be removed, and remove it if so
    // This method will hold ResultBatcher::mLock briefly
    void checkAndRemoveFirstBatch();
 
    // The following sendXXXX methods must be called while the InflightBatch::mLock is locked
    // HIDL IPC methods will be called during these methods.
    void sendBatchShutterCbsLocked(std::shared_ptr batch);
    // send buffers for all batched streams
    void sendBatchBuffersLocked(std::shared_ptr batch);
    // send buffers for specified streams
    void sendBatchBuffersLocked(
            std::shared_ptr batch, const std::vector& streams);
   // End of sendXXXX methods
 
    // helper methods
    void freeReleaseFences(hidl_vec&);
    void notifySingleMsg(NotifyMsg& msg);
    void processOneCaptureResult(CaptureResult& result);
    void invokeProcessCaptureResultCallback(hidl_vec &results, bool tryWriteFmq);
 
    // Protect access to mInflightBatches, mNumPartialResults and mStreamsToBatch
    // processCaptureRequest, processCaptureResult, notify will compete for this lock
    // Do NOT issue HIDL IPCs while holding this lock (except when HAL reports error)
    mutable Mutex mLock;
    std::deque> mInflightBatches;
    uint32_t mNumPartialResults;
    std::vector mStreamsToBatch;
    const sp mCallback;
    std::shared_ptr mResultMetadataQueue;
 
    // Protect against invokeProcessCaptureResultCallback()
    Mutex mProcessCaptureResultLock;
 
} mResultBatcher;

到这里，openSession工作就完成了，这个主要是设置了上层的回调到底层，并且底层返回可用的camera session到上层来，实现底层和上层的交互通信。

1.获取的session是什么？为什么这个重要？

此session是 ICameraDeviceSession 对象，这个对象是指为了操作camera device，camera provider与 camera service之间建立的一个会话机制，可以保证camera service IPC调用到camera provider进程中的代码。

1.1.获取session当前请求原数组队列

auto requestQueueRet = session->getCaptureRequestMetadataQueue(
    [&queue](const auto& descriptor) {
        queue = std::make_shared(descriptor);
        if (!queue->isValid() || queue->availableToWrite() <= 0) {
            ALOGE("HAL returns empty request metadata fmq, not use it");
            queue = nullptr;
            // don't use the queue onwards.
        }
    });

到HAL层的 CameraDeviceSession.cpp中调用 getCaptureRequestMetadataQueue

Return CameraDeviceSession::getCaptureRequestMetadataQueue(
    ICameraDeviceSession::getCaptureRequestMetadataQueue_cb _hidl_cb) {
    _hidl_cb(*mRequestMetadataQueue->getDesc());
    return Void();
}

这个mRequestMetadataQueue 是在 CameraDeviceSession::initialize 执行的时候初始化的。

int32_t reqFMQSize = property_get_int32("ro.camera.req.fmq.size", /*default*/-1);
if (reqFMQSize < 0) {
    reqFMQSize = CAMERA_REQUEST_METADATA_QUEUE_SIZE;
} else {
    ALOGV("%s: request FMQ size overridden to %d", __FUNCTION__, reqFMQSize);
}
 
mRequestMetadataQueue = std::make_unique(
        static_cast(reqFMQSize),
        false /* non blocking */);
if (!mRequestMetadataQueue->isValid()) {
    ALOGE("%s: invalid request fmq", __FUNCTION__);
    return true;
}

首先读取 ro.camera.req.fmq.size 属性，如果没有找到，则直接赋给一个 1M 大小的请求原数组队列。这个队列很重要，后续的camera capture请求都是通过这个队列处理的。

1.2.获取session 当前结果原数组队列

这个和请求原数组队列相似，不过结果原数组中保留的是 camera capture的结果数据。大家可以看下源码，这儿就不贴源码了

2.开始运转capture request线程

camera service 与 camera provider 建立session 会话之后，开始运转capture request请求线程，之后发送的capture request都会到这个线程中执行，这就是熟知的capture request轮转。

在Camera3Device::initializeCommonLocked 中执行了 capture request轮转。


/** Start up request queue thread */
mRequestThread = new RequestThread(this, mStatusTracker, mInterface, sessionParamKeys);
res = mRequestThread->run(String8::format("C3Dev-%s-ReqQueue", mId.string()).string());
if (res != OK) {
    SET_ERR_L("Unable to start request queue thread: %s (%d)",
            strerror(-res), res);
    mInterface->close();
    mRequestThread.clear();
    return res;
}

开始启动当前的capture request 队列，放在 RequestThread线程中执行，这个线程会一直执行，当有新的capture request发过来，会将capture request放进当前会话的请求队列中，继续执行。这个轮转很重要，这是camera能正常工作的前提。

轮转的主要工作在Camera3Device::RequestThread::threadLoop 函数中完成，这是native中定义的一个线程执行函数块。


bool Camera3Device::RequestThread::threadLoop() {
    ATRACE_CALL();
    status_t res;
 
    // Handle paused state.
    if (waitIfPaused()) {
        return true;
    }
 
    // Wait for the next batch of requests.
    waitForNextRequestBatch();
    if (mNextRequests.size() == 0) {
        return true;
    }
    //......
 
    // Prepare a batch of HAL requests and output buffers.
    res = prepareHalRequests();
    if (res == TIMED_OUT) {
        // Not a fatal error if getting output buffers time out.
        cleanUpFailedRequests(/*sendRequestError*/ true);
        // Check if any stream is abandoned.
        checkAndStopRepeatingRequest();
        return true;
    } else if (res != OK) {
        cleanUpFailedRequests(/*sendRequestError*/ false);
        return false;
    }
 
    // Inform waitUntilRequestProcessed thread of a new request ID
    {
        Mutex::Autolock al(mLatestRequestMutex);
 
        mLatestRequestId = latestRequestId;
        mLatestRequestSignal.signal();
    }
    //......
 
    bool submitRequestSuccess = false;
    nsecs_t tRequestStart = systemTime(SYSTEM_TIME_MONOTONIC);
    if (mInterface->supportBatchRequest()) {
        submitRequestSuccess = sendRequestsBatch();
    } else {
        submitRequestSuccess = sendRequestsOneByOne();
    }
    //......
 
    return submitRequestSuccess;
}

waitForNextRequestBatch() 不断去轮训底层是否有InputBuffer数据，获取的inputBuffer数据放在request中，这些数据会在之后被消费。
这儿先列个调用过程：对照着代码看一下，camera的producer与consumer模型之后还会详细讲解的。

Android Camera原理之camera service与camera provider session会话与capture request轮转_第2张图片

camera request轮转流程.jpg

这儿也为了大家快速进入代码，也列出来代码的对应位置：

Camera3Device::RequestThread : frameworks/av/services/camera/libcameraservice/device3/Camera3Device.cpp 中有内部类 RequestThread，这是一个线程类。

Camera3Stream : frameworks/av/services/camera/libcameraservice/device3/Camera3Stream.cpp

Camera3InputStream : frameworks/av/services/camera/libcameraservice/device3/Camera3InputStream.cpp

Camera3IOStreamBase : frameworks/av/services/camera/libcameraservice/device3/Camera3IOStreamBase.cpp

BufferItemConsumer : frameworks/native/libs/gui/BufferItemConsumer.cpp

ConsumerBase : frameworks/native/libs/gui/ConsumerBase.cpp

BnGraphicBufferConsumer : frameworks/native/libs/gui/IGraphicBufferConsumer.cpp

上层发过来来的capture request，手下到底层申请Consumer buffer，这个buffer数据存储在capture request缓存中，后期这些buffer数据会被复用，不断地生产数据，也不断地被消费。

capture request开启之后，camera hal层也会受到capture request批处理请求，让camera hal做好准备，开始和camera driver层交互。hal层的请求下一章讲解。

感谢关注公众号JeffMony，持续给你带来音视频方面的知识。

Android Camera原理之camera service与camera provider session会话与capture request轮转

1.获取的session是什么？为什么这个重要？

1.1.获取session当前请求原数组队列

1.2.获取session 当前结果原数组队列

2.开始运转capture request线程

你可能感兴趣的:(Android Camera原理之camera service与camera provider session会话与capture request轮转)