《Android Camera架构》
《Android Camera进程间通信类总结》
《Android Camera模块解析之拍照》
《Android Camera模块解析之视频录制》
《Android Camera原理之CameraDeviceCallbacks回调模块》
《Android Camera原理之openCamera模块(一)》
《Android Camera原理之openCamera模块(二)》
《Android Camera原理之createCaptureSession模块》
《Android Camera原理之setRepeatingRequest与capture模块》
《Android Camera原理之编译》
《Android Camera原理之camera provider启动》
《Android Camera原理之cameraserver与cameraprovider是怎样联系的》
《Android Camera原理之camera service与camera provider session会话与capture request轮转》
《Android Camera原理之camera HAL底层数据结构与类总结》
《Android Camera原理之camera service类与接口关系》
Camera操作过程中最重要的四个步骤：
- CameraManager-->openCamera ---> 打开相机
- CameraDeviceImpl-->createCaptureSession ---> 创建捕获会话
- CameraCaptureSession-->setRepeatingRequest ---> 设置预览界面
- CameraDeviceImpl-->capture ---> 开始捕获图片
之前我们介绍了openCamera流程和createCaptureSession流程，如下：
《Android Camera原理之openCamera模块(一)》
《Android Camera原理之openCamera模块(二)》
《Android Camera原理之createCaptureSession模块》
至此，Camera 会话已经创建成功，接下来我们可以开始预览了，预览回调onCaptureCompleted之后就可以拍照(回调到onCaptureCompleted，说明capture 完整frame数据已经返回了，可以捕捉其中的数据了。)，由于预览和拍照的很多流程很相似，拍照只是预览过程中的一个节点，所以我们把预览和拍照放在一文里讲解。
1.预览
预览发起的函数就是CameraCaptureSession-->setRepeatingRequest，本文我们就谈一下Camera 是如何发起预览操作的。
CameraCaptureSession-->setRepeatingRequest是createCaptureSession(List中输出流配置成功之后执行CameraCaptureSession.StateCallback.onConfigured(@NonNull CameraCaptureSession session)函数中执行的。
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()),
new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(@NonNull CameraCaptureSession cameraCaptureSession) {
// The camera is already closed
if (null == mCameraDevice) {
return;
}
// When the session is ready, we start displaying the preview.
mCaptureSession = cameraCaptureSession;
try {
// Auto focus should be continuous for camera preview.
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE,
CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
// Flash is automatically enabled when necessary.
setAutoFlash(mPreviewRequestBuilder);
// Finally, we start displaying the camera preview.
mPreviewRequest = mPreviewRequestBuilder.build();
mCaptureSession.setRepeatingRequest(mPreviewRequest,
mCaptureCallback, mBackgroundHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
@Override
public void onConfigureFailed(
@NonNull CameraCaptureSession cameraCaptureSession) {
showToast("Failed");
}
}, null
);
最终执行了
mCaptureSession.setRepeatingRequest(mPreviewRequest, mCaptureCallback, mBackgroundHandler);
来执行camera preview操作。像对焦等操作就可以在这个onConfigured回调中完成。
onConfigured回调表示当前的配置流已经完成，相机已经显示出来了，可以预览了。onConfigureFailed配置流失败，相机黑屏。
public int setRepeatingRequest(CaptureRequest request, CaptureCallback callback,
Handler handler) throws CameraAccessException {
checkRepeatingRequest(request);
synchronized (mDeviceImpl.mInterfaceLock) {
checkNotClosed();
handler = checkHandler(handler, callback);
return addPendingSequence(mDeviceImpl.setRepeatingRequest(request,
createCaptureCallbackProxy(handler, callback), mDeviceExecutor));
}
}
- 第一个参数CaptureRequest 标识当前capture 请求的属性，是请求一个camera还是多个camera，是否复用之前的请求等等。
- 第二个参数CaptureCallback 是捕捉回调，这是开发者直接接触的回调。
public interface CaptureCallback {
public static final int NO_FRAMES_CAPTURED = -1;
public void onCaptureStarted(CameraDevice camera,
CaptureRequest request, long timestamp, long frameNumber);
public void onCapturePartial(CameraDevice camera,
CaptureRequest request, CaptureResult result);
public void onCaptureProgressed(CameraDevice camera,
CaptureRequest request, CaptureResult partialResult);
public void onCaptureCompleted(CameraDevice camera,
CaptureRequest request, TotalCaptureResult result);
public void onCaptureFailed(CameraDevice camera,
CaptureRequest request, CaptureFailure failure);
public void onCaptureSequenceCompleted(CameraDevice camera,
int sequenceId, long frameNumber);
public void onCaptureSequenceAborted(CameraDevice camera,
int sequenceId);
public void onCaptureBufferLost(CameraDevice camera,
CaptureRequest request, Surface target, long frameNumber);
}
这需要开发者自己实现，这些回调是如何调用到上层的，请看《Android Camera原理之CameraDeviceCallbacks回调模块》，都是通过CameraDeviceCallbacks回调调上来的。
下面我们从camera 调用原理的角度分析一下
mCaptureSession.setRepeatingRequest
--->CameraDeviceImpl.setRepeatingRequest
--->CameraDeviceImpl.submitCaptureRequest
其中CameraDeviceImpl.setRepeatingRequest中第3个参数传入的是true。之所以这个强调一点，因为接下来执行CameraDeviceImpl.capture的时候也会执行setRepeatingRequest，这里第3个参数传入的就是false。第3个参数boolean repeating如果为true，表示当前捕获的是一个过程，camera frame不断在填充；如果为false，表示当前捕获的是一个瞬间，就是拍照。
public int setRepeatingRequest(CaptureRequest request, CaptureCallback callback,
Executor executor) throws CameraAccessException {
List requestList = new ArrayList();
requestList.add(request);
return submitCaptureRequest(requestList, callback, executor, /*streaming*/true);
}
private int submitCaptureRequest(List requestList, CaptureCallback callback,
Executor executor, boolean repeating) {
//......
}
CameraDeviceImpl.submitCaptureRequest核心工作就是3步：
- 1.验证当前CaptureRequest列表中的request是否合理：核心就是验证与request绑定的Surface是否存在。
- 2.向底层发送请求信息。
- 3.将底层返回的请求信息和传入的CaptureCallback 绑定，以便后续正确回调。
而这三步中，第二步却是核心工作。
1.1 向底层发送captureRequest请求
SubmitInfo requestInfo;
CaptureRequest[] requestArray = requestList.toArray(new CaptureRequest[requestList.size()]);
// Convert Surface to streamIdx and surfaceIdx
for (CaptureRequest request : requestArray) {
request.convertSurfaceToStreamId(mConfiguredOutputs);
}
requestInfo = mRemoteDevice.submitRequestList(requestArray, repeating);
if (DEBUG) {
Log.v(TAG, "last frame number " + requestInfo.getLastFrameNumber());
}
for (CaptureRequest request : requestArray) {
request.recoverStreamIdToSurface();
}
- 执行
request.convertSurfaceToStreamId(mConfiguredOutputs);将本地已经缓存的surface和stream记录在内存中，并binder传输到camera service层中，防止camera service端重复请求。requestInfo = mRemoteDevice.submitRequestList(requestArray, repeating);这儿直接调用到camera service端。这儿需要重点讲解一下的。request.recoverStreamIdToSurface();回调成功，清除之前在内存中的数据。
CameraDeviceClient::submitRequest
--->CameraDeviceClient::submitRequestList
这个函数代码很多，前面很多执行都是在复用检索之前的缓存是否可用，我们关注一下核心的执行：预览的情况下传入的streaming是true，执行上面；如果是拍照的话，那就执行下面的else。
err = mDevice->setStreamingRequestList(metadataRequestList, surfaceMapList, &(submitInfo->mLastFrameNumber));
传入的submitInfo就是要返回上层的回调参数，如果是预览状态，需要不断更新当前的的frame数据，所以每次更新最新的frame number。
if (streaming) {
err = mDevice->setStreamingRequestList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
if (err != OK) {
String8 msg = String8::format(
"Camera %s: Got error %s (%d) after trying to set streaming request",
mCameraIdStr.string(), strerror(-err), err);
ALOGE("%s: %s", __FUNCTION__, msg.string());
res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION,
msg.string());
} else {
Mutex::Autolock idLock(mStreamingRequestIdLock);
mStreamingRequestId = submitInfo->mRequestId;
}
} else {
err = mDevice->captureList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
if (err != OK) {
String8 msg = String8::format(
"Camera %s: Got error %s (%d) after trying to submit capture request",
mCameraIdStr.string(), strerror(-err), err);
ALOGE("%s: %s", __FUNCTION__, msg.string());
res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION,
msg.string());
}
ALOGV("%s: requestId = %d ", __FUNCTION__, submitInfo->mRequestId);
}
Camera3Device::setStreamingRequestList
--->Camera3Device::submitRequestsHelper
status_t Camera3Device::submitRequestsHelper(
const List &requests,
const std::list &surfaceMaps,
bool repeating,
/*out*/
int64_t *lastFrameNumber) {
ATRACE_CALL();
Mutex::Autolock il(mInterfaceLock);
Mutex::Autolock l(mLock);
status_t res = checkStatusOkToCaptureLocked();
if (res != OK) {
// error logged by previous call
return res;
}
RequestList requestList;
res = convertMetadataListToRequestListLocked(requests, surfaceMaps,
repeating, /*out*/&requestList);
if (res != OK) {
// error logged by previous call
return res;
}
if (repeating) {
res = mRequestThread->setRepeatingRequests(requestList, lastFrameNumber);
} else {
res = mRequestThread->queueRequestList(requestList, lastFrameNumber);
}
//......
return res;
}
预览的时候会执行mRequestThread->setRepeatingRequests(requestList, lastFrameNumber);
拍照的时候执行mRequestThread->queueRequestList(requestList, lastFrameNumber);
mRequestThread->setRepeatingRequests
status_t Camera3Device::RequestThread::setRepeatingRequests(
const RequestList &requests,
/*out*/
int64_t *lastFrameNumber) {
ATRACE_CALL();
Mutex::Autolock l(mRequestLock);
if (lastFrameNumber != NULL) {
*lastFrameNumber = mRepeatingLastFrameNumber;
}
mRepeatingRequests.clear();
mRepeatingRequests.insert(mRepeatingRequests.begin(),
requests.begin(), requests.end());
unpauseForNewRequests();
mRepeatingLastFrameNumber = hardware::camera2::ICameraDeviceUser::NO_IN_FLIGHT_REPEATING_FRAMES;
return OK;
}
将当前提交的CaptureRequest请求放入之前的预览请求队列中，告知HAL层有新的request请求，HAL层连接请求开始工作，源源不断地输出信息到上层。这儿是跑在Camera3Device中定义的RequestThread线程中，可以保证在预览的时候不断地捕获信息流，camera就不断处于预览的状态了。
1.2 将返回请求信息和 CaptureCallback 绑定
if (callback != null) {
mCaptureCallbackMap.put(requestInfo.getRequestId(),
new CaptureCallbackHolder(
callback, requestList, executor, repeating, mNextSessionId - 1));
} else {
if (DEBUG) {
Log.d(TAG, "Listen for request " + requestInfo.getRequestId() + " is null");
}
}
/** map request IDs to callback/request data */
private final SparseArray mCaptureCallbackMap =
new SparseArray();
1.向底层发送captureRequest请求--->回调的requestIinfo表示当前capture request的结果，将requestInfo.getRequestId()与CaptureCallbackHolder绑定，因为Camera 2架构支持发送多次CaptureRequest请求，如果不使用这种绑定机制，后续的回调会造成严重的错乱，甚至回调不上来，那么开发者无法继续使用了。
我们看看使用这些回调的地方的代码：
《Android Camera原理之CameraDeviceCallbacks回调模块》已经说明了CameraDeviceCallbacks.aidl才是camera service进程与用户进程通信的回调，到这个回调里面，再取出CaptureRequest绑定的CaptureCallback回调，调用到CaptureCallback回调函数，这样开发者可以直接使用。
下面看看CameraDeviceCallbacks的onCaptureStarted回调---->
public void onCaptureStarted(final CaptureResultExtras resultExtras, final long timestamp) {
int requestId = resultExtras.getRequestId();
final long frameNumber = resultExtras.getFrameNumber();
if (DEBUG) {
Log.d(TAG, "Capture started for id " + requestId + " frame number " + frameNumber);
}
final CaptureCallbackHolder holder;
synchronized(mInterfaceLock) {
if (mRemoteDevice == null) return; // Camera already closed
// Get the callback for this frame ID, if there is one
holder = CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId);
if (holder == null) {
return;
}
if (isClosed()) return;
// Dispatch capture start notice
final long ident = Binder.clearCallingIdentity();
try {
holder.getExecutor().execute(
new Runnable() {
@Override
public void run() {
if (!CameraDeviceImpl.this.isClosed()) {
final int subsequenceId = resultExtras.getSubsequenceId();
final CaptureRequest request = holder.getRequest(subsequenceId);
if (holder.hasBatchedOutputs()) {
// Send derived onCaptureStarted for requests within the
// batch
final Range fpsRange =
request.get(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE);
for (int i = 0; i < holder.getRequestCount(); i++) {
holder.getCallback().onCaptureStarted(
CameraDeviceImpl.this,
holder.getRequest(i),
timestamp - (subsequenceId - i) *
NANO_PER_SECOND/fpsRange.getUpper(),
frameNumber - (subsequenceId - i));
}
} else {
holder.getCallback().onCaptureStarted(
CameraDeviceImpl.this,
holder.getRequest(resultExtras.getSubsequenceId()),
timestamp, frameNumber);
}
}
}
});
} finally {
Binder.restoreCallingIdentity(ident);
}
}
}
holder = CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId);然后直接调用
holder.getCallback().onCaptureStarted(
CameraDeviceImpl.this,
holder.getRequest(i),
timestamp - (subsequenceId - i) *
NANO_PER_SECOND/fpsRange.getUpper(),
frameNumber - (subsequenceId - i));
简单明了，脉络清楚。
2.拍照
开发者如果想要拍照的话，直接调用
mCaptureSession.capture(mPreviewRequestBuilder.build(), mCaptureCallback, mBackgroundHandler);
拍照的调用流程和预览很相似，只是在调用函数中个传入的参数不同。
public int capture(CaptureRequest request, CaptureCallback callback, Executor executor)
throws CameraAccessException {
if (DEBUG) {
Log.d(TAG, "calling capture");
}
List requestList = new ArrayList();
requestList.add(request);
return submitCaptureRequest(requestList, callback, executor, /*streaming*/false);
}
拍照的时候也是调用submitCaptureRequest，只不过第3个参数传入的是false，表示不用循环获取HAL调用上来的帧数据，只获取瞬间的帧数据就可以。
拍照和预览调用的区分在：CameraDeviceClient::submitRequestList
if (streaming) {
//......
} else {
err = mDevice->captureList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
if (err != OK) {
String8 msg = String8::format(
"Camera %s: Got error %s (%d) after trying to submit capture request",
mCameraIdStr.string(), strerror(-err), err);
ALOGE("%s: %s", __FUNCTION__, msg.string());
res = STATUS_ERROR(CameraService::ERROR_INVALID_OPERATION,
msg.string());
}
ALOGV("%s: requestId = %d ", __FUNCTION__, submitInfo->mRequestId);
}
接下里调用到
mDevice->captureList
--->Camera3Device::submitRequestsHelper
status_t Camera3Device::submitRequestsHelper(
const List &requests,
const std::list &surfaceMaps,
bool repeating,
/*out*/
int64_t *lastFrameNumber) {
//......
RequestList requestList;
//......
if (repeating) {
res = mRequestThread->setRepeatingRequests(requestList, lastFrameNumber);
} else {
res = mRequestThread->queueRequestList(requestList, lastFrameNumber);
}
//......
return res;
}
执行Camera3Device::RequestThread线程中的queueRequestList。
status_t Camera3Device::RequestThread::queueRequestList(
List > &requests,
/*out*/
int64_t *lastFrameNumber) {
ATRACE_CALL();
Mutex::Autolock l(mRequestLock);
for (List >::iterator it = requests.begin(); it != requests.end();
++it) {
mRequestQueue.push_back(*it);
}
if (lastFrameNumber != NULL) {
*lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1;
ALOGV("%s: requestId %d, mFrameNumber %" PRId32 ", lastFrameNumber %" PRId64 ".",
__FUNCTION__, (*(requests.begin()))->mResultExtras.requestId, mFrameNumber,
*lastFrameNumber);
}
unpauseForNewRequests();
return OK;
}
*lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1;
这里有关键的执行代码，表示当前取最新的capture frame数据。
拍照的时候在什么地方捕捉image？
camera1的时候提供了PictureCallback回调方式来提供实时预览回调，可以在这里获取image数据回调。
camera2没有这个接口，但是提供了ImageReader.OnImageAvailableListener来实现回调。
public interface OnImageAvailableListener {
/**
* Callback that is called when a new image is available from ImageReader.
*
* @param reader the ImageReader the callback is associated with.
* @see ImageReader
* @see Image
*/
void onImageAvailable(ImageReader reader);
}
还记得《Android Camera模块解析之拍照》中提到openCamera之前要设置
mImageReader = ImageReader.newInstance(largest.getWidth(), largest.getHeight(),
ImageFormat.JPEG, /*maxImages*/2);
mImageReader.setOnImageAvailableListener(
mOnImageAvailableListener, mBackgroundHandler);
ImageReader中有一个getSurface()函数，这是ImageReader的拍照输出流，我们拍照的时候一般有两个输出流(outputSurface对象)，一个是预览流，还有一个是拍照流。不记得可以参考《Android Camera原理之createCaptureSession模块》，ImageReader设置的拍照流会设置到camera service端。
public Surface getSurface() {
return mSurface;
}
我们看看在什么时候回调这个接口。
看上面的调用流程，调用到
ImageReader.OnImageAvailableListener->onImageAvailable中，我们获取
ImageReader->acquireNextImage可以获取采集的image图片。其实ImageReader中也可以获取预览的流式数据。SurfacePlane 封装了返回的ByteBuffer数据，可供开发者实时获取。
private class SurfacePlane extends android.media.Image.Plane {
private SurfacePlane(int rowStride, int pixelStride, ByteBuffer buffer) {
mRowStride = rowStride;
mPixelStride = pixelStride;
mBuffer = buffer;
/**
* Set the byteBuffer order according to host endianness (native
* order), otherwise, the byteBuffer order defaults to
* ByteOrder.BIG_ENDIAN.
*/
mBuffer.order(ByteOrder.nativeOrder());
}
@Override
public ByteBuffer getBuffer() {
throwISEIfImageIsInvalid();
return mBuffer;
}
final private int mPixelStride;
final private int mRowStride;
private ByteBuffer mBuffer;
}
Note：很多开发者在camera1使用Camera.PreviewCallback的
void onPreviewFrame(byte[] data, Camera camera)
可以获取实时数据，但是在camera2中没有这个接口了，虽然camera1的接口方法也能用，camera2替代的接口就是ImageReader.OnImageAvailableListener->onImageAvailable
感谢关注公众号JeffMony，持续给你带来音视频方面的知识。
