Android Camera 原理之拍照流程zsl优化方案

一、背景介绍

拍照的手机基本的功能，优化拍照性能，主要是优化点击拍照到生成照片的这一段时间，看看可以在什么地方减少耗时。下面将打开camera到拍照完成这段时间拆解一下。

普通拍照流程与优化流程.jpg

这段过程主要分为：

• capture session配置阶段：这是预览之前的阶段。
• 预览流程：这段时间，camera不断出帧，显示在TextureView 上。
• 拍照流程：点击拍照到最终生效图片的流程。

Note：将预览流程与拍照流程合成一个大的流程，因为我们本文所说的优化重点就在这里。

二、核心思想

---

预览出帧是为了让用户感觉到此时camera正在运行，但是预览的帧数据是不能直接用作拍照的帧数据，为什么？因为预览的帧数据太小，拍照的帧数据很大，所以不能直接复用。那如果能直接复用呢？就是预览的帧数据可以直接被拍照来使用。
这也是我们本文讨论的重点，直接复用预览的帧数据。

---

直接复用预览的帧数据，那么首先需要保证的是 预览帧的大小必须和 实际拍照的帧大小是相同的，不然获取的预览帧数据也是没用的，没有意义。
预览的surface我们需要自定义，而且大小要和拍照的ImageReader的surface大小相同的。

2.1 定义Yuv Full ImageReader

private ImageReader mYuv1ImageReader;

初始化的时候需要创建这个 ImageReader的实例：

        mYuv1ImageReader = ImageReader.newInstance(
                mCameraInfoCache.getYuvStream1Size().getWidth(),
                mCameraInfoCache.getYuvStream1Size().getHeight(),
                ImageFormat.YUV_420_888,
                YUV1_IMAGEREADER_SIZE);
        mYuv1ImageReader.setOnImageAvailableListener(mYuv1ImageListener, mOpsHandler);

2.2 ImageReader的监听回调

    ImageReader.OnImageAvailableListener mYuv1ImageListener =
            new ImageReader.OnImageAvailableListener() {
                @Override
                public void onImageAvailable(ImageReader reader) {
                    Image img = reader.acquireLatestImage();
                    if (img == null) {
                        Log.e(TAG, "Null image returned YUV1");
                        return;
                    }
                    if (mYuv1LastReceivedImage != null) {
                        mYuv1LastReceivedImage.close();
                    }
                    mYuv1LastReceivedImage = img;
                    if (++mYuv1ImageCounter % LOG_NTH_FRAME == 0) {
                        Log.v(TAG, "YUV1 buffer available, Frame #=" + mYuv1ImageCounter + " w=" + img.getWidth() + " h=" + img.getHeight() + " time=" + img.getTimestamp());
                    }

                }
            };

只要是处于预览状态，底层的sensor会一直出帧数据，这个onImageAvailable(ImageReader reader)会一直回调，发现我们在其中又定义了一个Image变量。

2.3 定义实时的Image返回值

    // Handle to last received Image: allows ZSL to be implemented.
    private Image mYuv1LastReceivedImage = null;

这个mYuv1LastReceivedImage从定义的变量名上就能看出来，是预览的最后一帧的数据，显然这个帧数据是完全的，和出图的大小完全一样的。

mYuv1LastReceivedImage保证本地总是存储预览的最后一帧数据。

2.4 创建captureSession

Camera打开的时候onOpened回调的时候，开始创建captureSession：

    private CameraDevice.StateCallback mCameraStateCallback = new LoggingCallbacks.DeviceStateCallback() {
        @Override
        public void onOpened(CameraDevice camera) {
            super.onOpened(camera);
            startCaptureSession();
        }
    };

    // Create CameraCaptureSession. Callback will start repeating request with current parameters.
    private void startCaptureSession() {
        Log.v(TAG, "Configuring session..");
        List outputSurfaces = new ArrayList(4);

        outputSurfaces.add(mPreviewSurface);
        Log.v(TAG, "  .. added SurfaceView " + mCameraInfoCache.getPreviewSize().getWidth() +
                " x " + mCameraInfoCache.getPreviewSize().getHeight());

        outputSurfaces.add(mYuv1ImageReader.getSurface());
        Log.v(TAG, "  .. added YUV ImageReader " + mCameraInfoCache.getYuvStream1Size().getWidth() +
                " x " + mCameraInfoCache.getYuvStream1Size().getHeight());

        if (mIsDepthCloudSupported) {
            outputSurfaces.add(mDepthCloudImageReader.getSurface());
            Log.v(TAG, "  .. added Depth cloud ImageReader");
        }

        if (SECOND_YUV_IMAGEREADER_STREAM) {
            outputSurfaces.add(mYuv2ImageReader.getSurface());
            Log.v(TAG, "  .. added YUV ImageReader " + mCameraInfoCache.getYuvStream2Size().getWidth() +
                    " x " + mCameraInfoCache.getYuvStream2Size().getHeight());
        }

        if (SECOND_SURFACE_TEXTURE_STREAM) {
            outputSurfaces.add(mSurfaceTextureSurface);
            Log.v(TAG, "  .. added SurfaceTexture");
        }

        if (RAW_STREAM_ENABLE && mCameraInfoCache.rawAvailable()) {
            outputSurfaces.add(mRawImageReader.getSurface());
            Log.v(TAG, "  .. added Raw ImageReader " + mCameraInfoCache.getRawStreamSize().getWidth() +
                    " x " + mCameraInfoCache.getRawStreamSize().getHeight());
        }

        if (USE_REPROCESSING_IF_AVAIL && mCameraInfoCache.isYuvReprocessingAvailable()) {
            outputSurfaces.add(mJpegImageReader.getSurface());
            Log.v(TAG, "  .. added JPEG ImageReader " + mCameraInfoCache.getJpegStreamSize().getWidth() +
                    " x " + mCameraInfoCache.getJpegStreamSize().getHeight());
        }

        try {
            if (USE_REPROCESSING_IF_AVAIL && mCameraInfoCache.isYuvReprocessingAvailable()) {
                InputConfiguration inputConfig = new InputConfiguration(mCameraInfoCache.getYuvStream1Size().getWidth(),
                        mCameraInfoCache.getYuvStream1Size().getHeight(), ImageFormat.YUV_420_888);
                mCameraDevice.createReprocessableCaptureSession(inputConfig, outputSurfaces,
                        mSessionStateCallback, null);
                Log.v(TAG, "  Call to createReprocessableCaptureSession complete.");
            } else {
                mCameraDevice.createCaptureSession(outputSurfaces, mSessionStateCallback, null);
                Log.v(TAG, "  Call to createCaptureSession complete.");
            }

        } catch (CameraAccessException e) {
            Log.e(TAG, "Error configuring ISP.");
        }
    }

使用zsl的方式的话，就需要输入InputConfiguration配置数据，好让底层的camera hal复用这部分数据，我们也能真正达到zsl的目的。

                InputConfiguration inputConfig = new InputConfiguration(mCameraInfoCache.getYuvStream1Size().getWidth(),
                        mCameraInfoCache.getYuvStream1Size().getHeight(), ImageFormat.YUV_420_888);
                mCameraDevice.createReprocessableCaptureSession(inputConfig, outputSurfaces,
                        mSessionStateCallback, null);

mSessionStateCallback是当前captureSession所处状态的回调，我们会在captureSession的onReady回调函数中设置ImageWriter对象：

    ImageWriter mImageWriter;

    private CameraCaptureSession.StateCallback mSessionStateCallback = new LoggingCallbacks.SessionStateCallback() {
        @Override
        public void onReady(CameraCaptureSession session) {
            Log.v(TAG, "capture session onReady().  HAL capture session took: (" + (SystemClock.elapsedRealtime() - CameraTimer.t_session_go) + " ms)");
            mCurrentCaptureSession = session;
            issuePreviewCaptureRequest(false);

            if (session.isReprocessable()) {
                mImageWriter = ImageWriter.newInstance(session.getInputSurface(), IMAGEWRITER_SIZE);
                mImageWriter.setOnImageReleasedListener(
                        new ImageWriter.OnImageReleasedListener() {
                            @Override
                            public void onImageReleased(ImageWriter writer) {
                                Log.v(TAG, "ImageWriter.OnImageReleasedListener onImageReleased()");
                            }
                        }, null);
                Log.v(TAG, "Created ImageWriter.");
            }
            super.onReady(session);
        }
    };

session.getInputSurface() 表示之前输入的inputConfiguration数据，这个数据暂时初始化放在ImageWriter中。后续每次得到的预览的最后一帧数据都会放在ImageWriter对象中，直接送入到底层。

2.5 设置预览

        try {
            CaptureRequest.Builder b1 = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
            b1.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_USE_SCENE_MODE);
            b1.set(CaptureRequest.CONTROL_SCENE_MODE, CameraMetadata.CONTROL_SCENE_MODE_FACE_PRIORITY);
            if (AFtrigger) {
                b1.set(CaptureRequest.CONTROL_AF_MODE, CameraMetadata.CONTROL_AF_MODE_AUTO);
            } else {
                b1.set(CaptureRequest.CONTROL_AF_MODE, CameraMetadata.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
            }

            b1.set(CaptureRequest.NOISE_REDUCTION_MODE, mCaptureNoiseMode);
            b1.set(CaptureRequest.EDGE_MODE, mCaptureEdgeMode);
            b1.set(CaptureRequest.STATISTICS_FACE_DETECT_MODE, mCaptureFace ? mCameraInfoCache.bestFaceDetectionMode() : CaptureRequest.STATISTICS_FACE_DETECT_MODE_OFF);

            Log.v(TAG, "  .. NR=" + mCaptureNoiseMode + "  Edge=" + mCaptureEdgeMode + "  Face=" + mCaptureFace);

            if (mCaptureYuv1) {
                b1.addTarget(mYuv1ImageReader.getSurface());
                Log.v(TAG, "  .. YUV1 on");
            }

            if (mCaptureRaw) {
                b1.addTarget(mRawImageReader.getSurface());
            }

            b1.addTarget(mPreviewSurface);

            if (mIsDepthCloudSupported && !mCaptureYuv1 && !mCaptureYuv2 && !mCaptureRaw) {
                b1.addTarget(mDepthCloudImageReader.getSurface());
            }

            if (mCaptureYuv2) {
                if (SECOND_SURFACE_TEXTURE_STREAM) {
                    b1.addTarget(mSurfaceTextureSurface);
                }
                if (SECOND_YUV_IMAGEREADER_STREAM) {
                    b1.addTarget(mYuv2ImageReader.getSurface());
                }
                Log.v(TAG, "  .. YUV2 on");
            }

            if (AFtrigger) {
                b1.set(CaptureRequest.CONTROL_AF_TRIGGER, CameraMetadata.CONTROL_AF_TRIGGER_START);
                mCurrentCaptureSession.capture(b1.build(), mCaptureCallback, mOpsHandler);
                b1.set(CaptureRequest.CONTROL_AF_TRIGGER, CameraMetadata.CONTROL_AF_TRIGGER_IDLE);
            }
            mCurrentCaptureSession.setRepeatingRequest(b1.build(), mCaptureCallback, mOpsHandler);
        } catch (CameraAccessException e) {
            Log.e(TAG, "Could not access camera for issuePreviewCaptureRequest.");
        }

这儿很多代码，核心的代码只有3行：

CaptureRequest.Builder b1 = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
b1.addTarget(mYuv1ImageReader.getSurface());
mCurrentCaptureSession.setRepeatingRequest(b1.build(), mCaptureCallback, mOpsHandler);

传入了初始定义的full yuv的ImageReader的surface结构，然后在CaptureCallback中需要获取captureResult，这个数据在拍照的时候还有用处。

2.6 CaptureCallback处理

    private CameraCaptureSession.CaptureCallback mCaptureCallback = new LoggingCallbacks.SessionCaptureCallback() {
        @Override
        public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request, TotalCaptureResult result) {
            if (!mFirstFrameArrived) {
                mFirstFrameArrived = true;
                long now = SystemClock.elapsedRealtime();
                long dt = now - CameraTimer.t0;
                long camera_dt = now - CameraTimer.t_session_go + CameraTimer.t_open_end - CameraTimer.t_open_start;
                long repeating_req_dt = now - CameraTimer.t_burst;
                Log.v(TAG, "App control to first frame: (" + dt + " ms)");
                Log.v(TAG, "HAL request to first frame: (" + repeating_req_dt + " ms) " + " Total HAL wait: (" + camera_dt + " ms)");
                mMyCameraCallback.receivedFirstFrame();
                mMyCameraCallback.performanceDataAvailable((int) dt, (int) camera_dt, null);
            }
            publishFrameData(result);
            // Used for reprocessing.
            mLastTotalCaptureResult = result;
            super.onCaptureCompleted(session, request, result);
        }
    };

这个mLastTotalCaptureResult是预览capture的时候捕获的一个captureResult，后续处理的时候会用到

    // Last total capture result
    TotalCaptureResult mLastTotalCaptureResult;

2.7 拍照处理

终于来到了最核心的步骤，这儿的拍照处理，当然不会像之前那样直接调用CaptureSession的capture方法，因为执行capture方法，就必定要重新发送capture request，重新获取帧数据。
但是我们现在已经有了帧数据，就是之前保存的帧数据，这时候帧数据就起到了非常重要的作用。

    void runReprocessing() {
        if (mYuv1LastReceivedImage == null) {
            Log.e(TAG, "No YUV Image available.");
            return;
        }
        mImageWriter.queueInputImage(mYuv1LastReceivedImage);
        Log.v(TAG, "  Sent YUV1 image to ImageWriter.queueInputImage()");
        try {
            CaptureRequest.Builder b1 = mCameraDevice.createReprocessCaptureRequest(mLastTotalCaptureResult);
            // Todo: Read current orientation instead of just assuming device is in native orientation
            b1.set(CaptureRequest.JPEG_ORIENTATION, mCameraInfoCache.sensorOrientation());
            b1.set(CaptureRequest.JPEG_QUALITY, (byte) 95);
            b1.set(CaptureRequest.NOISE_REDUCTION_MODE, mReprocessingNoiseMode);
            b1.set(CaptureRequest.EDGE_MODE, mReprocessingEdgeMode);
            b1.addTarget(mJpegImageReader.getSurface());
            mCurrentCaptureSession.capture(b1.build(), mReprocessingCaptureCallback, mOpsHandler);
            mReprocessingRequestNanoTime = System.nanoTime();
        } catch (CameraAccessException e) {
            Log.e(TAG, "Could not access camera for issuePreviewCaptureRequest.");
        }
        mYuv1LastReceivedImage = null;
        Log.v(TAG, "  Reprocessing request submitted.");
    }

mImageWriter.queueInputImage(mYuv1LastReceivedImage);将预览最后一帧数据放入ImageWriter的input 队列中。

    // Reprocessing capture completed.
    private CameraCaptureSession.CaptureCallback mReprocessingCaptureCallback = new LoggingCallbacks.SessionCaptureCallback() {
        @Override
        public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request, TotalCaptureResult result) {
            Log.v(TAG, "Reprocessing onCaptureCompleted()");
        }
    };

处理完成之后回调onCaptureCompleted(...)函数。

三、总结

zsl方案有多快：原图拍照一张150ms，快得一笔
下面是截图样例：

test_camera.gif

---

优化之后的流程可以总结成如下：

普通拍照流程与优化流程2.jpg

---