Multi-Camera Support

Android 9 introduces API support for multi-camera devices via a new logical camera device composed of two or more physical camera devices pointing in the same direction. The logical camera device is exposed as a single CameraDevice/CaptureSession to an application allowing for interaction with HAL-integrated multi-camera features. Applications can optionally access and control underlying physical camera streams, metadata, and controls.

                          Figure 1. Multi-camera support

In this diagram, different camera IDs are color coded. The application can stream raw buffers from each physical camera at the same time. It is also possible to set separate controls and receive separate metadata from different physical cameras.

Examples and sources

Multi-camera devices must be advertised via the logical multi-camera capability.

Camera clients can query the camera ID of the physical devices a particular logical camera is made of by callinggetPhysicalCameraIds(). The IDs returned as part of the result are then used to control physical devices individually via setPhysicalCameraId(). The results from such individual requests can be queried from the complete result by invoking getPhysicalCameraResults().

Individual physical camera requests may support only a limited subset of parameters. To receive a list of the supported parameters, developers can call getAvailablePhysicalCameraRequestKeys().

Physical camera streams are supported only for non-reprocessing requests and only for monochrome and bayer sensors.

Implementation

Support checklist

To add logical multi-camera devices on the HAL side:

• Add a ANDROID_REQUEST_AVAILABLE_CAPABILITIES_LOGICAL_MULTI_CAMERA capability for any logical camera device backed by two or more physical cameras that are also exposed to an application.
• Populate the static ANDROID_LOGICAL_MULTI_CAMERA_PHYSICAL_IDS metadata field with a list of physical camera IDs.
• Populate the depth-related static metadata required to correlate between physical camera streams' pixels:ANDROID_LENS_POSE_ROTATION, ANDROID_LENS_POSE_TRANSLATION,ANDROID_LENS_INTRINSIC_CALIBRATION, ANDROID_LENS_DISTORTION,ANDROID_LENS_POSE_REFERENCE.

• Set the static ANDROID_LOGICAL_MULTI_CAMERA_SENSOR_SYNC_TYPE metadata field to:

  • ANDROID_LOGICAL_MULTI_CAMERA_SENSOR_SYNC_TYPE_APPROXIMATE: For sensors in master-master mode, no hardware shutter/exposure sync.
  • ANDROID_LOGICAL_MULTI_CAMERA_SENSOR_SYNC_TYPE_CALIBRATED: For sensors in master-slave mode, hardware shutter/exposure sync.

• Populate ANDROID_REQUEST_AVAILABLE_PHYSICAL_CAMERA_REQUEST_KEYS with a list of supported parameters for individual physical cameras. The list can be empty if the logical device doesn't support individual requests.

• If individual requests are supported, process and apply the individual physicalCameraSettings that can arrive as part of capture requests and append the individual physicalCameraMetadata accordingly.

• For Camera HAL device versions 3.5 (introduced in Android 10) or higher, populate theANDROID_LOGICAL_MULTI_CAMERA_ACTIVE_PHYSICAL_ID result key using the ID of the current active physical camera backing the logical camera.

For devices running Android 9, camera devices must support replacing one logical YUV/RAW stream with physical streams of the same size (doesn't apply to RAW streams) and the same format from two physical cameras. This doesn't apply to devices running Android 10.

For devices running Android 10 where the camera HAL device version is 3.5 or higher, the camera device must support isStreamCombinationSupported for applications to query whether a particular stream combination containing physical streams is supported.

Stream configuration map

For a logical camera, the mandatory stream combinations for the camera device of a certain hardware level is the same as what's required in CameraDevice.createCaptureSession. All of the streams in the stream configuration map must be logical streams.

For a logical camera device supporting RAW capability with physical sub-cameras of different sizes, if an application configures a logical RAW stream, the logical camera device must not switch to physical sub-cameras with different sensor sizes. This ensures that existing RAW capture applications don't break.

To take advantage of HAL-implemented optical zoom by switching between physical sub-cameras during RAW capture, applications must configure physical sub-camera streams instead of a logical RAW stream.

Guaranteed stream combination

Both the logical camera and its underlying physical cameras must guarantee the mandatory stream combinationsrequired for their device levels.

A logical camera device should operate in the same way as a physical camera device based on its hardware level and capabilities. It's recommended that its feature set is a superset of that of individual physical cameras.

On devices running Android 9, for each guaranteed stream combination, the logical camera must support:

• 在physical camera支持的情况下，用两个physical camera出两路（分别出一路）size和format相同的physical stream替换一路logical stream (YUV_420_888或RAW).

• 如果logical camera不支持RAW capability，而底层的physical camera支持，则添加两路raw stream，每个physical camera输出一路。当physical camera具有不同的senso size时，通常会发生这种情况。

• 使用physical stream代替相同size和format的logical stream。当physical stream和logical stream的minimum_frame_duration相同时，这一定不能减慢Cpature的帧率。

Performance and power considerations

• Performance:

  • Configuring and streaming physical streams may slow down the logical camera's capture rate due to resource constraints.
  • Applying physical camera settings may slow down the capture rate if the underlying cameras are put into different frame rates.

• Power:

  • HAL's power optimization continues to work in the default case.
  • Configuring or requesting physical streams may override HAL's internal power optimization and incur more power use.

Customization

You can customize your device implementation in the following ways.

• Logical_camera_device输出的logical stream完全取决于HAL实现。 如何从底层的physical camera生成并输出的ogic stream对于应用程序和Android Camera Framework是透明的。
• 对Individual physical requests 和 results 的支持是可选的. 此类请求中的可用参数集也完全取决于特定的HAL实现。
• 从Android 10开始，HAL可以在调用getCameraIdList时不将部分或全部PHYSICAL_ID返回，从而减少应用程序可以直接打开的Camera数量。但是，调用getPhysicalCameraCharacteristics 必须返回physical camera的 characteristics。

    /**
     * getCameraIdList:
     *
     * Returns the list of internal camera device interfaces known to this
     * camera provider. These devices can then be accessed via the hardware
     * service manager.
     *
     * External camera devices (camera facing EXTERNAL) must be reported through
     * the device status change callback, not in this list. Only devices with
     * facing BACK or FRONT must be listed here.
     *
     * @return status Status code for the operation, one of:
     *     OK:
     *         On a succesful generation of camera ID list
     *     INTERNAL_ERROR:
     *         A camera ID list cannot be created. This may be due to
     *         a failure to initialize the camera subsystem, for example.
     * @return cameraDeviceServiceNames The vector of internal camera device
     *     names known to this provider.
     */
    getCameraIdList()
            generates (Status status, vec cameraDeviceNames);

 

Validation

Logical multi-camera devices must pass Camera CTS like any other regular camera. The test cases that target this type of device can be found in the LogicalCameraDeviceTest module.

These three ITS tests target multi-camera systems to facilitate the proper fusing of images:

• scene1/test_multi_camera_match.py
• scene4/test_multi_camera_alignment.py
• sensor_fusion/test_multi_camera_frame_sync.py

The scene1 and scene4 tests run with the ITS-in-a-box test rig. The test_multi_camera_match test asserts that the brightness of the center of the images match when the two cameras are both enabled. The test_multi_camera_alignment test asserts that camera spacings, orientations, and distortion parameters are properly loaded. If the multi-camera system includes a Wide FoV camera (>90o), the rev2 version of the ITS box is required.

Sensor_fusion is a second test rig that enables repeated, prescribed phone motion and asserts that the gyroscope and image sensor timestamps match and that the multi-camera frames are in sync.

All boxes are available through AcuSpec, Inc. (www.acuspecinc.com, [email protected]) and MYWAY Manufacturing (www.myway.tw, [email protected]). Additionally, the rev1 ITS box can be purchased through West-Mark (www.west-mark.com, [email protected]).

原文地址

        https://source.android.google.cn/devices/camera/multi-camera