Android Camera在使用SurfaceTexture获取帧数据时，SurfaceTexture是如何获取帧数据并将帧数据绑定到GL_TEXTURE_EXTERNAL_OES纹理上的

在学习Android Camera API 2使用OpenGL ES 2.0和GLSurfaceView对预览进行实时二次处理（黑白滤镜），发现是通过SurfaceTexture接收相机帧数据并最终通过一个GL_TEXTURE_EXTERNAL_OES纹理将帧数据返回给相机应用的，文本主要分析下

1. SurfaceTexture是如何创建的
2. SurfaceTexture如何获取相机帧数据
3. SurfaceTexture如何将帧数据绑定到GL_TEXTURE_EXTERNAL_OES纹理上

1.SurfaceTexture是如何创建的

首先，我们需要分析下SurfaceTexture是如何创建的，都做了哪些工作

1. 应用层SurfaceTexture创建代码如下：

    //创建纹理id
    int[] tex = new int[1];
    GLES20.glGenTextures(1, tex, 0);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, tex[0]);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
            GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_NEAREST);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
            GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
            GL10.GL_TEXTURE_WRAP_S, GL10.GL_CLAMP_TO_EDGE);
    GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
            GL10.GL_TEXTURE_WRAP_T, GL10.GL_CLAMP_TO_EDGE);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
    //创建SurfaceTexture并传入tex[0]
    mSurfaceTexture = new SurfaceTexture(tex[0]);

2. Framework 层SurfaceTexture创建代码如下：

//frameworks\base\graphics\java\android\graphics
//构造函数
public SurfaceTexture(int texName) {
    this(texName, false);
}
//构造函数
//singleBufferMode是否是单buffer，默认为false
public SurfaceTexture(int texName, boolean singleBufferMode) {
   mCreatorLooper = Looper.myLooper();
   mIsSingleBuffered = singleBufferMode;
   //native方法nativeInit
   nativeInit(false, texName, singleBufferMode, new WeakReference<SurfaceTexture>(this));
}

发现SurfaceTexture构造函数只是简单的调用了下nativeInit方，接着分析下nativeInit方法，SurfaceTexture Native 方法注册表如下：

//frameworks\base\core\jni\SurfaceTexture.cpp
static const JNINativeMethod gSurfaceTextureMethods[] = {
    {"nativeInit",                 "(ZIZLjava/lang/ref/WeakReference;)V", (void*)SurfaceTexture_init },
    ...
    {"nativeUpdateTexImage",       "()V",   (void*)SurfaceTexture_updateTexImage },
    ....
};

发现nativeInit对应的Native方法为SurfaceTexture_init，
接着分析下SurfaceTexture_init

//frameworks\base\core\jni\SurfaceTexture.cpp
//isDetached为false
//texName为应用创建的GL_TEXTURE_EXTERNAL_OES型texture
//singleBufferMode为false
//weakThiz为SurfaceTexture对象弱引用
static void SurfaceTexture_init(JNIEnv* env, jobject thiz, jboolean isDetached,
        jint texName, jboolean singleBufferMode, jobject weakThiz)
{
    sp<IGraphicBufferProducer> producer;
    sp<IGraphicBufferConsumer> consumer;
    //创建IGraphicBufferProducer及IGraphicBufferConsumer
    BufferQueue::createBufferQueue(&producer, &consumer);

    if (singleBufferMode) {
        consumer->setMaxBufferCount(1);
    }

    sp<GLConsumer> surfaceTexture;
    //isDetached为false
    if (isDetached) {
        ....
    } else {
        //将consumer和texName封装为GLConsumer类对象surfaceTexture
        surfaceTexture = new GLConsumer(consumer, texName,
                GL_TEXTURE_EXTERNAL_OES, true, !singleBufferMode);
    }

    ....
    //设置surfaceTexture名字
    surfaceTexture->setName(String8::format("SurfaceTexture-%d-%d-%d",
            (isDetached ? 0 : texName),
            getpid(),
            createProcessUniqueId()));

    // If the current context is protected, inform the producer.
    consumer->setConsumerIsProtected(isProtectedContext());
    //将surfaceTexture保存到env中
    SurfaceTexture_setSurfaceTexture(env, thiz, surfaceTexture);
    //将producer保存到env中
    SurfaceTexture_setProducer(env, thiz, producer);

    jclass clazz = env->GetObjectClass(thiz);
    //JNISurfaceTextureContext继承了GLConsumer::FrameAvailableListener
    sp<JNISurfaceTextureContext> ctx(new JNISurfaceTextureContext(env, weakThiz,
            clazz));
    //surfaceTexture设置帧回调对象ctx，
    //收到帧数据是会触发ctx->onFrameAvailable方法
    surfaceTexture->setFrameAvailableListener(ctx);
    SurfaceTexture_setFrameAvailableListener(env, thiz, ctx);
}

通过上边的分析可见，创建SurfaceTexture时会通过BufferQueue分别创建一对IGraphicBufferProducer和IGraphicBufferConsumer类的生产者、消费者对象：producer、consumer。然后给消费者consumer注册一个继承了GLConsumer::FrameAvailableListener的帧回调对象 JNISurfaceTextureContext。

在分析Android Camera学习总结时，分析过Android Camera 在createCaptureSession时需要向Camera系统设置至少一个surface。而创建SurfaceTexture时创建了一对IGraphicBufferProducer和IGraphicBufferConsumer型的生产者、消费者对象：producer、consumer,生产者producer就可以封装为一个surface对象供Camera系统使用，实现代码如下：

///frameworks/base/core/jni/android_view_Surface.cpp
static jlong nativeCreateFromSurfaceTexture(JNIEnv* env, jclass clazz,
        jobject surfaceTextureObj) {
    sp<IGraphicBufferProducer> producer(SurfaceTexture_getProducer(env, surfaceTextureObj));
    ....
    sp<Surface> surface(new Surface(producer, true));
   ......
    surface->incStrong(&sRefBaseOwner);
    return jlong(surface.get());
}

在Camera 系统收到帧数据后，通过queueBuffer归还GraphicBuffer给surface时，会触发消费者consumer注册的帧回调JNISurfaceTextureContext::onFrameAvailable，该方法又会调用Java层注册的SurfaceTexture.OnFrameAvailableListener的onFrameAvailable方法，代码如下：

void JNISurfaceTextureContext::onFrameAvailable(const BufferItem& /* item */)
{
  ....
  //调用Java层SurfaceTexture.OnFrameAvailableListener的onFrameAvailable方法
  env->CallStaticVoidMethod(mClazz, fields.postEvent, mWeakThiz);
  .....
}

至此，可以了解到：
创建SurfaceTexture时，在Native层会创建一对IGraphicBufferProducer和IGraphicBufferConsumer的生产者和消费者对象：producer、consumer。生产者producer可以封装为surface供Camera系统使用，消费者consumer在收到相机帧数据时会触发消费者consumer注册的帧回调函数JNISurfaceTextureContext::onFrameAvailable。

2. SurfaceTexture如何获取相机帧数据

SurfaceTexture在收到帧回调onFrameAvailable时，可以通过调用SurfaceTexture.updateTexImage() 来更新SurfaceTexture中的GL_TEXTURE_EXTERNAL_OES型纹理内容。更新纹理内容首先需要获取帧数据，我们分析下其代码

//frameworks\base\graphics\java\android\graphics\SurfaceTexture.java
    public void updateTexImage() {
        nativeUpdateTexImage();
    }

接着分析下Native方法nativeUpdateTexImage，在jni中对用的函数是SurfaceTexture_updateTexImage

//frameworks\base\core\jni\SurfaceTexture.cpp
static void SurfaceTexture_updateTexImage(JNIEnv* env, jobject thiz)
{
    //获取env中保存的surfaceTexture
    sp<GLConsumer> surfaceTexture(SurfaceTexture_getSurfaceTexture(env, thiz));
    //更新纹理,surfaceTexture为GLConsumer类型
    status_t err = surfaceTexture->updateTexImage();
    ....
}

接着分析下GLConsumer::updateTexImage()方法

//frameworks\native\libs\gui\GLConsumer.cpp
status_t GLConsumer::updateTexImage() {
      ....
    //更新EGL状态，如：
    //更新mEglDisplay为eglGetCurrentDisplay
    //mEglContext为eglGetCurrentContext
    // Make sure the EGL state is the same as in previous calls.
    status_t err = checkAndUpdateEglStateLocked();
    ....
    BufferItem item;
    //获取帧数据buffer
    // Acquire the next buffer.
    // In asynchronous mode the list is guaranteed to be one buffer
    // deep, while in synchronous mode we use the oldest buffer.
    err = acquireBufferLocked(&item, 0);
    .....
    //Update the Current GLConsumer state.
    //Release the previous buffer.
    err = updateAndReleaseLocked(item);
    .....
    //将新buffer绑定到纹理
    // Bind the new buffer to the GL texture, and wait until it's ready.
    return bindTextureImageLocked();
}

分析下GLConsumer::acquireBufferLocked

//frameworks\native\libs\gui\GLConsumer.cpp
status_t GLConsumer::acquireBufferLocked(BufferItem *item,
        nsecs_t presentWhen, uint64_t maxFrameNumber) {
     //获取Consumer当前的显示内容BufferItem 
     //既获取相机预览帧数据
    status_t err = ConsumerBase::acquireBufferLocked(item, presentWhen,
            maxFrameNumber);
    ...
    // If item->mGraphicBuffer is not null, this buffer has not been acquired
    // before, so any prior EglImage created is using a stale buffer. This
    // replaces any old EglImage with a new one (using the new buffer).
    if (item->mGraphicBuffer != NULL) {
        int slot = item->mSlot;
        //由获取的item->mGraphicBuffer生成一个EglImage，并赋值给mEglSlots[slot].mEglImage
        mEglSlots[slot].mEglImage = new EglImage(item->mGraphicBuffer);
    }

    return NO_ERROR;
}

至此，可以了解到surfaceTexture通过 GLConsumer::acquireBufferLocked就可以获取了当前的相机帧数据，相机帧数据存放在了item->mGraphicBuffer中。

3. SurfaceTexture如何将帧数据绑定到GL_TEXTURE_EXTERNAL_OES纹理上

我们接着分析下GLConsumer::updateAndReleaseLocked

status_t GLConsumer::updateAndReleaseLocked(const BufferItem& item,
        PendingRelease* pendingRelease)
{
    status_t err = NO_ERROR;
    //acquireBufferLocked获取的BufferItem
    int slot = item.mSlot;

    .....
    //重新检查EGL状态
    // Confirm state.
    err = checkAndUpdateEglStateLocked();
    ....

    // Ensure we have a valid EglImageKHR for the slot, creating an EglImage
    // if nessessary, for the gralloc buffer currently in the slot in
    // ConsumerBase.
    // We may have to do this even when item.mGraphicBuffer == NULL (which
    // means the buffer was previously acquired).
    err = mEglSlots[slot].mEglImage->createIfNeeded(mEglDisplay, item.mCrop);
    .....

    // Do whatever sync ops we need to do before releasing the old slot.
    if (slot != mCurrentTexture) {
        err = syncForReleaseLocked(mEglDisplay);
       ....
    }

    GLC_LOGV("updateAndRelease: (slot=%d buf=%p) -> (slot=%d buf=%p)",
            mCurrentTexture, mCurrentTextureImage != NULL ?
                    mCurrentTextureImage->graphicBufferHandle() : 0,
            slot, mSlots[slot].mGraphicBuffer->handle);

    // Hang onto the pointer so that it isn't freed in the call to
    // releaseBufferLocked() if we're in shared buffer mode and both buffers are
    // the same.
    //将 mEglSlots[slot].mEglImage赋值到nextTextureImage
    sp<EglImage> nextTextureImage = mEglSlots[slot].mEglImage;
    
    // release old buffer
    if (mCurrentTexture != BufferQueue::INVALID_BUFFER_SLOT) {
        if (pendingRelease == nullptr) {
            status_t status = releaseBufferLocked(
                    mCurrentTexture, mCurrentTextureImage->graphicBuffer(),
                    mEglDisplay, mEglSlots[mCurrentTexture].mEglFence);
        }
        ....
    }

    // Update the GLConsumer state.
    mCurrentTexture = slot;
    mCurrentTextureImage = nextTextureImage;
    mCurrentCrop = item.mCrop;
    mCurrentTransform = item.mTransform;
    mCurrentScalingMode = item.mScalingMode;
    mCurrentTimestamp = item.mTimestamp;
    mCurrentDataSpace = item.mDataSpace;
    mCurrentFence = item.mFence;
    mCurrentFenceTime = item.mFenceTime;
    mCurrentFrameNumber = item.mFrameNumber;
   //跟新转换矩阵
    computeCurrentTransformMatrixLocked();

    return err;
}

该方法主要是更新下mCurrentTextureImage和mCurrentTexture并释放就得buffer。

接着分析下如何将mCurrentTextureImage绑定到GL_TEXTURE_EXTERNAL_OES纹理上，方法是GLConsumer::bindTextureImageLocked()

status_t GLConsumer::bindTextureImageLocked() {
    ....
    GLenum error;
    ....
    //mTexTarget为应用创建的GL_TEXTURE_EXTERNAL_OES型texture
    glBindTexture(mTexTarget, mTexName);
    ...
   //由mGraphicBuffer生成EGLImageKHR
    status_t err = mCurrentTextureImage->createIfNeeded(mEglDisplay,
                                                        mCurrentCrop);
    .....
   //将mCurrentTextureImage内容绑定到mTexTarget上
    mCurrentTextureImage->bindToTextureTarget(mTexTarget);

    .....
    // Wait for the new buffer to be ready.
    return doGLFenceWaitLocked();
}

我们先看一下GLConsumer::EglImage::createIfNeeded是如何通过mGraphicBuffer生成EGLImageKHR图像得

status_t GLConsumer::EglImage::createIfNeeded(EGLDisplay eglDisplay,
                                              const Rect& cropRect,
                                              bool forceCreation) {
    .....
    // If there's no image, create one.
    if (mEglImage == EGL_NO_IMAGE_KHR) {
        mEglDisplay = eglDisplay;
        mCropRect = cropRect;
        //创建EGLImageKHR对象
        mEglImage = createImage(mEglDisplay, mGraphicBuffer, mCropRect);
    }
   ....
    return OK;
}

接着分析下GLConsumer::EglImage::createImage

EGLImageKHR GLConsumer::EglImage::createImage(EGLDisplay dpy,
        const sp<GraphicBuffer>& graphicBuffer, const Rect& crop) {
    EGLClientBuffer cbuf =
            static_cast<EGLClientBuffer>(graphicBuffer->getNativeBuffer());
    const bool createProtectedImage =
            (graphicBuffer->getUsage() & GRALLOC_USAGE_PROTECTED) &&
            hasEglProtectedContent();
    EGLint attrs[] = {
        EGL_IMAGE_PRESERVED_KHR,        EGL_TRUE,
        EGL_IMAGE_CROP_LEFT_ANDROID,    crop.left,
        EGL_IMAGE_CROP_TOP_ANDROID,     crop.top,
        EGL_IMAGE_CROP_RIGHT_ANDROID,   crop.right,
        EGL_IMAGE_CROP_BOTTOM_ANDROID,  crop.bottom,
        createProtectedImage ? EGL_PROTECTED_CONTENT_EXT : EGL_NONE,
        createProtectedImage ? EGL_TRUE : EGL_NONE,
        EGL_NONE,
    };
    .....
    eglInitialize(dpy, 0, 0);
    //调用eglCreateImageKHR创建EGLImageKHR 对象
    EGLImageKHR image = eglCreateImageKHR(dpy, EGL_NO_CONTEXT,
            EGL_NATIVE_BUFFER_ANDROID, cbuf, attrs);
    ...
    return image;
}

通过上边的分析发现，createIfNeeded 就是通过eglCreateImageKHR(dpy, EGL_NO_CONTEXT, EGL_NATIVE_BUFFER_ANDROID, cbuf, attrs)将一个graphicbuffer生成了一个EGLImageKHR图像。

现在已经生成了EGLImageKHR图像，接下来分析下 EGLImageKHR 图像是如何绑定的GL_TEXTURE_EXTERNAL_OES纹理上的

//frameworks\native\libs\gui\GLConsumer.cpp
void GLConsumer::EglImage::bindToTextureTarget(uint32_t texTarget) {
    //更新纹理texTarget的数据为mEglImage
    //相当于glTexImage2D或者glTexSubImage2D
    glEGLImageTargetTexture2DOES(texTarget,
            static_cast<GLeglImageOES>(mEglImage));
}

从上述分析看，整个更新纹理的过程大致分为两步

1. 生成纹理内容EGLImageKHR 对象
   通过ConsumerBase::acquireBufferLocked获取当前的显示内容GraphicBuffer（在这里就是相机帧数据），然后由此GraphicBuffer生成一个EGLImageKHR图像
2. 将EGLImageKHR图像通过glEGLImageTargetTexture2DOES绑定到GL_TEXTURE_EXTERNAL_OES型纹理上。

其中关于将graphic buffer直接作为纹理的文章，请参考

• using direct textures on android
• Android中的EGL扩展
  至此分析完成了