利用MediaCodec可以录制视频，可是可以将图片合成视频吗？之前使用ffmpeg来实现。但是，ffmpeg却是c++写的，而且非常占用内存，虽然它是非常棒的音视频处理库，但是杀鸡焉用牛刀，所以今天就讲一下：如何利用Android API中的MediaCodec来实现图片合成视频
YUV是为了解决彩色电视与黑白电视的兼容性。黑白视频只有Y值，也就是灰度。而彩色电视则有YUV3个分量，如果只读取Y值，就只能显示黑白画面了。YUV最大的优点在于只需占用极少的带宽。
参考
Android MediaCodec 硬编码器封装 - https://blog.csdn.net/devil__lee/article/details/49508773
图文详解YUV420数据格式 - https://www.cnblogs.com/Sharley/p/5595768.html
如何正确使用ImageReader与YUV_420_888和MediaCodec将视频编码为h264格式？- https://stackoverrun.com/cn/q/12725625
AVFrame 与 yuv420那些事 - https://blog.csdn.net/lanxiaziyi/article/details/74139729?utm_source=blogxgwz6
YUV420P和YUV420有什么区别? - https://bbs.csdn.net/topics/80129347
Java实现的RGB转YUV420方法 - https://blog.csdn.net/u012149399/article/details/78799990
YUV简介
YUV是一种也是编码方法。比如我们常用的RGB，是指红(red)、绿(green)，蓝(blue)，而YUV则是指：
- Y：亮度(Luminance、Luma)；
- U：色度(Chrominance )；
- V：浓度(Chroma)。
rgb与yuv互转：
rgb转yuv
Y = 0.299 R + 0.587 G + 0.114 B
U = - 0.1687 R - 0.3313 G + 0.5 B + 128
V = 0.5 R - 0.4187 G - 0.0813 B + 128
yuv转rgb
R = Y + 1.402 (V-128)
G = Y - 0.34414 (U-128) - 0.71414 (V-128)
B = Y + 1.772 (U-128)
颜色取样
将图片编码为YUV格式的数据时，将对图片上的点进行采样存储。
以黑点表示采样该像素点的Y分量，以空心圆圈表示采用该像素点的UV分量
以黑点表示采样该像素点的Y分量，以空心圆圈表示采用该像素点的UV分量
通常采用的方式有
- YUV 4:4:4采样(全采样)，每一个Y对应一组UV分量；
- YUV 4:2:2采样，每两个Y共用一组UV分量；
- YUV 4:2:0采样，每四个Y共用一组UV分量。
存储方式
- planar：平面格式，先存储所有的Y，然后存储所有的U，然后存储V
- packed：打包格式，YUV交替存储
这里举例YUV420p与YUV420sp的区别：
- YUV420p：YYYYYYYY VV UU
- YUV420sp：YYYYYYYY UVUV
由VU顺序的不同YUV420p可分为I420和YV12，上诉例子是YV12；YUV420sp可分为 NV12与NV21，上诉例子是NV12；
难点
1. 获取设备可渲染的颜色空间模式
由于不同手机生产商对颜色空间的渲染模式不尽相同，所以需要区别对待。不过大多是手机都是支持YUV420p、YUV420sp其中的一种。
public int[] getMediaCodecList() {
//获取解码器列表
int numCodecs = MediaCodecList.getCodecCount();
MediaCodecInfo codecInfo = null;
for (int i = 0; i < numCodecs && codecInfo == null; i++) {
MediaCodecInfo info = MediaCodecList.getCodecInfoAt(i);
if (!info.isEncoder()) {
continue;
}
String[] types = info.getSupportedTypes();
boolean found = false;
//轮训所要的解码器
for (int j = 0; j < types.length && !found; j++) {
if (types[j].equals("video/avc")) {
found = true;
}
}
if (!found) {
continue;
}
codecInfo = info;
}
Log.d(TAG, "found" + codecInfo.getName() + "supporting" + " video/avc");
MediaCodecInfo.CodecCapabilities capabilities = codecInfo.getCapabilitiesForType("video/avc");
return capabilities.colorFormats;
}
得到颜色空间模式后，可以判断选择其中一种来对图片进行编码：
public int getColorFormat(){
int colorFormat;
int[] formats = this.getMediaCodecList();
lab:
for (int format : formats) {
switch (format) {
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar: // yuv420sp
colorFormat = format;
break lab;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar: // yuv420p
colorFormat = format;
break lab;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar: // yuv420psp
colorFormat = format;
break lab;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar: // yuv420pp
colorFormat = format;
break lab;
}
}
if (colorFormat <= 0) {
colorFormat = MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar;
}
return colorFormat;
}
然后就是设置MediaFormat:
MediaFormat mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, width, * height);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 16);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 10);
2. rgb转YUV420p、YUV420sp、YUV420pp、YUV420psp
这里只贴出rgb转YUV420p、YUV420sp，rgb转YUV420pp和YUV420psp的代码并没有找到，只能自己写，虽然也写了，但是还没有验证过，就不贴出来了。
由于YUV420不是全采样，U和V的数据都是with*height*(1/4)，所以数据长度为：1(Y)+1/4(U)+1/4(V) = 3/2。
具体代码如下：
encodeYUV420SP：
private void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
V = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128; // Previously U
U = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128; // Previously V
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}
encodeYUV420P:
private void encodeYUV420P(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uIndex = frameSize;
int vIndex = frameSize + width * height / 4;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
V = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128; // Previously U
U = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128; // Previously V
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[vIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
yuv420sp[uIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
}
index++;
}
}
}
然后统一处理：
private byte[] getNV12(int inputWidth, int inputHeight, Bitmap scaled) {
int[] argb = new int[inputWidth * inputHeight];
//Log.i(TAG, "scaled : " + scaled);
scaled.getPixels(argb, 0, inputWidth, 0, 0, inputWidth, inputHeight);
byte[] yuv = new byte[inputWidth * inputHeight * 3 / 2];
switch (colorFormat) {
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar: // yuv420sp
encodeYUV420SP(yuv, argb, inputWidth, inputHeight);
break;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar: // yuv420p
encodeYUV420P(yuv, argb, inputWidth, inputHeight);
break;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar: // yuv420psp
encodeYUV420PSP(yuv, argb, inputWidth, inputHeight);
break;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar: // yuv420pp
encodeYUV420PP(yuv, argb, inputWidth, inputHeight);
break;
}
// scaled.recycle();
return yuv;
}
3. 保存为mp4格式的视频
视频处理需要用到MediaMuxer：
mediaMuxer = new MediaMuxer(out.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
其中out为视频输出文件。
生成MediaCodec对象：
try {
mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
//创建生成MP4初始化对象
mediaMuxer = new MediaMuxer(out.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
} catch (IOException e) {
e.printStackTrace();
}
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
导出视频文件后的处理：
public void finish() {
isRunning = false;
if (mediaCodec != null) {
mediaCodec.stop();
mediaCodec.release();
}
if (mediaMuxer != null) {
try {
if (mMuxerStarted) {
mediaMuxer.stop();
mediaMuxer.release();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
核心处理逻辑：
public void encode(Bitmap bitmap) {
final int TIMEOUT_USEC = 10000;
isRunning = true;
long generateIndex = 0;
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
ByteBuffer[] buffers = null;
if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
buffers = mediaCodec.getInputBuffers();
}
while (isRunning) {
int inputBufferIndex = mediaCodec.dequeueInputBuffer(TIMEOUT_USEC);
if (inputBufferIndex >= 0) {
long ptsUsec = computePresentationTime(generateIndex);
if (generateIndex >= mProvider.size()) {
mediaCodec.queueInputBuffer(inputBufferIndex, 0, 0, ptsUsec,
MediaCodec.BUFFER_FLAG_END_OF_STREAM);
isRunning = false;
drainEncoder(true, info);
} else {
if (bitmap == null) {
bitmap = mProvider.next();
}
byte[] input = getNV12(getSize(bitmap.getWidth()), getSize(bitmap.getHeight()), bitmap);
bitmap = null;
//有效的空的缓存区
ByteBuffer inputBuffer = null;
if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
inputBuffer = buffers[inputBufferIndex];
} else {
inputBuffer = mediaCodec.getInputBuffer(inputBufferIndex);//inputBuffers[inputBufferIndex];
}
inputBuffer.clear();
inputBuffer.put(input);
//将数据放到编码队列
mediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, ptsUsec, 0);
drainEncoder(false, info);
}
generateIndex++;
} else {
Log.i(TAG, "input buffer not available");
try {
Thread.sleep(50);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
其中对InputBuffer的获取做了兼容处理，mProvider是一个接口，用来获取位图。computePresentationTime的代码：
private long computePresentationTime(long frameIndex) {
return 132 + frameIndex * 1000000 / mFrameRate;
}
然后就是buffer输出：
private void drainEncoder(boolean endOfStream, MediaCodec.BufferInfo bufferInfo) {
final int TIMEOUT_USEC = 10000;
ByteBuffer[] buffers = null;
if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
buffers = mediaCodec.getOutputBuffers();
}
if (endOfStream) {
try {
mediaCodec.signalEndOfInputStream();
} catch (Exception e) {
}
}
while (true) {
int encoderStatus = mediaCodec.dequeueOutputBuffer(bufferInfo, TIMEOUT_USEC);
if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
if (!endOfStream) {
break; // out of while
} else {
Log.i(TAG, "no output available, spinning to await EOS");
}
} else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
if (mMuxerStarted) {
throw new RuntimeException("format changed twice");
}
MediaFormat mediaFormat = mediaCodec.getOutputFormat();
mTrackIndex = mediaMuxer.addTrack(mediaFormat);
mediaMuxer.start();
mMuxerStarted = true;
} else if (encoderStatus < 0) {
Log.i(TAG, "unexpected result from encoder.dequeueOutputBuffer: " + encoderStatus);
} else {
ByteBuffer outputBuffer = null;
if (Build.VERSION.SDK_INT <= Build.VERSION_CODES.LOLLIPOP) {
outputBuffer = buffers[encoderStatus];
} else {
outputBuffer = mediaCodec.getOutputBuffer(encoderStatus);
}
if (outputBuffer == null) {
throw new RuntimeException("encoderOutputBuffer "
+ encoderStatus + " was null");
}
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
bufferInfo.size = 0;
}
if (bufferInfo.size != 0) {
if (!mMuxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
// adjust the ByteBuffer values to match BufferInfo
outputBuffer.position(bufferInfo.offset);
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
Log.d(TAG, "BufferInfo: " + bufferInfo.offset + ","
+ bufferInfo.size + ","
+ bufferInfo.presentationTimeUs);
try {
mediaMuxer.writeSampleData(mTrackIndex, outputBuffer, bufferInfo);
} catch (Exception e) {
Log.i(TAG, "Too many frames");
}
}
mediaCodec.releaseOutputBuffer(encoderStatus, false);
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
if (!endOfStream) {
Log.i(TAG, "reached end of stream unexpectedly");
} else {
Log.i(TAG, "end of stream reached");
}
break; // out of while
}
}
}
}
如果看过其他类似博客的代码，就会发现代码其实差不多，但是却不是我想要的，所以只能自己改。其中一些博客还是写的非常棒，从中学到了很多。
这篇文章讲的是利用纯Android API实现的图片合成视频文件，其中我有查询到利用ffmpeg的，利用opencv/javacv的，但是这边文章介绍的方式没有引用第三方库，因此打包出来的apk文件肯定是很小的。
为了解决这个问题查了不少资料，花了不少时间，不过还好，终于搞定了。
git地址：https://github.com/xiaoyifan6/videocreator