最近在side project中遇到了需要从视频中抓取多张图片的需求。安卓已经提供了从视频获取预览图片的ThumbnailUtils, 但此类不能根据timestamp获取bitmap。
以下记录自己找出的解决方案。
需求
在本地视频根据时间戳(timestamp)抓取bitmap图像。
解决方案
- MediaMetadataRetriever.getFrameAtTime()
配合使用这些flag可以达到不同的程度的时间精确度
-
OPTION_PREVIOUS_SYNC： 前一个i-frame -
OPTION_NEXT_SYNC： 后一个 i-frame -
OPTION_CLOSEST_SYNC： 最近的i-frame，不管前后 -
OPTION_CLOSEST： 最近的frame，不一定是i-frame。上面三个flag只需要decode一张frame即可（而且是i-frame）。但这个flag需要decode多张frame才能接近输入的timestamp，因此速度会慢些。
值得一提的是此类会用binder IBC给system media service发送请求， 因此decoder是在系统服务中进行的，并非在我自己的app进程中。
方法调用时logcat可看到如下log
918 13498 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in [email protected] process
918 28062 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in [email protected] process
918 1854 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in [email protected] process
918 13498 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in [email protected] process
具体实现可参考：http://androidxref.com/9.0.0_r3/xref/frameworks/av/media/libmedia/mediametadataretriever.cpp#154
需要注意的坑
如果只看文档就会以为用getFrameAtTime + OPTION_CLOSEST就能非常精准的返回在输入timestamp附近的视频帧，但在不同device上跑过代码才发现其行为其实很不统一。:(
高端机基本都能做到预期效果，返回的bitmap在输入timestamp附近。但低端机型直接无视OPTION_CLOSEST 返回附近I-frame的bitmap，因此有时得到的图像跟输入的timestamp相差甚远。例如输入的是3.3sec，返回的有可能是2.8sec的i-frame图像。
出于好奇，我在低端机上做了个小实验。扫描一个视频文件的video timestamp，再把他们逐一用来调用getFrameAtTime，看看返回的bitmap有没有重复。
很明显的看到高端机基本无重复的bitmap，低端机却有大把大把的重复。猜测应该是厂商为了性能而只做了附近i-frame的解码就返回了。有点坑。：（
如果需要在所有机器上返回精准timestamp的bitmap，恐怕只能自己操作decoder了。估计非常复杂。
此外 API28+ 新增加了MediaMetadataRetriever.getFramesAtIndex() 。 这应该是比pts更加稳定的抓取方式，毕竟index是连续的int。但不知低端机型会不会继续坑。
==高端
D/MediaMetadataRetrieverRunner: pts 33375 digest :ym71cu9iO1H94190FWVgeg==
D/MediaMetadataRetrieverRunner: pts 66625 digest :y9hG93MqABw3ILMta/noUg==
D/MediaMetadataRetrieverRunner: pts 100000 digest :0l2CkOYDNyxvZCfCeLuq9A==
D/MediaMetadataRetrieverRunner: pts 133375 digest :OHSVQQOxrO77mh9f7tTorQ==
D/MediaMetadataRetrieverRunner: pts 166625 digest :XZ8kDFxsdjgBkUzKl41dgA==
D/MediaMetadataRetrieverRunner: pts 200000 digest :0/13IPmw1gP9FLdpsJc/Wg==
D/MediaMetadataRetrieverRunner: pts 233375 digest :TeHChk9OWbb8nLbrFphlEg==
D/MediaMetadataRetrieverRunner: pts 266625 digest :GEzE3LtFzG88bT7WtMTbxA==
D/MediaMetadataRetrieverRunner: pts 300000 digest :QjdeoZYwYgNxV144kKLTNw==
D/MediaMetadataRetrieverRunner: pts 333375 digest :0DZCQer3BqMPfSYTJaafMg==
D/MediaMetadataRetrieverRunner: pts 366625 digest :mfjEap/awknngWIeyPAScg==
D/MediaMetadataRetrieverRunner: pts 400000 digest :Q1oSKau7n9boi0vDSFZJFA==
D/MediaMetadataRetrieverRunner: pts 433375 digest :kW0vjMra+boztvH+PXQrlw==
D/MediaMetadataRetrieverRunner: pts 466625 digest :a6C/BpHAcr4BxSDg7Gt4bQ==
D/MediaMetadataRetrieverRunner: pts 500000 digest :odAEN5ESKWrLmXwDu09arA==
D/MediaMetadataRetrieverRunner: pts 533375 digest :+zTTQsil/s6on0EVqFuH/Q==
D/MediaMetadataRetrieverRunner: pts 566625 digest :ZqsNHZd1r12UL7cFHdc9vw==
D/MediaMetadataRetrieverRunner: pts 600000 digest :CgUZWadeCe+S+e28C/4qkA==
D/MediaMetadataRetrieverRunner: pts 633375 digest :hUaJy2jWWa9RU4dR24Om7w==
D/MediaMetadataRetrieverRunner: pts 666625 digest :a5HlUYUJcz3xaQrn/hNsIg==
D/MediaMetadataRetrieverRunner: pts 700000 digest :YF70StAojixZxjk8epZL3w==
D/MediaMetadataRetrieverRunner: pts 733375 digest :rolF61sMxRMSP09ePtUGcQ==
D/MediaMetadataRetrieverRunner: pts 766625 digest :K+qZUFNP5EgzWPscmmnFew==
D/MediaMetadataRetrieverRunner: pts 800000 digest :d8iVGhHf3VpMn+vMnBdmng==
D/MediaMetadataRetrieverRunner: pts 833375 digest :cKREwaki8AJmOuSVa8Zvbw==
D/MediaMetadataRetrieverRunner: pts 866625 digest :b3QXkX+wTE1CuCe79JK7Ww==
D/MediaMetadataRetrieverRunner: pts 900000 digest :atzAyrZNcOf8Ghgf04lftw==
D/MediaMetadataRetrieverRunner: pts 933375 digest :hS9ukOCLMCobHplBeRNdOA==
D/MediaMetadataRetrieverRunner: pts 966625 digest :rZyr6vVt5ae+TiMMVTCRrg==
=====
==低端
D/MediaMetadataRetrieverRunner: pts 1601625 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1635000 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1668375 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1701750 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1735000 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1768375 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1801750 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1835125 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1868500 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1901875 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1935250 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 1968625 digest :Ba29dyKehU1o6EanhG0wvg==
D/MediaMetadataRetrieverRunner: pts 2002000 digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2035375 digest :830odrRgw9UyAhNKeBUWAA==
D/MediaMetadataRetrieverRunner: pts 2068750 digest :8WFabowQKX+I2q3XNC2HJg==
D/MediaMetadataRetrieverRunner: pts 2102125 digest :8WFabowQKX+I2q3XNC2HJg==
D/MediaMetadataRetrieverRunner: pts 2135500 digest :8WFabowQKX+I2q3XNC2HJg==
D/MediaMetadataRetrieverRunner: pts 2168875 digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2202250 digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2235500 digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2268875 digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2302250 digest :dcMXdpTton6/YQHJA2zCjg==
D/MediaMetadataRetrieverRunner: pts 2335625 digest :h5ZNxcxO6WrbYFQPsknjnw==
D/MediaMetadataRetrieverRunner: pts 2369000 digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2402375 digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2435750 digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2469125 digest :KXZQ788bP25Oqi7sHxWPLg==
D/MediaMetadataRetrieverRunner: pts 2502500 digest :2h341j90Scn3kRtA4Fr+IA==
D/MediaMetadataRetrieverRunner: pts 2535875 digest :2h341j90Scn3kRtA4Fr+IA==
D/MediaMetadataRetrieverRunner: pts 2569250 digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2602625 digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2636000 digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2669375 digest :OOdPxTb4/NWbUrC8HDzpng==
D/MediaMetadataRetrieverRunner: pts 2702750 digest :OOdPxTb4/NWbUrC8HDzpng==
相关代码
public class MediaMetadataRetrieverRunner {
private String TAG = MediaMetadataRetrieverRunner.class.getSimpleName();
void scanAndDigest(String path) throws Exception {
List ptsUsList = VideoPtsScanner.scanPts(path);
Log.d(TAG, "pts list in micro sec: " + ptsUsList);
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(path);
for (long ptsUs : ptsUsList) {
Bitmap bitmap = retriever.getFrameAtTime(ptsUs, MediaMetadataRetriever.OPTION_CLOSEST);
if (bitmap == null) {
Log.e(TAG, "got null in pts " + ptsUs);
continue;
}
int size = bitmap.getAllocationByteCount();
ByteBuffer buffer = ByteBuffer.allocateDirect(size);
bitmap.copyPixelsToBuffer(buffer);
buffer.flip();
String digest = makeString(buffer);
Log.d(TAG, "pts " + ptsUs + " \t digest :" + digest);
}
retriever.release();
}
private String makeString(ByteBuffer buffer) throws NoSuchAlgorithmException {
MessageDigest md5 = MessageDigest.getInstance("MD5");
md5.update(buffer);
byte[] digest = md5.digest();
// covert into base64 string, since it's already provided in android
return Base64.encodeToString(digest, Base64.DEFAULT);
}
}
-----
public class VideoPtsScanner {
public static List scanPts(String path) throws IOException {
MediaExtractor mediaExtractor = new MediaExtractor();
mediaExtractor.setDataSource(path);
mediaExtractor.selectTrack(getVideoTrack(mediaExtractor));
List ret = new ArrayList();
long pts = -1;
while ((pts = mediaExtractor.getSampleTime()) >= 0) {
ret.add(pts);
mediaExtractor.advance();
}
mediaExtractor.release();
return ret;
}
private static int getVideoTrack(MediaExtractor extractor) {
for (int i = 0; i < extractor.getTrackCount(); i++) {
MediaFormat format = extractor.getTrackFormat(i);
if (isVideo(format)) {
return i;
}
}
return -1;
}
private static boolean isVideo(MediaFormat format) {
return format.getString(MediaFormat.KEY_MIME).toLowerCase().contains("video");
}
}