CMSampleBuffer的使用
- 通过摄像头采样的视频数据。
- 通过mic采集的音频信息。
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection;
- AVAssetWriter保存视频AVAssetWriterInput
- (BOOL)appendSampleBuffer:(CMSampleBufferRef)sampleBuffer;
- 读取视频文件的输出流AVAssetReaderOutput
- (nullable CMSampleBufferRef)copyNextSampleBuffer CF_RETURNS_RETAINED;
- AVSampleBufferDisplayLayer 直接显示解码后的sampleBuffer
- (void)enqueueSampleBuffer:(CMSampleBufferRef)sampleBuffer;
- ARKit ARSessionObserver输出捕获的audioSampleBuffer
/**
This is called when the session outputs a new audio sample buffer.
@param session The session being run.
@param audioSampleBuffer The captured audio sample buffer.
*/
- (void)session:(ARSession *)session didOutputAudioSampleBuffer:(CMSampleBufferRef)audioSampleBuffer;
- VTCompressionSession硬件编码作为VTCompressionOutputCallback输出
typedef void (*VTCompressionOutputCallback)(
void * CM_NULLABLE outputCallbackRefCon,
void * CM_NULLABLE sourceFrameRefCon,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CM_NULLABLE CMSampleBufferRef sampleBuffer );
创建压缩session
注意：
Clients of CMSampleBuffer must explicitly manage the retain count by calling CFRetain and CFRelease, even in processes using garbage collection.
CMSampleBuffer 数据结构
- CMSampleBufferRef 这是一个包含零个或多个解码后（未解码）特定媒体类型的样本（音频，视频，多路复用等）。
API注释
A reference to a CMSampleBuffer, a CF object containing zero or more compressed (or uncompressed) samples of a particular media type (audio, video, muxed, etc).
- CaptureOutput输出video CMSampleBuffer,编码前内容
(lldb) po sampleBuffer
CMSampleBuffer retainCount: 1 allocator: 0x1b6708538
invalid = NO
dataReady = YES
makeDataReadyCallback = 0x0
makeDataReadyRefcon = 0x0
buffer-level attachments:
Orientation(P) = 1
{Exif} (P) {//省略}
DPIWidth (P) = 72
{TIFF} (P) {//省略}
DPIHeight (P) = 72
formatDescription = {
mediaType:'vide'
mediaSubType:'420f'
mediaSpecific: {
codecType: '420f' dimensions: 480 x 640
}
extensions: {{type = immutable dict, count = 6,
entries =>
0 :{contents = "CVImageBufferYCbCrMatrix"} = {contents = "ITU_R_601_4"}
1 : {contents = "CVImageBufferTransferFunction"} ={contents = "ITU_R_709_2"}
2 : {contents = "CVBytesPerRow"} ={value = +724, type = kCFNumberSInt32Type}
3 :{contents = "Version"} = {value = +2, type = kCFNumberSInt32Type}
5 : {contents = "CVImageBufferColorPrimaries"} = {contents = "ITU_R_709_2"}
6 : {contents = "CVImageBufferChromaLocationTopField"} = {contents = "Center"}
}
}
}
sbufToTrackReadiness = 0x0
numSamples = 1
sampleTimingArray[1] = {
{PTS = {6510691197875/1000000000 = 6510.691}, DTS = {INVALID}, duration = {INVALID}},
}
imageBuffer = 0x1c4134c80
(lldb)
- 硬编码后的数据,相比编码前多了sampleAttachmentsArray，颜色空间是'avc1'
CMSampleBuffer 0x11be1fda0 retainCount:
1 allocator: 0x1b6708538
invalid = NO
dataReady = YES
makeDataReadyCallback = 0x0
makeDataReadyRefcon = 0x0
formatDescription = {
mediaType:'vide'
mediaSubType:'avc1'
mediaSpecific: {
codecType: 'avc1' dimensions: 480 x 640
}
2 : {contents = "FormatName"} = H.264
sbufToTrackReadiness = 0x0
numSamples = 1
sampleTimingArray[1] = {
{PTS = {65/1000 = 0.065}, DTS = {64/1000 = 0.064}, duration = {INVALID}},
}
sampleSizeArray[1] = {
sampleSize = 212,
}
sampleAttachmentsArray[1] = {
sample 0:
NotSync = true
DependsOnOthers = true
}
dataBuffer = 0x1c410ac20
- CaptureOutput输出音频，buffer内容
CMSampleBuffer 0x125e41940 retainCount: 1 allocator: 0x1b6708538
invalid = NO
dataReady = YES
makeDataReadyCallback = 0x0
makeDataReadyRefcon = 0x0
formatDescription = {
mediaType:'soun'
mediaSubType:'lpcm'
mediaSpecific: {
ASBD: {
mSampleRate: 44100.000000
mFormatID: 'lpcm'
mFormatFlags: 0xc
mBytesPerPacket: 2
mFramesPerPacket: 1
mBytesPerFrame: 2
mChannelsPerFrame: 1
mBitsPerChannel: 16 }
cookie: {(null)}
ACL: {(null)}
FormatList Array: {(null)}
}
extensions: {(null)}
}
sbufToTrackReadiness = 0x0
numSamples = 1024
sampleTimingArray[1] = {
{PTS = {10713597355917/1000000000 = 10713.597, rounded}, DTS = {INVALID}, duration = {1/44100 = 0.000}},
}
sampleSizeArray[1] = {
sampleSize = 2,
}
dataBuffer = 0x1c4304650
- AVAssetReaderOutput读取MP4文件sampleBuffer内容
CMSampleBuffer retainCount: 1 allocator: 0x1b6708538
invalid = NO
dataReady = YES
makeDataReadyCallback = 0x0
makeDataReadyRefcon = 0x0
formatDescription = {
mediaType:'vide'
mediaSubType:'BGRA'
mediaSpecific: {
codecType: 'BGRA' dimensions: 640 x 360
}
extensions: {{type = immutable dict, count = 9,
entries =>
0 : {contents = "CVFieldCount"} = {value = +1, type = kCFNumberSInt32Type}
1 : {contents = "CVImageBufferYCbCrMatrix"} = {contents = "YCbCrMatrix#5"}
2 : {contents = "CVBytesPerRow"} ={value = +2560, type = kCFNumberSInt32Type}
3 : {contents = "CVPixelAspectRatio"} = {type = immutable dict, count = 2,
entries =>
1 : {contents = "HorizontalSpacing"} = {value = +1, type = kCFNumberSInt32Type}
2 : {contents = "VerticalSpacing"} = {value = +1, type = kCFNumberSInt32Type}
}
4 : {contents = "Version"} = {value = +2, type = kCFNumberSInt32Type}
5 : {contents = "CVImageBufferColorPrimaries"} = SMPTE_C
8 : {contents = "CVImageBufferTransferFunction"} = {contents = "TransferFunction#5"}
10 : {contents = "CVImageBufferChromaLocationBottomField"} = Left
12 : {contents = "CVImageBufferChromaLocationTopField"} = Left
}
}
}
sbufToTrackReadiness = 0x0
numSamples = 1
sampleTimingArray[1] = {
{PTS = {0/600 = 0.000}, DTS = {INVALID}, duration = {INVALID}},
}
imageBuffer = 0x1c4139280
以上可以看出未编码\解码后的CMSampleBuffer都包含了imageBuffer数据、formatDescription信息和sampleTimingArray信息。录像多了exif和tiff信息。
CVPixelBuffer：编码前和解码后的图像数据结构.
无论是读取还是录制
output的settings的kCVPixelBufferPixelFormatTypeKey字段 控制的mediaSubType输出的色彩空间，转换色彩空间过程有GPU参与，性能较高。可以按需设置。
CMSampleBuffer的创建及示例
- (void)appendVideoPixelBuffer:(CVPixelBufferRef)pixelBuffer withPresentationTime:(CMTime)presentationTime
{
CMSampleBufferRef sampleBuffer = NULL;
CMFormatDescriptionRef outputFormatDescription = NULL;
CMVideoFormatDescriptionCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, &outputFormatDescription );
CMSampleTimingInfo timingInfo = {0,};
timingInfo.duration = kCMTimeInvalid;
timingInfo.decodeTimeStamp = kCMTimeInvalid;
timingInfo.presentationTimeStamp = presentationTime;
OSStatus err = CMSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, true, NULL, NULL, outputFormatDescription, &timingInfo, &sampleBuffer );
if ( sampleBuffer ) {
// do some thing
CFRelease( sampleBuffer );
}
else {
NSString *exceptionReason = [NSString stringWithFormat:@"sample buffer create failed (%i)", (int)err];
@throw [NSException exceptionWithName:NSInvalidArgumentException reason:exceptionReason userInfo:nil];
}
}
- CMSampleBufferCreateReady与CMSampleBufferCreate相同，只是dataReady始终为true，因此不需要传递makeDataReadyCallback或refcon。