人脸识别技术 （二） —— 基于CoreImage实现视频中人脸的识别

版本记录

版本号	时间
V1.0	2018.01.31

前言

人脸识别是图像识别技术中的一种，广泛的应用于很多领域，接下来这几篇我们就一起来研究几种关于人脸识别的技术。感兴趣的可以参考上面几篇文章。
1. 人脸识别技术 （一） —— 基于CoreImage实现对静止图片中人脸的识别

基于CoreImage的视频中人脸识别技术

第一篇文章我们利用CoreImage对静止的图像进行人脸识别，相对来说，静止图像还是好识别的，如果要识别由摄像头采集来的视频中的人脸，那就相对来说难了，因为会有很多的性能问题。下面我们就一起看一下，利用AVFoundation进行图像采集，利用CoreImage识别视频中的人脸。

---

功能实现

还是直接看一下代码。

#import "ViewController.h"
#import 

@interface ViewController () 

@property (nonatomic, strong) AVCaptureSession *captureSession;
@property (nonatomic, strong) AVCaptureDevice *captureDevice;
@property (nonatomic, strong) AVCaptureDeviceInput *captureVideoDeviceInput;
@property (nonatomic, strong) AVCaptureVideoDataOutput *captureMovieFileOutput;
@property (nonatomic, strong) AVCaptureConnection *captureConnection;
@property (nonatomic, strong) AVCaptureVideoPreviewLayer *previewLayer;
@property (nonatomic, strong) NSMutableArray  *faceViewArrM;

@end

@implementation ViewController

- (void)viewDidLoad
{
    [super viewDidLoad];
    
    self.faceViewArrM = [NSMutableArray array];
    
    self.captureSession = [[AVCaptureSession alloc] init];
    if ([self.captureSession canSetSessionPreset:AVCaptureSessionPresetHigh]) {
        self.captureSession.sessionPreset = AVCaptureSessionPresetHigh;
    }
    else {
        self.captureSession.sessionPreset = AVCaptureSessionPreset1280x720;
    }
    
    for (AVCaptureDevice *device in [AVCaptureDevice devices]) {
        if ([device hasMediaType:AVMediaTypeVideo]) {
            if (device.position == AVCaptureDevicePositionFront) {
                self.captureDevice = device;
            }
        }
    }
    
    //添加输入
    [self addVideoInput];
    
    //添加输出
    [self addVideoOutput];
    
    //添加预览图层
    [self addPreviewLayer];
    
    [self.captureSession commitConfiguration];
    [self.captureSession startRunning];

}

#pragma mark -  Object Private Function

- (void)addVideoInput
{
    NSError *error;
    self.captureVideoDeviceInput = [AVCaptureDeviceInput deviceInputWithDevice:self.captureDevice error:&error];
    if (error) {
        return;
    }
    if ([self.captureSession canAddInput:self.captureVideoDeviceInput]) {
        [self.captureSession addInput:self.captureVideoDeviceInput];
    }
}

- (void)addVideoOutput
{
    self.captureMovieFileOutput = [[AVCaptureVideoDataOutput alloc] init];
    [self.captureMovieFileOutput setSampleBufferDelegate:self queue:dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0)];
    self.captureMovieFileOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil];
    if ([self.captureSession canAddOutput:self.captureMovieFileOutput]) {
        [self.captureSession addOutput:self.captureMovieFileOutput];
    }
    
    //设置链接管理对象
    self.captureConnection = [self.captureMovieFileOutput connectionWithMediaType:AVMediaTypeVideo];
    //视频旋转方向设置
    self.captureConnection.videoScaleAndCropFactor = self.captureConnection.videoMaxScaleAndCropFactor;;
    //视频稳定设置
    if ([self.captureConnection isVideoStabilizationSupported]) {
        self.captureConnection.preferredVideoStabilizationMode = AVCaptureVideoStabilizationModeAuto;
    }
    
//    AVCaptureFileOutputDelegate *del = nil;
}

- (void)addPreviewLayer
{
    self.previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:self.captureSession];
    [self.previewLayer setVideoGravity:AVLayerVideoGravityResizeAspect];
    self.previewLayer.frame = self.view.bounds;
    [self.view.layer addSublayer:self.previewLayer];
}

- (void)detectFaceWithImage:(UIImage *)image
{
    // 图像识别能力：可以在CIDetectorAccuracyHigh(较强的处理能力)与CIDetectorAccuracyLow(较弱的处理能力)中选择，因为想让准确度高一些在这里选择CIDetectorAccuracyHigh
    NSDictionary *opts = [NSDictionary dictionaryWithObject:
                          CIDetectorAccuracyHigh forKey:CIDetectorAccuracy];
    // 将图像转换为CIImage
    CIImage *faceImage = [CIImage imageWithCGImage:image.CGImage];
    CIDetector *faceDetector = [CIDetector detectorOfType:CIDetectorTypeFace context:nil options:opts];
    // 识别出人脸数组
    NSArray *features = [faceDetector featuresInImage:faceImage];
    // 得到图片的尺寸
    CGSize inputImageSize = [faceImage extent].size;
    //将image沿y轴对称
    CGAffineTransform transform = CGAffineTransformScale(CGAffineTransformIdentity, 1, -1);
    //将图片上移
    transform = CGAffineTransformTranslate(transform, 0, -inputImageSize.height);
    
    //清空数组
    dispatch_async(dispatch_get_main_queue(), ^{
        [self.faceViewArrM enumerateObjectsUsingBlock:^(UIView * _Nonnull obj, NSUInteger idx, BOOL * _Nonnull stop) {
            [obj removeFromSuperview];
             obj = nil;
        }];
    });
    
    // 取出所有人脸
    for (CIFaceFeature *faceFeature in features){
        //获取人脸的frame
        CGRect faceViewBounds = CGRectApplyAffineTransform(faceFeature.bounds, transform);
        CGSize viewSize = self.previewLayer.bounds.size;
        CGFloat scale = MIN(viewSize.width / inputImageSize.width,
                            viewSize.height / inputImageSize.height);
        CGFloat offsetX = (viewSize.width - inputImageSize.width * scale) / 2;
        CGFloat offsetY = (viewSize.height - inputImageSize.height * scale) / 2;
        // 缩放
        CGAffineTransform scaleTransform = CGAffineTransformMakeScale(scale, scale);
        // 修正
        faceViewBounds = CGRectApplyAffineTransform(faceViewBounds,scaleTransform);
        faceViewBounds.origin.x += offsetX;
        faceViewBounds.origin.y += offsetY;
        
        //描绘人脸区域
        dispatch_async(dispatch_get_main_queue(), ^{
            UIView* faceView = [[UIView alloc] initWithFrame:faceViewBounds];
            faceView.layer.borderWidth = 2;
            faceView.layer.borderColor = [[UIColor redColor] CGColor];
            [self.view addSubview:faceView];
            [self.faceViewArrM addObject:faceView];
        });
        
        // 判断是否有左眼位置
        if(faceFeature.hasLeftEyePosition){
            NSLog(@"检测到左眼");
        }
        // 判断是否有右眼位置
        if(faceFeature.hasRightEyePosition){
            NSLog(@"检测到右眼");
        }
        // 判断是否有嘴位置
        if(faceFeature.hasMouthPosition){
            NSLog(@"检测到嘴部");
        }
    }
}

#pragma mark -  AVCaptureVideoDataOutputSampleBufferDelegate

- (void)captureOutput:(AVCaptureFileOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    NSLog(@"----------");

    [connection setVideoOrientation:AVCaptureVideoOrientationPortrait];
    CVImageBufferRef buffer;
    buffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    
    CVPixelBufferLockBaseAddress(buffer, 0);
    uint8_t *base;
    size_t width, height, bytesPerRow;
    base = (uint8_t *)CVPixelBufferGetBaseAddress(buffer);
    width = CVPixelBufferGetWidth(buffer);
    height = CVPixelBufferGetHeight(buffer);
    bytesPerRow = CVPixelBufferGetBytesPerRow(buffer);
    
    CGColorSpaceRef colorSpace;
    CGContextRef cgContext;
    colorSpace = CGColorSpaceCreateDeviceRGB();
    cgContext = CGBitmapContextCreate(base, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
    CGColorSpaceRelease(colorSpace);
    
    CGImageRef cgImage;
    UIImage *image;
    cgImage = CGBitmapContextCreateImage(cgContext);
    image = [UIImage imageWithCGImage:cgImage];
    [self detectFaceWithImage:image];
    CGImageRelease(cgImage);
    CGContextRelease(cgContext);
    
    CVPixelBufferUnlockBaseAddress(buffer, 0);
}

@end

下面看一下部分输出

2018-01-31 14:18:07.001789+0800 JJFaceDetector_demo2[4700:1444754] ----------
2018-01-31 14:18:07.168074+0800 JJFaceDetector_demo2[4700:1444754] 检测到左眼
2018-01-31 14:18:07.168400+0800 JJFaceDetector_demo2[4700:1444754] 检测到右眼
2018-01-31 14:18:07.168557+0800 JJFaceDetector_demo2[4700:1444754] 检测到嘴部

2018-01-31 14:18:07.174485+0800 JJFaceDetector_demo2[4700:1444754] ----------
2018-01-31 14:18:07.388472+0800 JJFaceDetector_demo2[4700:1444754] 检测到左眼
2018-01-31 14:18:07.389386+0800 JJFaceDetector_demo2[4700:1444754] 检测到右眼
2018-01-31 14:18:07.389440+0800 JJFaceDetector_demo2[4700:1444754] 检测到嘴部

2018-01-31 14:18:07.398383+0800 JJFaceDetector_demo2[4700:1444754] ----------
2018-01-31 14:18:07.587945+0800 JJFaceDetector_demo2[4700:1444754] 检测到左眼
2018-01-31 14:18:07.588429+0800 JJFaceDetector_demo2[4700:1444754] 检测到右眼
2018-01-31 14:18:07.588796+0800 JJFaceDetector_demo2[4700:1444754] 检测到嘴部

... ... 

下面看一下识别的效果

这个是我自己，不露脸了怕吓到诸位，不过还是可以识别的

另外一个手机中的范爷图片

移动另外一个手机中的范爷图片

---

几个需要说明的问题

1. info.plist文件添加key

这个简单的说一下就可以了，iOS 10以后，相机权限需要增加key了。

2. 性能问题

移动的时候如果移动过快会有检测不准确的现象，这个是由于，识别和计算脸部位置并进行标记，但是计算好如果正好进行了移动，那么标记的可能还是上一帧的位置，所有有时候标记不那么准确。

3. 部分代码说明

先说一下这一句代码，假如不添加下面这句代码

self.captureMovieFileOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey, nil];

我们运行下，看输出

2018-01-31 14:32:57.312082+0800 JJFaceDetector_demo2[4706:1448810] ----------
2018-01-31 14:32:57.312320+0800 JJFaceDetector_demo2[4706:1448810] [Unknown process name] CGBitmapContextCreate: invalid data bytes/row: should be at least 2880 for 8 integer bits/component, 3 components, kCGImageAlphaPremultipliedFirst.
2018-01-31 14:32:57.312431+0800 JJFaceDetector_demo2[4706:1448810] [Unknown process name] CGBitmapContextCreateImage: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
2018-01-31 14:32:57.312468+0800 JJFaceDetector_demo2[4706:1448810] [api] -[CIImage initWithCGImage:options:] failed because the CGImage is nil.

这里提示的意思是CGBitmapContextCreate创建上下文和图像失败了，是一个无效的数据位，我在stackOverFlow中找到了答案，有人和我碰到了一样的问题。

看一下别人的Answers

Your best bet will be to set the capture video data output's videoSettings to a dictionary that specifies the pixel format you want, which you'll need to set to some variation on RGB that CGBitmapContext can handle.
The documentation has a list of all of the pixel formats that Core Video can process. Only a tiny subset of those are supported by CGBitmapContext. The format that the code you found on the internet is expecting is kCVPixelFormatType_32BGRA, but that might have been written for Macs—on iOS devices, kCVPixelFormatType_32ARGB (big-endian) might be faster. Try them both, on the device, and compare frame rates.

下面我给大家翻译下

您最好的选择是将捕获视频数据输出的videoSettings设置为一个字典，该字典指定了您想要的像素格式，您需要在CGBitmapContext可以处理的RGB上设置一些变量。
文档中列出了a list of all of the pixel formats that Core Video can process。CGBitmapContext仅支持其中的一小部分。 您在互联网上找到的代码的格式是kCVPixelFormatType_32BGRA，但可能已经为iOS设备上的Mac编写，kCVPixelFormatType_32ARGB（big-endian）可能会更快。 在设备上试用它们，并比较帧速率。

所以加上上面那个setting字典就解决了问题。

后记

本篇已结束，后面更精彩~~~