【融云分析】iOS 基于实时音视频 SDK 实现屏幕共享功能

Replaykit 介绍

在之前的 iOS 版本中,iOS 开发者只能拿到编码后的数据,拿不到原始的 PCM 和 YUV,到 iOS 10 之后,开发者可以拿到原始数据,但是只能录制 App 内的内容,如果切到后台,将停止录制,直到 iOS 11,苹果对屏幕共享进行了升级并开放了权限,既可以拿到原始数据,又可以录制整个系统,以下我们重点来说 iOS 11 之后的屏幕共享功能。

系统屏幕共享

- (void)initMode_1 {
    self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, ScreenWidth, 80)];
    self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.replaytest.Recoder";
    self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
    self.systemBroadcastPickerView.showsMicrophoneButton = NO;
    [self.view addSubview:self.systemBroadcastPickerView];
}

在 iOS 11 创建一个 Extension 之后,调用上面的代码就可以开启屏幕共享了,然后系统会为我们生成一个 SampleHandler 的类,在这个方法中,苹果会根据 RPSampleBufferType 上报不同类型的数据。

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType

那怎么通过融云的 RongRTCLib 将屏幕共享数据发送出去呢?

1. 基于 Socket 的逼格玩法

1.1. Replaykit 框架启动和创建 Socket

//
//  ViewController.m
//  Socket_Replykit
//
//  Created by Sun on 2020/5/19.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "ViewController.h"
#import 
#import "RongRTCServerSocket.h"

@interface ViewController ()

@property (nonatomic, strong) RPSystemBroadcastPickerView *systemBroadcastPickerView;
/**
 server socket
 */
@property(nonatomic , strong)RongRTCServerSocket *serverSocket;

@end

@implementation ViewController

- (void)viewDidLoad {
    [super viewDidLoad];
    self.view.backgroundColor = [UIColor whiteColor];
    // Do any additional setup after loading the view.
    [self.serverSocket createServerSocket];
  
    self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, [UIScreen mainScreen].bounds.size.width, 80)];
    self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.sealrtc.RongRTCRP";
    self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
    self.systemBroadcastPickerView.showsMicrophoneButton = NO;
    [self.view addSubview:self.systemBroadcastPickerView];
}

- (RongRTCServerSocket *)serverSocket {
    if (!_serverSocket) {
        RongRTCServerSocket *socket = [[RongRTCServerSocket alloc] init];
        socket.delegate = self;
        
        _serverSocket = socket;
    }
    return _serverSocket;
}

- (void)didProcessSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    // 这里拿到了最终的数据,比如最后可以使用融云的音视频SDK RTCLib 进行传输就可以了
}

@end

其中,包括了创建 Server Socket 的步骤,我们把主 App 当做 Server,然后屏幕共享 Extension 当做 Client ,通过 Socket 向我们的主 APP 发送数据。

Extension 里面,我们拿到 ReplayKit 框架上报的屏幕视频数据后:

//
//  SampleHandler.m
//  SocketReply
//
//  Created by Sun on 2020/5/19.
//  Copyright © 2020 RongCloud. All rights reserved.
//


#import "SampleHandler.h"
#import "RongRTCClientSocket.h"
@interface SampleHandler()

/**
 Client Socket
 */
@property (nonatomic, strong) RongRTCClientSocket *clientSocket;

@end

@implementation SampleHandler

- (void)broadcastStartedWithSetupInfo:(NSDictionary *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
    self.clientSocket = [[RongRTCClientSocket alloc] init];
    [self.clientSocket createCliectSocket];
}

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            [self sendData:sampleBuffer];
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

- (void)sendData:(CMSampleBufferRef)sampleBuffer {
    [self.clientSocket encodeBuffer:sampleBuffer];
}

@end

可见 ,这里我们创建了一个 Client Socket,然后拿到屏幕共享的视频 sampleBuffer 之后,通过 Socket 发给我们的主 App,这就是屏幕共享的流程。

1.2 Local Socket 的使用

//
//  RongRTCSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCSocket.h"
#import 
#import 
#import 
#import 
#import 
#import "RongRTCThread.h"

@interface RongRTCSocket()

/**
 receive thread
 */
@property (nonatomic, strong) RongRTCThread *receiveThread;

@end

@implementation RongRTCSocket
- (int)createSocket {
    int socket = socket(AF_INET, SOCK_STREAM, 0);
    self.socket = socket;
    if (self.socket == -1) {
        close(self.socket);
        NSLog(@"socket error : %d", self.socket);
    }
    
    self.receiveThread = [[RongRTCThread alloc] init];
    [self.receiveThread run];
    return socket;
}

- (void)setSendBuffer {
    int optVal = 1024 * 1024 * 2;
    int optLen = sizeof(int);
    int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDBUF, (char *)&optVal,optLen);
    NSLog(@"set send buffer:%d", res);
}

- (void)setReceiveBuffer {
    int optVal = 1024 * 1024 * 2;
    int optLen = sizeof(int);
    int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVBUF, (char*)&optVal,optLen );
    NSLog(@"set send buffer:%d",res);
}

- (void)setSendingTimeout {
    struct timeval timeout = {10,0};
    int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(int));
    NSLog(@"set send timeout:%d", res);
}

- (void)setReceiveTimeout {
    struct timeval timeout = {10, 0};
    int  res = setsockopt(self.socket, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(int));
    NSLog(@"set send timeout:%d", res);
}

- (BOOL)connect {
    NSString *serverHost = [self ip];
    struct hostent *server = gethostbyname([serverHost UTF8String]);
    if (server == NULL) {
        close(self.socket);
        NSLog(@"get host error");
        return NO;
    }
    
    struct in_addr *remoteAddr = (struct in_addr *)server->h_addr_list[0];
    struct sockaddr_in addr;
    addr.sin_family = AF_INET;
    addr.sin_addr = *remoteAddr;
    addr.sin_port = htons(CONNECTPORT);
    int res = connect(self.socket, (struct sockaddr *) &addr, sizeof(addr));
    if (res == -1) {
        close(self.socket);
        NSLog(@"connect error");
        return NO;
    }
    
    NSLog(@"socket connect to server success");
    return YES;
}

- (BOOL)bind {
    struct sockaddr_in client;
    client.sin_family = AF_INET;
    NSString *ipStr = [self ip];
    if (ipStr.length <= 0) {
        return NO;
    }
    
    const char *ip = [ipStr cStringUsingEncoding:NSASCIIStringEncoding];
    client.sin_addr.s_addr = inet_addr(ip);
    client.sin_port = htons(CONNECTPORT);
    int bd = bind(self.socket, (struct sockaddr *) &client, sizeof(client));
    if (bd == -1) {
        close(self.socket);
        NSLog(@"bind error: %d", bd);
        return NO;
    }
    return YES;
}

- (BOOL)listen {
    int ls = listen(self.socket, 128);
    if (ls == -1) {
        close(self.socket);
        NSLog(@"listen error: %d", ls);
        return NO;
    }
    return YES;
}

- (void)receive {
    dispatch_async(dispatch_get_global_queue(0, 0), ^{
        [self receiveData];
    });
}

- (NSString *)ip {
    NSString *ip = nil;
    struct ifaddrs *addrs = NULL;
    struct ifaddrs *tmpAddrs = NULL;
    BOOL res = getifaddrs(&addrs);
    if (res == 0) {
        tmpAddrs = addrs;
        while (tmpAddrs != NULL) {
            if (tmpAddrs->ifa_addr->sa_family == AF_INET) {
                // Check if interface is en0 which is the wifi connection on the iPhone
                NSLog(@"%@", [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]);
                if ([[NSString stringWithUTF8String:tmpAddrs->ifa_name] isEqualToString:@"en0"]) {
                    // Get NSString from C String
                    ip = [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)];
                }
            }
            tmpAddrs = tmpAddrs->ifa_next;
        }
    }
    
    // Free memory
    freeifaddrs(addrs);
    NSLog(@"%@",ip);
    return ip;
}

- (void)close {
    int res = close(self.socket);
    NSLog(@"shut down: %d", res);
}

- (void)receiveData {
}

- (void)dealloc {
    [self.receiveThread stop];
}
@end

首先创建了一个 Socket 的父类,然后用 Server SocketClient Socket 分别继承类来实现链接、绑定等操作。可以看到有些数据可以设置,有些则不用,这里不是核心,核心是怎样收发数据。

1.3 发送屏幕共享数据

//
//  RongRTCClientSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCClientSocket.h"
#import 
#import 
#import 
#import 
#import 
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoEncoder.h"

@interface RongRTCClientSocket()  {
    pthread_mutex_t lock;
}

/**
 video encoder
 */
@property (nonatomic, strong) RongRTCVideoEncoder *encoder;

/**
 encode queue
 */
@property (nonatomic, strong) dispatch_queue_t encodeQueue;

@end

@implementation RongRTCClientSocket

- (BOOL)createClientSocket {
    if ([self createSocket] == -1) {
        return NO;
    }
    
    BOOL isC = [self connect];
    [self setSendBuffer];
    [self setSendingTimeout];
    
    if (isC) {
        _encodeQueue = dispatch_queue_create("cn.rongcloud.encodequeue", NULL);
        [self createVideoEncoder];
        return YES;
    } else {
        return NO;
    }
}

- (void)createVideoEncoder {
    self.encoder = [[RongRTCVideoEncoder alloc] init];
    self.encoder.delegate = self;
    
    RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
    settings.width = 720;
    settings.height = 1280;
    settings.startBitrate = 300;
    settings.maxFramerate = 30;
    settings.minBitrate = 1000;
    [self.encoder configWithSettings:settings onQueue:_encodeQueue];
}

- (void)clientSend:(NSData *)data {
    //data length
    NSUInteger dataLength = data.length;
    
    // data header struct
    DataHeader dataH;
    memset((void *)&dataH, 0, sizeof(dataH));
    
    // pre
    PreHeader preH;
    memset((void *)&preH, 0, sizeof(preH));
    preH.pre[0] = '&';
    preH.dataLength = dataLength;
    
    dataH.preH = preH;
    
    // buffer
    int headerlength = sizeof(dataH);
    int totalLength = dataLength + headerlength;
    
    // srcbuffer
    Byte *src = (Byte *)[data bytes];
    
    // send buffer
    char *buffer = (char *)malloc(totalLength * sizeof(char));
    memcpy(buffer, &dataH, headerlength);
    memcpy(buffer + headerlength, src, dataLength);
    
    // tosend
    [self sendBytes:buffer length:totalLength];
    free(buffer);
}

- (void)encodeBuffer:(CMSampleBufferRef)sampleBuffer {
    [self.encoder encode:sampleBuffer];
}

- (void)sendBytes:(char *)bytes length:(int)length {
    LOCK(self->lock);
    int hasSendLength = 0;
    
    while (hasSendLength < length) {
        // connect socket success
        if (self.socket > 0) {
            // send
            int sendRes = send(self.socket, bytes, length - hasSendLength, 0);
            if (sendRes == -1 || sendRes == 0) {
                UNLOCK(self->lock);
                NSLog(@"send buffer error");
                [self close];
                break;
            }
            
            hasSendLength += sendRes;
            bytes += sendRes;
        } else {
            NSLog(@"client socket connect error");
            UNLOCK(self->lock);
        }
    }
    UNLOCK(self->lock); 
}

- (void)spsData:(NSData *)sps ppsData:(NSData *)pps {
    [self clientSend:sps];
    [self clientSend:pps];
}

- (void)naluData:(NSData *)naluData {
    [self clientSend:naluData];
}

- (void)deallo c{
    NSLog(@"dealoc cliect socket");
}

@end

这里的核心思想是拿到屏幕共享的数据之后,先进行压缩,当压缩完成后会通过回调上报给当前类。既而通过 clientSend 方法,发给主 App。发给主 App 的数据中自定义了一个头部,头部添加了一个前缀和一个每次发送字节的长度,当接收端收到数据包后解析即可。

- (void)clientSend:(NSData *)data {
    //data length
    NSUInteger dataLength = data.length;
    
    // data header struct
    DataHeader dataH;
    memset((void *)&dataH, 0, sizeof(dataH));
    
    // pre
    PreHeader preH;
    memset((void *)&preH, 0, sizeof(preH));
    preH.pre[0] = '&';
    preH.dataLength = dataLength;
    
    dataH.preH = preH;
    
    // buffer
    int headerlength = sizeof(dataH);
    int totalLength = dataLength + headerlength;
    
    // srcbuffer
    Byte *src = (Byte *)[data bytes];
    
    // send buffer
    char *buffer = (char *)malloc(totalLength * sizeof(char));
    memcpy(buffer, &dataH, headerlength);
    memcpy(buffer + headerlength, src, dataLength);
    
    // to send
    [self sendBytes:buffer length:totalLength];
    free(buffer);
}

1.4 接收屏幕共享数据

//
//  RongRTCServerSocket.m
//  SealRTC
//
//  Created by Sun on 2020/5/7.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCServerSocket.h"
#import 
#import 
#import 
#import 
#import 
#import 
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoDecoder.h"

@interface RongRTCServerSocket() 
{
    pthread_mutex_t lock;
    int _frameTime;
    CMTime _lastPresentationTime;
    Float64 _currentMediaTime;
    Float64 _currentVideoTime;
    dispatch_queue_t _frameQueue;
}

@property (nonatomic, assign) int acceptSocket;

/**
 data length
 */
@property (nonatomic, assign) NSUInteger dataLength;

/**
 timeData
 */
@property (nonatomic, strong) NSData *timeData;

/**
 decoder queue
 */
@property (nonatomic, strong) dispatch_queue_t decoderQueue;

/**
 decoder
 */
@property (nonatomic, strong) RongRTCVideoDecoder *decoder;

@end

@implementation RongRTCServerSocket

- (BOOL)createServerSocket {
    if ([self createSocket] == -1) {
        return NO;
    }
    
    [self setReceiveBuffer];
    [self setReceiveTimeout];
    BOOL isB = [self bind];
    BOOL isL = [self listen];
    
    if (isB && isL) {
        _decoderQueue = dispatch_queue_create("cn.rongcloud.decoderQueue", NULL);
        _frameTime = 0;
        [self createDecoder];
        [self receive];
        return YES;
    } else {
        return NO;
    }
}

- (void)createDecoder {
    self.decoder = [[RongRTCVideoDecoder alloc] init];
    self.decoder.delegate = self;
    RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
    settings.width = 720;
    settings.height = 1280;
    settings.startBitrate = 300;
    settings.maxFramerate = 30;
    settings.minBitrate = 1000;
    [self.decoder configWithSettings:settings onQueue:_decoderQueue];
}

- (void)receiveData {
    struct sockaddr_in rest;
    socklen_t rest_size = sizeof(struct sockaddr_in);
    self.acceptSocket = accept(self.socket, (struct sockaddr *) &rest, &rest_size);
    while (self.acceptSocket != -1) {
        DataHeader dataH;
        memset(&dataH, 0, sizeof(dataH));
        
        if (![self receiveData:(char *)&dataH length:sizeof(dataH)]) {
            continue;
        }
        
        PreHeader preH = dataH.preH;
        char pre = preH.pre[0];
        if (pre == '&') {
            // rongcloud socket
            NSUInteger dataLenght = preH.dataLength;
            char *buff = (char *)malloc(sizeof(char) * dataLenght);
            if ([self receiveData:(char *)buff length:dataLenght]) {
                NSData *data = [NSData dataWithBytes:buff length:dataLenght];
                [self.decoder decode:data];
                free(buff);
            }
        } else {
            NSLog(@"pre is not &");
            return;
        }
    }
}

- (BOOL)receiveData:(char *)data length:(NSUInteger)length {
    LOCK(lock);
    int receiveLength = 0;
    while (receiveLength < length) {
        ssize_t res = recv(self.acceptSocket, data, length - receiveLength, 0);
        if (res == -1 || res == 0) {
            UNLOCK(lock);
            NSLog(@"receive data error");
            break;
        }
        
        receiveLength += res;
        data += res;
    }
    
    UNLOCK(lock);
    return YES;
}

- (void)didGetDecodeBuffer:(CVPixelBufferRef)pixelBuffer {
    _frameTime += 1000;
    CMTime pts = CMTimeMake(_frameTime, 1000);
    CMSampleBufferRef sampleBuffer = [RongRTCBufferUtil sampleBufferFromPixbuffer:pixelBuffer time:pts];
    // Check to see if there is a problem with the decoded data. If the image appears, you are right.
    UIImage *image = [RongRTCBufferUtil imageFromBuffer:sampleBuffer];
    [self.delegate didProcessSampleBuffer:sampleBuffer];
    CFRelease(sampleBuffer);
}

- (void)close {
    int res = close(self.acceptSocket);
    self.acceptSocket = -1;
    NSLog(@"shut down server: %d", res);
    [super close];
}

- (void)dealloc {
    NSLog(@"dealoc server socket");
}

@end

主 App 通过 Socket 会持续收到数据包,再将数据包进行解码,将解码后的数据通过代理 didGetDecodeBuffer 代理方法回调给 App 层。App 层就可以通过融云 RongRTCLib 的发送自定义流方法将视频数据发送到对端。

1.5 VideotoolBox 硬编码

//
//  RongRTCVideoEncoder.m
//  SealRTC
//
//  Created by Sun on 2020/5/13.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCVideoEncoder.h"

#import "helpers.h"

@interface RongRTCVideoEncoder() {
    VTCompressionSessionRef _compressionSession;
    int _frameTime;
}

/**
 settings
 */
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;

/**
 callback queue
 */
@property (nonatomic , strong ) dispatch_queue_t callbackQueue;

- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer;
- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer;

@end

void compressionOutputCallback(void *encoder,
                               void *params,
                               OSStatus status,
                               VTEncodeInfoFlags infoFlags,
                               CMSampleBufferRef sampleBuffer) {
    RongRTCVideoEncoder *videoEncoder = (__bridge RongRTCVideoEncoder *)encoder;
    if (status != noErr) {
        return;
    }
    
    if (infoFlags & kVTEncodeInfo_FrameDropped) {
        return;
    }
    
    BOOL isKeyFrame = NO;
    CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
    
    if (attachments != nullptr && CFArrayGetCount(attachments)) {
        CFDictionaryRef attachment = static_cast(CFArrayGetValueAtIndex(attachments, 0)) ;
        isKeyFrame = !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync);
    }
    
    CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    CMBlockBufferRef contiguous_buffer = nullptr;
    
    if (!CMBlockBufferIsRangeContiguous(block_buffer, 0, 0)) {
        status = CMBlockBufferCreateContiguous(nullptr, block_buffer, nullptr, nullptr, 0, 0, 0, &contiguous_buffer);
        if (status != noErr) {
            return;
        }
    } else {
        contiguous_buffer = block_buffer;
        CFRetain(contiguous_buffer);
        block_buffer = nullptr;
    }
    
    size_t block_buffer_size = CMBlockBufferGetDataLength(contiguous_buffer);
    if (isKeyFrame) {
        [videoEncoder sendSpsAndPPSWithSampleBuffer:sampleBuffer];
    }
    
    if (contiguous_buffer) {
        CFRelease(contiguous_buffer);
    }
    
    [videoEncoder sendNaluData:sampleBuffer];
}


@implementation RongRTCVideoEncoder

@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;

- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(nonnull dispatch_queue_t)queue {
    self.settings = settings;
    if (queue) {
        _callbackQueue = queue;
    } else {
        _callbackQueue = dispatch_get_main_queue();
    }
    
    if ([self resetCompressionSession:settings]) {
        _frameTime = 0;
        return YES;
    } else {
        return NO;
    }
}

- (BOOL)resetCompressionSession:(RongRTCVideoEncoderSettings *)settings {
    [self destroyCompressionSession];
    OSStatus status = VTCompressionSessionCreate(nullptr, settings.width, settings.height, kCMVideoCodecType_H264, nullptr, nullptr, nullptr, compressionOutputCallback, (__bridge void * _Nullable)(self), &_compressionSession);
    if (status != noErr) {
        return NO;
    }
    
    [self configureCompressionSession:settings];
    return YES;
}

- (void)configureCompressionSession:(RongRTCVideoEncoderSettings *)settings {
    if (_compressionSession) {
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_RealTime, true);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AllowFrameReordering, false);
        
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, 10);
        uint32_t targetBps = settings.startBitrate * 1000;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, targetBps);
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ExpectedFrameRate, settings.maxFramerate);
        int bitRate = settings.width * settings.height * 3 * 4 * 4;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, bitRate);
        int bitRateLimit = settings.width * settings.height * 3 * 4;
        SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimit);
    }
}

- (void)encode:(CMSampleBufferRef)sampleBuffer {
    CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
    CMTime pts = CMTimeMake(self->_frameTime++, 1000);
    VTEncodeInfoFlags flags;
    OSStatus res = VTCompressionSessionEncodeFrame(self->_compressionSession,
                                                   imageBuffer,
                                                   pts,
                                                   kCMTimeInvalid,
                                                   NULL, NULL, &flags);
    
    if (res != noErr) {
        NSLog(@"encode frame error:%d", (int)res);
        VTCompressionSessionInvalidate(self->_compressionSession);
        CFRelease(self->_compressionSession);
        self->_compressionSession = NULL;
        return;
    }
}

- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
    const uint8_t *sps ;
    const uint8_t *pps;
    size_t spsSize ,ppsSize , spsCount,ppsCount;
    OSStatus spsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sps, &spsSize, &spsCount, NULL);
    OSStatus ppsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pps, &ppsSize, &ppsCount, NULL);
    if (spsStatus == noErr && ppsStatus == noErr) {
        const char bytes[] = "\x00\x00\x00\x01";
        size_t length = (sizeof bytes) - 1;
        
        NSMutableData *spsData = [NSMutableData dataWithCapacity:4+ spsSize];
        NSMutableData *ppsData  = [NSMutableData dataWithCapacity:4 + ppsSize];
        [spsData appendBytes:bytes length:length];
        [spsData appendBytes:sps length:spsSize];
        
        [ppsData appendBytes:bytes length:length];
        [ppsData appendBytes:pps length:ppsSize];
        if (self && self.callbackQueue) {
            dispatch_async(self.callbackQueue, ^{
                if (self.delegate && [self.delegate respondsToSelector:@selector(spsData:ppsData:)]) {
                    [self.delegate spsData:spsData ppsData:ppsData];
                }
            });
        }
    } else {
        NSLog(@"sps status:%@, pps status:%@", @(spsStatus), @(ppsStatus));
    }
}

- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer {
    size_t totalLength = 0;
    size_t lengthAtOffset=0;
    char *dataPointer;
    CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    OSStatus status1 = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPointer);
    
    if (status1 != noErr) {
        NSLog(@"video encoder error, status = %d", (int)status1);
        return;
    }
    
    static const int h264HeaderLength = 4;
    size_t bufferOffset = 0;
    
    while (bufferOffset < totalLength - h264HeaderLength) {
        uint32_t naluLength = 0;
        memcpy(&naluLength, dataPointer + bufferOffset, h264HeaderLength);
        naluLength = CFSwapInt32BigToHost(naluLength);

        const char bytes[] = "\x00\x00\x00\x01";
        NSMutableData *naluData = [NSMutableData dataWithCapacity:4 + naluLength];
        [naluData appendBytes:bytes length:4];
        [naluData appendBytes:dataPointer + bufferOffset + h264HeaderLength length:naluLength];
        
        dispatch_async(self.callbackQueue, ^{
            if (self.delegate && [self.delegate respondsToSelector:@selector(naluData:)]) {
                [self.delegate naluData:naluData];
            }
        });
        
        bufferOffset += naluLength + h264HeaderLength;
    }
}

- (void)destroyCompressionSession {
    if (_compressionSession) {
        VTCompressionSessionInvalidate(_compressionSession);
        CFRelease(_compressionSession);
        _compressionSession = nullptr;
    }
}

- (void)dealloc {
    if (_compressionSession) {
        VTCompressionSessionCompleteFrames(_compressionSession, kCMTimeInvalid);
        VTCompressionSessionInvalidate(_compressionSession);
        CFRelease(_compressionSession);
        _compressionSession = NULL;
    }
}
@end

1.6 VideotoolBox 解码

//
//  RongRTCVideoDecoder.m
//  SealRTC
//
//  Created by Sun on 2020/5/14.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCVideoDecoder.h"
#import 
#import "helpers.h"

@interface RongRTCVideoDecoder() {
    uint8_t *_sps;
    NSUInteger _spsSize;
    uint8_t *_pps;
    NSUInteger _ppsSize;
    CMVideoFormatDescriptionRef _videoFormatDescription;
    VTDecompressionSessionRef _decompressionSession;
}

/**
 settings
 */
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;

/**
 callback queue
 */
@property (nonatomic, strong) dispatch_queue_t callbackQueue;

@end

void DecoderOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,
                           void * CM_NULLABLE sourceFrameRefCon,
                           OSStatus status,
                           VTDecodeInfoFlags infoFlags,
                           CM_NULLABLE CVImageBufferRef imageBuffer,
                           CMTime presentationTimeStamp,
                           CMTime presentationDuration ) {
    if (status != noErr) {
        NSLog(@" decoder callback error :%@", @(status));
        return;
    }
    
    CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;
    *outputPixelBuffer = CVPixelBufferRetain(imageBuffer);
    RongRTCVideoDecoder *decoder = (__bridge RongRTCVideoDecoder *)(decompressionOutputRefCon);
    dispatch_async(decoder.callbackQueue, ^{
        [decoder.delegate didGetDecodeBuffer:imageBuffer];
        CVPixelBufferRelease(imageBuffer);
    });
}

@implementation RongRTCVideoDecoder

@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;

- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(dispatch_queue_t)queue {
    self.settings = settings;
    if (queue) {
        _callbackQueue = queue;
    } else {
        _callbackQueue = dispatch_get_main_queue();
    }
    return YES;
}

- (BOOL)createVT {
    if (_decompressionSession) {
        return YES;
    }
    
    const uint8_t * const parameterSetPointers[2] = {_sps, _pps};
    const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};
    int naluHeaderLen = 4;
    OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_videoFormatDescription );
    if (status != noErr) {
        NSLog(@"CMVideoFormatDescriptionCreateFromH264ParameterSets error:%@", @(status));
        return false;
    }
    
    NSDictionary *destinationImageBufferAttributes =
                                        @{
                                            (id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange],
                                            (id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:self.settings.width],
                                            (id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:self.settings.height],
                                            (id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]
                                        };
                                        
    VTDecompressionOutputCallbackRecord CallBack;
    CallBack.decompressionOutputCallback = DecoderOutputCallback;
    CallBack.decompressionOutputRefCon = (__bridge void * _Nullable)(self);
    status = VTDecompressionSessionCreate(kCFAllocatorDefault, _videoFormatDescription, NULL, (__bridge CFDictionaryRef _Nullable)(destinationImageBufferAttributes), &CallBack, &_decompressionSession);

    if (status != noErr) {
        NSLog(@"VTDecompressionSessionCreate error:%@", @(status));
        return false;
    }
    
    status = VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);
    return YES;
}

- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {
    CVPixelBufferRef outputPixelBuffer = NULL;
    CMBlockBufferRef blockBuffer = NULL;
    CMBlockBufferFlags flag0 = 0;
    
    OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);
    
    if (status != kCMBlockBufferNoErr) {
        NSLog(@"VCMBlockBufferCreateWithMemoryBlock code=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }
    
    CMSampleBufferRef sampleBuffer = NULL;
    const size_t sampleSizeArray[] = {frameSize};
    
    status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _videoFormatDescription, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);
    
    if (status != noErr || !sampleBuffer) {
        NSLog(@"CMSampleBufferCreateReady failed status=%d", (int)status);
        CFRelease(blockBuffer);
        return outputPixelBuffer;
    }
    
    VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;
    VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous;
    
    status = VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);
    
    if (status == kVTInvalidSessionErr) {
        NSLog(@"decode frame error with session err status =%d", (int)status);
        [self resetVT];
    } else  {
        if (status != noErr) {
            NSLog(@"decode frame error with  status =%d", (int)status);
        }
    }

    CFRelease(sampleBuffer);
    CFRelease(blockBuffer);
    
    return outputPixelBuffer;
}

- (void)resetVT {
    [self destorySession];
    [self createVT];
}

- (void)decode:(NSData *)data {
    uint8_t *frame = (uint8_t*)[data bytes];
    uint32_t length = data.length;
    uint32_t nalSize = (uint32_t)(length - 4);
    uint32_t *pNalSize = (uint32_t *)frame;
    *pNalSize = CFSwapInt32HostToBig(nalSize);
    
    int type = (frame[4] & 0x1F);
    CVPixelBufferRef pixelBuffer = NULL;
    switch (type) {
        case 0x05:
            if ([self createVT]) {
                pixelBuffer= [self decode:frame withSize:length];
            }
            break;
        case 0x07:
            self->_spsSize = length - 4;
            self->_sps = (uint8_t *)malloc(self->_spsSize);
            memcpy(self->_sps, &frame[4], self->_spsSize);
            break;
        case 0x08:
            self->_ppsSize = length - 4;
            self->_pps = (uint8_t *)malloc(self->_ppsSize);
            memcpy(self->_pps, &frame[4], self->_ppsSize);
            break;
        default:
            if ([self createVT]) {
                pixelBuffer = [self decode:frame withSize:length];
            }
            break;
    }
}

- (void)dealloc {
    [self destorySession];
}

- (void)destorySession {
    if (_decompressionSession) {
        VTDecompressionSessionInvalidate(_decompressionSession);
        CFRelease(_decompressionSession);
        _decompressionSession = NULL;
    }
}

@end

1.7 工具类

//
//  RongRTCBufferUtil.m
//  SealRTC
//
//  Created by Sun on 2020/5/8.
//  Copyright © 2020 RongCloud. All rights reserved.
//

#import "RongRTCBufferUtil.h"

// 下面的这些方法,一定要记得release,有的没有在方法里面release,但是在外面release了,要不然会内存泄漏

@implementation RongRTCBufferUtil

+ (UIImage *)imageFromBuffer:(CMSampleBufferRef)buffer {    
    CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(buffer);
    CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];
    
    CIContext *temporaryContext = [CIContext contextWithOptions:nil];
    CGImageRef videoImage = [temporaryContext createCGImage:ciImage fromRect:CGRectMake(0, 0, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer))];
    
    UIImage *image = [UIImage imageWithCGImage:videoImage];
    CGImageRelease(videoImage);
    
    return image;
}

+ (UIImage *)compressImage:(UIImage *)image newWidth:(CGFloat)newImageWidth {
    if (!image) return nil;
    
    float imageWidth = image.size.width;
    float imageHeight = image.size.height;
    float width = newImageWidth;
    float height = image.size.height/(image.size.width/width);
    float widthScale = imageWidth /width;
    float heightScale = imageHeight /height;
    UIGraphicsBeginImageContext(CGSizeMake(width, height));
    
    if (widthScale > heightScale) {
        [image drawInRect:CGRectMake(0, 0, imageWidth /heightScale , height)];
    }
    else {
        [image drawInRect:CGRectMake(0, 0, width , imageHeight /widthScale)];
    }
    
    UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();
    return newImage;
}

+ (CVPixelBufferRef)CVPixelBufferRefFromUiImage:(UIImage *)img {
    CGSize size = img.size;
    CGImageRef image = [img CGImage];
    
    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil];
    CVPixelBufferRef pxbuffer = NULL;
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer);
    
    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
    
    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(pxdata != NULL);
    
    CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
    CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, 4*size.width, rgbColorSpace, kCGImageAlphaPremultipliedFirst);
    NSParameterAssert(context);
    
    CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image)), image);
    
    CGColorSpaceRelease(rgbColorSpace);
    CGContextRelease(context);
    
    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
    
    return pxbuffer;
}

+ (CMSampleBufferRef)sampleBufferFromPixbuffer:(CVPixelBufferRef)pixbuffer time:(CMTime)time {
    CMSampleBufferRef sampleBuffer = NULL;
    
    //获取视频信息
    CMVideoFormatDescriptionRef videoInfo = NULL;
    OSStatus result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixbuffer, &videoInfo);
    CMTime currentTime = time;
  
    //    CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
    CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
    result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,pixbuffer, true, NULL, NULL, videoInfo, &timing, &sampleBuffer);
    CFRelease(videoInfo);
    return sampleBuffer;
}

+ (size_t)getCMTimeSize {
    size_t size = sizeof(CMTime);
    return size;
}

@end

此工具类中实现是由 CPU 处理,当进行 CMSampleBufferRefUIImageUIImageCVPixelBufferRefCVPixelBufferRefCMSampleBufferRef 以及裁剪图片时,这里需要注意将使用后的对象及时释放,否则会出现内存大量泄漏。

2. 视频发送

2.1 准备阶段

使用融云的 RongRTCLib 的前提需要一个 AppKey,请在官网(https://www.rongcloud.cn/)获取,通过 AppKey 取得 token 之后进行 IM 连接,在连接成功后加入 RTC 房间,这是屏幕共享发送的准备阶段。

- (void)broadcastStartedWithSetupInfo:(NSDictionary *)setupInfo {
    // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
    
    // 请填写您的 AppKey
    self.appKey = @"";
    // 请填写用户的 Token
    self.token = @"";
    // 请指定房间号
    self.roomId = @"123456";
    
    [[RCIMClient sharedRCIMClient] initWithAppKey:self.appKey];
    [[RCIMClient sharedRCIMClient] setLogLevel:RC_Log_Level_Verbose];
    
    // 连接 IM
    [[RCIMClient sharedRCIMClient] connectWithToken:self.token
                                           dbOpened:^(RCDBErrorCode code) {
        NSLog(@"dbOpened: %zd", code);
    } success:^(NSString *userId) {
        NSLog(@"connectWithToken success userId: %@", userId);
        // 加入房间
        [[RCRTCEngine sharedInstance] joinRoom:self.roomId
                                    completion:^(RCRTCRoom * _Nullable room, RCRTCCode code) {
            self.room = room;
            self.room.delegate = self;
            [self publishScreenStream];
        }];
    } error:^(RCConnectErrorCode errorCode) {
        NSLog(@"ERROR status: %zd", errorCode);
    }];
}

如上是连接 IM 和加入 RTC 房间的全过程,其中还包含调用发布自定义视频 [self publishScreenStream]; 此方法在加入房间成功后才可以进行。

- (void)publishScreenStream {
    RongRTCStreamParams *param = [[RongRTCStreamParams alloc] init];
    param.videoSizePreset = RongRTCVideoSizePreset1280x720;
    self.videoOutputStream = [[RongRTCAVOutputStream alloc] initWithParameters:param tag:@"RongRTCScreenVideo"];
    [self.room publishAVStream:self.videoOutputStream extra:@"" completion:^(BOOL isSuccess, RongRTCCode desc) {
        if (isSuccess) {
            NSLog(@"发布自定义流成功");
        }
    }];
}

自定义一个 RongRTCAVOutputStream 流即可,使用此流发送屏幕共享数据。

2.2 开始发送屏幕共享数据

上面我们已经连接了融云的 IM 和加入了 RTC 房间,并且自定义了一个发送屏幕共享的自定义流,接下来,如何将此流发布出去呢?

- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
    switch (sampleBufferType) {
        case RPSampleBufferTypeVideo:
            // Handle video sample buffer
            [self.videoOutputStream write:sampleBuffer error:nil];
            break;
        case RPSampleBufferTypeAudioApp:
            // Handle audio sample buffer for app audio
            break;
        case RPSampleBufferTypeAudioMic:
            // Handle audio sample buffer for mic audio
            break;
        default:
            break;
    }
}

但我们接收到了苹果上报的数据之后,调用 RongRTCAVOutputStream 中的 write:error: 方法,将 sampleBuffer 发送给远端,至此,屏幕共享数据就发送出去啦。

[self.videoOutputStream write:sampleBuffer error:nil];

融云的核心代码就是通过上面的连接 IM,加入房间,发布自定义流,然后通过自定义流的 write:error: 方法将 sampleBuffer 发送出去。

不管是通过 ReplayKit 取得屏幕视频,还是使用 Socket 在进程间传输,都是为最终的 write:error: 服务。

总结

  1. Extension 内存是有限制的,最大 50M,所以在 Extension 里面处理数据需要格外注意内存释放;
  2. 如果 VideotoolBox 在后台解码一直失败,只需把 VideotoolBox 重启一下即可,此步骤在上面的代码中有体现;
  3. 如果不需要将 Extension 的数据传到主 App,只需在 Extension 里直接将流通过 RongRTCLib 发布出去即可,缺点是 Extension 中发布自定义流的用户与主 App 中的用户不是同一个,这也是上面通过 Socket 将数据传递给主 App 要解决的问题;
  4. 如果主 App 需要拿到屏幕共享的数据处理,使用 Socket 将流先发给主 App,然后在主 App 里面通过 RongRTCLib 将流发出去。

最后附上 Demo

你可能感兴趣的:(【融云分析】iOS 基于实时音视频 SDK 实现屏幕共享功能)