GB28181设备接入侧如何对接外部编码后音视频数据并实现预览播放
技术背景
我们在对接GB28181设备接入模块的时候，遇到这样的技术诉求，好多开发者期望能提供编码后（H.264/H.265、AAC/PCMA）数据对接，确保外部采集设备，比如无人机类似回调过来的数据，直接通过模块，对接到GB28181平台侧，此外，还期望不支持或者内网没有外部网络权限的RTSP设备，也能间接接入到国标平台。
技术实现
编码后音视频数据
本文以Android平台为例，基于上述诉求，我们设计的接口如下，简单来说，GB28181交互流程不变，只要提供数据接入接口即可：
/**
* 设置编码后视频数据(H.264)
*
* @param codec_id, H.264对应 1
*
* @param data 编码后的video数据
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
*
* @param timestamp video timestamp
*
* @param pts Presentation Time Stamp, 显示时间戳
*
* @return {0} if successful
*/
public native int SmartPublisherPostVideoEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp, long pts);
/**
* 设置编码后视频数据(H.264)
*
* @param codec_id, H.264对应 1
*
* @param data 编码后的video数据
*
*@param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
*
* @param timestamp video timestamp
*
* @param pts Presentation Time Stamp, 显示时间戳
*
* @return {0} if successful
*/
public native int SmartPublisherPostVideoEncodedDataV2(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp, long pts,
byte[] sps, int sps_len,
byte[] pps, int pps_len);
/**
* 设置编码后视频数据(H.264)，如需录制编码后的数据，用此接口，且设置实际宽高
*
* @param codec_id, H.264对应 1
*
* @param data 编码后的video数据
*
*@param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0.
*
* @param timestamp video timestamp
*
* @param pts Presentation Time Stamp, 显示时间戳
*
* @param width, height: 编码后视频宽高
*
* @return {0} if successful
*/
public native int SmartPublisherPostVideoEncodedDataV3(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp, long pts,
byte[] sps, int sps_len,
byte[] pps, int pps_len,
int width, int height);
/**
* 设置音频数据(AAC/PCMA/PCMU/SPEEX)
*
* @param codec_id:
*
* NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
* NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
* NT_MEDIA_CODEC_ID_PCMU,
* NT_MEDIA_CODEC_ID_AAC,
* NT_MEDIA_CODEC_ID_SPEEX,
* NT_MEDIA_CODEC_ID_SPEEX_NB,
* NT_MEDIA_CODEC_ID_SPEEX_WB,
* NT_MEDIA_CODEC_ID_SPEEX_UWB,
*
* @param data audio数据
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
*
* @param timestamp video timestamp
*
* @param parameter_info 用于AAC special config信息填充
*
* @param parameter_info_size parameter info size
*
* @return {0} if successful
*/
public native int SmartPublisherPostAudioEncodedData(long handle, int codec_id, ByteBuffer data, int size, int is_key_frame, long timestamp,ByteBuffer parameter_info, int parameter_info_size);
/**
* 设置音频数据(AAC/PCMA/PCMU/SPEEX)
*
* @param codec_id:
*
* NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
* NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
* NT_MEDIA_CODEC_ID_PCMU,
* NT_MEDIA_CODEC_ID_AAC,
* NT_MEDIA_CODEC_ID_SPEEX,
* NT_MEDIA_CODEC_ID_SPEEX_NB,
* NT_MEDIA_CODEC_ID_SPEEX_WB,
* NT_MEDIA_CODEC_ID_SPEEX_UWB,
*
* @param data audio数据
*
* @param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
*
* @param timestamp video timestamp
*
* @param parameter_info 用于AAC special config信息填充
*
* @param parameter_info_size parameter info size
*
* @return {0} if successful
*/
public native int SmartPublisherPostAudioEncodedDataV2(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp,
byte[] parameter_info, int parameter_info_size);
/**
* 设置音频数据(AAC/PCMA/PCMU/SPEEX)
*
* @param codec_id:
*
* NT_MEDIA_CODEC_ID_AUDIO_BASE = 0x10000,
* NT_MEDIA_CODEC_ID_PCMA = NT_MEDIA_CODEC_ID_AUDIO_BASE,
* NT_MEDIA_CODEC_ID_PCMU,
* NT_MEDIA_CODEC_ID_AAC,
* NT_MEDIA_CODEC_ID_SPEEX,
* NT_MEDIA_CODEC_ID_SPEEX_NB,
* NT_MEDIA_CODEC_ID_SPEEX_WB,
* NT_MEDIA_CODEC_ID_SPEEX_UWB,
*
* @param data audio数据
*
* @param offset data的偏移
*
* @param size data length
*
* @param is_key_frame 是否I帧, if with key frame, please set 1, otherwise, set 0, audio忽略
*
* @param timestamp video timestamp
*
* @param parameter_info 用于AAC special config信息填充
*
* @param parameter_info_size parameter info size
*
* @param sample_rate 采样率,如果需要录像的话必须传正确的值
*
*@param channels 通道数, 如果需要录像的话必须传正确的值, 一般是1或者2
*
* @return {0} if successful
*/
public native int SmartPublisherPostAudioEncodedDataV3(long handle, int codec_id,
ByteBuffer data, int offset, int size,
int is_key_frame, long timestamp,
byte[] parameter_info, int parameter_info_size,
int sample_rate, int channels);拉取RTSP流接入到GB28181平台
简单那来说，把摄像机的RTSP流数据拉下来，然后回调编码后的数据到上层，上层根据GB28181数据格式要求，实现PS打包，然后通过对接GB28181平台信令和数据交互，国标平台侧需要预览的时候，信令交互后，拉RTSP即可。
如何拉流：
private boolean StartPull()
{
if ( isPulling )
return false;
if (!OpenPullHandle())
return false;
libPlayer.SmartPlayerSetAudioDataCallback(playerHandle, new PlayerAudioDataCallback());
libPlayer.SmartPlayerSetVideoDataCallback(playerHandle, new PlayerVideoDataCallback());
int is_pull_trans_code = 1;
libPlayer.SmartPlayerSetPullStreamAudioTranscodeAAC(playerHandle, is_pull_trans_code);
int startRet = libPlayer.SmartPlayerStartPullStream(playerHandle);
if (startRet != 0) {
Log.e(TAG, "Failed to start pull stream!");
if(!isPlaying && !isRecording && isPushing && !isRTSPPublisherRunning)
{
libPlayer.SmartPlayerClose(playerHandle);
playerHandle = 0;
}
return false;
}
isPulling = true;
return true;
}
private void StopPull()
{
if ( !isPulling )
return;
libPlayer.SmartPlayerStopPullStream(playerHandle);
if ( !isPlaying && !isRecording && !isPushing && !isRTSPPublisherRunning)
{
libPlayer.SmartPlayerClose(playerHandle);
playerHandle = 0;
}
isPulling = false;
}拉到的音视频数据，投递到GB28181接入模块：
class PlayerAudioDataCallback implements NTAudioDataCallback
{
private int audio_buffer_size = 0;
private int param_info_size = 0;
private ByteBuffer audio_buffer_ = null;
private ByteBuffer parameter_info_ = null;
@Override
public ByteBuffer getAudioByteBuffer(int size)
{
//Log.i("getAudioByteBuffer", "size: " + size);
if( size < 1 )
{
return null;
}
if ( size <= audio_buffer_size && audio_buffer_ != null )
{
return audio_buffer_;
}
audio_buffer_size = size + 512;
audio_buffer_size = (audio_buffer_size+0xf) & (~0xf);
audio_buffer_ = ByteBuffer.allocateDirect(audio_buffer_size);
// Log.i("getAudioByteBuffer", "size: " + size + " buffer_size:" + audio_buffer_size);
return audio_buffer_;
}
@Override
public ByteBuffer getAudioParameterInfo(int size)
{
//Log.i("getAudioParameterInfo", "size: " + size);
if(size < 1)
{
return null;
}
if ( size <= param_info_size && parameter_info_ != null )
{
return parameter_info_;
}
param_info_size = size + 32;
param_info_size = (param_info_size+0xf) & (~0xf);
parameter_info_ = ByteBuffer.allocateDirect(param_info_size);
//Log.i("getAudioParameterInfo", "size: " + size + " buffer_size:" + param_info_size);
return parameter_info_;
}
public void onAudioDataCallback(int ret, int audio_codec_id, int sample_size, int is_key_frame, long timestamp, int sample_rate, int channel, int parameter_info_size, long reserve)
{
if ( audio_buffer_ == null)
return;
audio_buffer_.rewind();
if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning)) {
libPublisher.SmartPublisherPostAudioEncodedData(publisherHandle, audio_codec_id, audio_buffer_, sample_size, is_key_frame, timestamp, parameter_info_, parameter_info_size);
}
}
}
class PlayerVideoDataCallback implements NTVideoDataCallback
{
private int video_buffer_size = 0;
private ByteBuffer video_buffer_ = null;
@Override
public ByteBuffer getVideoByteBuffer(int size)
{
if( size < 1 )
{
return null;
}
if ( size <= video_buffer_size && video_buffer_ != null )
{
return video_buffer_;
}
video_buffer_size = size + 1024;
video_buffer_size = (video_buffer_size+0xf) & (~0xf);
video_buffer_ = ByteBuffer.allocateDirect(video_buffer_size);
return video_buffer_;
}
public void onVideoDataCallback(int ret, int video_codec_id, int sample_size, int is_key_frame, long timestamp, int width, int height, long presentation_timestamp)
{
if ( video_buffer_ == null)
return;
video_buffer_.rewind();
if ( ret == 0 && (isPushing || isRTSPPublisherRunning || isGB28181StreamRunning) ) {
libPublisher.SmartPublisherPostVideoEncodedData(publisherHandle, video_codec_id, video_buffer_, sample_size, is_key_frame, timestamp, presentation_timestamp);
}
}
}如何预览播放外部音视频数据？
除了想把编码后的音视频数据转至GB28181外，有些场景下，还需要本地预览甚至对数据做二次处理（视频分析、实时水印字符叠加等，然后二次编码），基于这样的场景诉求，我们实现了Android平台外部编码数据实时预览播放模块。
外部（H.264/H.265）投递接口设计如下：
// SmartPlayerJniV2.java
// Author: daniusdk.com
/**
* 投递视频包给外部Live Source
*
* @param codec_id: 编码id, 当前仅支持H264和H265, 1:H264, 2:H265
*
* @param packet: 视频数据, ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如:
* 0x00000001 nal_unit 0x00000001 ...
* H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
* H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
*
* @param offset: 偏移量
* @param size: packet size
* @param timestamp_ms: 时间戳, 单位毫秒
* @param is_timestamp_discontinuity: 是否时间戳间断，0:未间断，1:间断
* @param is_key: 是否是关键帧, 0:非关键帧, 1:关键帧
* @param extra_data: 可选参数，可传null, 对于H264关键帧包, 如果packet不含sps和pps, 可传0x00000001 sps 0x00000001 pps
* ,对于H265关键帧包, 如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps
* @param extra_data_size: extra_data size
* @param width: 图像宽, 可传0
* @param height: 图像高, 可传0
*
* @return {0} if successful
*/
public native int PostVideoPacketByteBuffer(long handle, int codec_id,
java.nio.ByteBuffer packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key,
byte[] extra_data, int extra_data_size, int width, int height);
/*
* 请参考 PostVideoPacketByteBuffer说明
*/
public native int PostVideoPacketByteArray(long handle, int codec_id,
byte[] packet, int offset, int size, long timestamp_ms, int is_timestamp_discontinuity, int is_key,
byte[] extra_data, int extra_data_size, int width, int height);PostVideoPacketByteBuffer()和PostVideoPacketByteArray()接口设计基本类似，唯一的区别在于，一个数据类型是ByteBuffer，一个是byte数组。
其中codec_id，系编码id，目前仅支持H.264和H.265类型。
packet视频数据，需要注意的是，ByteBuffer必须是DirectBuffer, 包格式请参考H264/H265 Annex B Byte stream format, 例如:
0x00000001 nal_unit 0x00000001 ...
H264 IDR: 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
H265 IDR: 0x00000001 vps 0x00000001 sps 0x00000001 pps 0x00000001 IDR_nal_unit .... 或 0x00000001 IDR_nal_unit ....
extra_data: 可选参数，可传null, 对于H264关键帧包，如果packet不含sps和pps，可传0x00000001 sps 0x00000001 pps，对于H265关键帧包，如果packet不含vps,sps和pps, 可传0x00000001 vps 0x00000001 sps 0x00000001 pps
音频（AAC/PCMA/PCMU）投递接口设计如下：
/**
* 投递音频包给外部Live source, 注意ByteBuffer对象必须是DirectBuffer
*
* @param handle: return value from SmartPlayerOpen()
*
* @param codec_id: 编码id, 当前支持PCMA、PCMU和AAC, 65536:PCMA, 65537:PCMU, 65538:AAC
* @param packet: 音频数据
* @param offset：packet偏移量
* @param size: packet size
* @param pts_ms: 时间戳, 单位毫秒
* @param is_pts_discontinuity: 是否时间戳间断，false:未间断，true:间断
* @param extra_data: 如果是AAC的话，需要传 Audio Specific Configuration
* @param extra_data_offset: extra_data 偏移量
* @param extra_data_size: extra_data size
* @param sample_rate: 采样率
* @param channels: 通道数
*
* @return {0} if successful
*/
public native int PostAudioPacket(long handle, int codec_id,
java.nio.ByteBuffer packet, int offset, int size, long pts_ms, boolean is_pts_discontinuity,
java.nio.ByteBuffer extra_data, int extra_data_offset, int extra_data_size, int sample_rate, int channels);
/*
* 投递音频包给外部Live source, byte数组版本, 具体请参考PostAudioPacket
*
* @param is_pts_discontinuity: 是否时间戳间断，0:未间断，1:间断
* @return {0} if successful
*/
public native int PostAudioPacketByteArray(long handle, int codec_id,
byte[] packet, int offset, int size, long pts_ms, int is_pts_discontinuity,
byte[] extra_data, int extra_data_size, int sample_rate, int channels);总结
通过以上描述，大家可以看到，GB/T 28181音视频数据源接入，无论是编码前还是编码后数据，或外部RTSP流数据，包括数据预览，如果有技术积累的话，实现起来也没那么麻烦，感兴趣的开发者，可以尝试看。