使用Gstreamer处理RTSP视频流

RTSP视频流处理方法

这里使用Gstreamer + OpenCV来处理RTSP视频流，因此对Gstreamer进行调查。

1. Gstreamer整体框架

Gstreamer是一个用于开发流式多媒体应用的开源框架，采用了基于插件（plugin）和管道（pipeline）的体系结构，框架中的所有的功能模块都被实现成可以插拔的组件（component）， 并且能够很方便地安装到任意一个管道上。由于所有插件都通过管道机制进行统一的数据交换，因此很容易利用已有的各种插件“组装”出一个功能完善的多媒体应用程序。
Nvidia为Gstreamer开发了许多plugin，这些plugin能够利用Nvidia硬件进行加速。Nvidia的deepstream就是基于gstreamer开发的。

下图是对基于Gstreamer框架的应用的简单分层：

1.1 Media Applications

最上面一层为应用，比如gstreamer自带的一些工具（gst-launch，gst-inspect等），以及基于gstreamer封装的库（gst-player，gst-rtsp-server，gst-editing-services等)根据不同场景实现的应用。

1.2 Core Framework

中间一层为Core Framework，主要提供：

• 上层应用所需接口
• Plugin的框架
• Pipline的框架
• 数据在各个Element间的传输及处理机制
• 多个媒体流（Streaming）间的同步（比如音视频同步）
• 其他各种所需的工具库

1.3 Plugins

最下层为各种插件，实现具体的数据处理及音视频输出，应用不需要关注插件的细节，会由Core Framework层负责插件的加载及管理。主要分类为：

• Protocols：负责各种协议的处理，file，http，rtsp等。
• Sources：负责数据源的处理，alsa，v4l2，tcp/udp等。
• Formats：负责媒体容器的处理，avi，mp4，ogg等。
• Codecs：负责媒体的编解码，mp3，vorbis等。
• Filters：负责媒体流的处理，converters，mixers，effects等。
• Sinks：负责媒体流输出到指定设备或目的地，alsa，xvideo，tcp/udp等。

2. Gstreamer组件

Gstreamer由许多基础的组件构成。

2.1 Element

Element是Gstreamer中最重要的对象类型之一。一个element实现了一个功能（读取文件，解码，输出等），程序需要创建多个element，并按顺序将其串连起来，构成一个完整的pipeline。

• Source Element 数据源元件　只有输出端，它仅能用来产生供管道消费的数据，而不能对数据做任何处理。一个典型的数据源元件的例子是音频捕获单元，它负责从声卡读取原始的音频数据，然后作为数据源提供给其它模块使用。
  
• Filter Element 过滤器元件　既有输入端又有输出端，它从输入端获得相应的数据，并在经过特殊处理之后传递给输出端。一个典型的过滤器元件的例子是音频编码单元，它首先从外界获得音频数据，然后根据特定的压缩算法对其进行编码，最后再将编码后的结果提供给其它模块使用。
  
• Sink Element 接收器元件　只有输入端，它仅具有消费数据的能力，是整条媒体管道的终端。一个典型的接收器元件的例子是音频回放单元，它负责将接收到的数据写到声卡上，通常这也是音频处理过程中的最后一个环节。
  

2.2 Pad

Pad是一个element的输入/输出接口，分为src pad（生产数据）和sink pad（消费数据）两种。
两个element必须通过pad才能连接起来，pad拥有当前element能处理数据类型的能力（capabilities），会在连接时通过比较src pad和sink pad中所支持的能力，来选择最恰当的数据类型用于传输。如果element不支持，程序会直接退出。
在element通过pad连接成功后，数据会从上一个element的src pad传到下一个element的sink pad然后进行处理。当element支持多种数据处理能力时，我们可以通过Cap来指定数据类型.

例如，下面的命令通过Cap指定了视频的宽高，videotestsrc会根据指定的宽高产生相应数据：

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1280,height=720" ! autovideosink

2.3 Bin和Pipeline

Bin是一个容器，用于管理多个element，改变bin的状态时，bin会自动去修改所包含的element的状态，也会转发所收到的消息。如果没有bin，我们需要依次操作我们所使用的element。通过bin降低了应用的复杂度。
Pipeline继承自bin，为程序提供一个bus用于传输消息，并且对所有子element进行同步。当将pipeline的状态设置为PLAYING时，pipeline会在一个/多个新的线程中通过element处理数据。

下面通过一个文件播放的例子来熟悉上述提及的概念：
测试文件：sintel_trailer-480p.ogv

gst-launch-1.0 filesrc location=sintel_trailer-480p.ogv ! oggdemux name=demux ! queue ! vorbisdec ! autoaudiosink demux. ! queue ! theoradec ! videoconvert ! autovideosink

通过上面的命令播放文件时，会创建如下pipeline：

可以看到这个pipeline由8个element构成，每个element都实现各自的功能：

• filesrc读取文件
• oggdemux解析文件，分别提取audio，video数据
• queue缓存数据
• vorbisdec解码audio
• autoaudiosink自动选择音频设备并输出
• theoradec解码video
• videoconvert转换video数据格式
• autovideosink自动选择显示设备并输出

不同的element拥有不同数量及类型的pad，只有src pad的element被称为source element，只有sink pad的被称为sink element。

element可以同时拥有多个相同的pad，例如oggdemux在解析文件后，会将audio，video通过不同的pad输出。

3. gstreamer tools

Gstreamer自带了gst-inspect-1.0和gst-launch-1.0等其他命令行工具，我们可以使用这些工具完成常见的处理任务。

3.1 gst-inspect-1.0

查看gstreamer的plugin、element的信息。直接将plugin/element作为参数，会列出其详细信息，包括plugin的功能、Pad的输入输出类型、plugin的属性等。
如果不跟任何参数，会列出当前系统gstreamer所能查找到的所有插件。

# gst-inspect-1.0 rtspsrc
Factory Details:
  Rank                     none (0)
  Long-name                RTSP packet receiver
  Klass                    Source/Network
  Description              Receive data over the network via RTSP (RFC 2326)
  Author                   Wim Taymans , Thijs Vermeir , Lutz Mueller 

Plugin Details:
  Name                     rtsp
  Description              transfer data via RTSP
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstrtsp.so
  Version                  1.14.5
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2019-05-29
  Binary package           GStreamer Good Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBin
                         +----GstRTSPSrc

Implemented Interfaces:
  GstChildProxy
  GstURIHandler

Pad Templates:
  SRC template: 'stream_%u'
    Availability: Sometimes
    Capabilities:
      application/x-rtp
      application/x-rdt

Element has no clocking capabilities.

URI handling capabilities:
  Element can act as source.
  Supported URI protocols:
    rtsp
    rtspu
    rtspt
    rtsph
    rtsp-sdp
    rtsps
    rtspsu
    rtspst
    rtspsh

Pads:
  none

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "rtspsrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  async-handling      : The bin will handle Asynchronous state changes
                        flags: readable, writable
                        Boolean. Default: false
  message-forward     : Forwards all children messages
                        flags: readable, writable
                        Boolean. Default: false
  location            : Location of the RTSP url to read
                        flags: readable, writable
                        String. Default: null
  protocols           : Allowed lower transport protocols
                        flags: readable, writable
                        Flags "GstRTSPLowerTrans" Default: 0x00000007, "tcp+udp-mcast+udp"
                           (0x00000000): unknown          - GST_RTSP_LOWER_TRANS_UNKNOWN
                           (0x00000001): udp              - GST_RTSP_LOWER_TRANS_UDP
                           (0x00000002): udp-mcast        - GST_RTSP_LOWER_TRANS_UDP_MCAST
                           (0x00000004): tcp              - GST_RTSP_LOWER_TRANS_TCP
                           (0x00000010): http             - GST_RTSP_LOWER_TRANS_HTTP
                           (0x00000020): tls              - GST_RTSP_LOWER_TRANS_TLS
  debug               : Dump request and response messages to stdout(DEPRECATED: Printed all RTSP message to gstreamer log as 'log' level)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  retry               : Max number of retries when allocating RTP ports.
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 65535 Default: 20
  timeout             : Retry TCP transport after UDP timeout microseconds (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 5000000
  tcp-timeout         : Fail after timeout microseconds on TCP connections (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 20000000
  latency             : Amount of ms to buffer
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 2000
  drop-on-latency     : Tells the jitterbuffer to never exceed the given latency in size
                        flags: readable, writable
                        Boolean. Default: false
  connection-speed    : Network connection speed in kbps (0 = unknown)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551 Default: 0
  nat-method          : Method to use for traversing firewalls and NAT
                        flags: readable, writable
                        Enum "GstRTSPNatMethod" Default: 1, "dummy"
                           (0): none             - None
                           (1): dummy            - Send Dummy packets
  do-rtcp             : Send RTCP packets, disable for old incompatible server.
                        flags: readable, writable
                        Boolean. Default: true
  do-rtsp-keep-alive  : Send RTSP keep alive packets, disable for old incompatible server.
                        flags: readable, writable
                        Boolean. Default: true
  proxy               : Proxy settings for HTTP tunneling. Format: [http://][user:passwd@]host[:port]
                        flags: readable, writable
                        String. Default: null
  proxy-id            : HTTP proxy URI user id for authentication
                        flags: readable, writable
                        String. Default: null
  proxy-pw            : HTTP proxy URI user password for authentication
                        flags: readable, writable
                        String. Default: null
  rtp-blocksize       : RTP package size to suggest to server (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 65536 Default: 0
  user-id             : RTSP location URI user id for authentication
                        flags: readable, writable
                        String. Default: null
  user-pw             : RTSP location URI user password for authentication
                        flags: readable, writable
                        String. Default: null
  buffer-mode         : Control the buffering algorithm in use
                        flags: readable, writable
                        Enum "GstRTSPSrcBufferMode" Default: 3, "auto"
                           (0): none             - Only use RTP timestamps
                           (1): slave            - Slave receiver to sender clock
                           (2): buffer           - Do low/high watermark buffering
                           (3): auto             - Choose mode depending on stream live
                           (4): synced           - Synchronized sender and receiver clocks
  port-range          : Client port range that can be used to receive RTP and RTCP data, eg. 3000-3005 (NULL = no restrictions)
                        flags: readable, writable
                        String. Default: null
  udp-buffer-size     : Size of the kernel UDP receive buffer in bytes, 0=default
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 524288
  short-header        : Only send the basic RTSP headers for broken encoders
                        flags: readable, writable
                        Boolean. Default: false
  probation           : Consecutive packet sequence numbers to accept the source
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 2
  udp-reconnect       : Reconnect to the server if RTSP connection is closed when doing UDP
                        flags: readable, writable
                        Boolean. Default: true
  multicast-iface     : The network interface on which to join the multicast group
                        flags: readable, writable
                        String. Default: null
  ntp-sync            : Synchronize received streams to the NTP clock
                        flags: readable, writable
                        Boolean. Default: false
  use-pipeline-clock  : Use the pipeline running-time to set the NTP time in the RTCP SR messages(DEPRECATED: Use ntp-time-source property)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  sdes                : The SDES items of this session
                        flags: readable, writable
                        Boxed pointer of type "GstStructure"
  tls-validation-flags: TLS certificate validation flags used to validate the server certificate
                        flags: readable, writable
                        Flags "GTlsCertificateFlags" Default: 0x0000007f, "validate-all"
                           (0x00000001): unknown-ca       - G_TLS_CERTIFICATE_UNKNOWN_CA
                           (0x00000002): bad-identity     - G_TLS_CERTIFICATE_BAD_IDENTITY
                           (0x00000004): not-activated    - G_TLS_CERTIFICATE_NOT_ACTIVATED
                           (0x00000008): expired          - G_TLS_CERTIFICATE_EXPIRED
                           (0x00000010): revoked          - G_TLS_CERTIFICATE_REVOKED
                           (0x00000020): insecure         - G_TLS_CERTIFICATE_INSECURE
                           (0x00000040): generic-error    - G_TLS_CERTIFICATE_GENERIC_ERROR
                           (0x0000007f): validate-all     - G_TLS_CERTIFICATE_VALIDATE_ALL
  tls-database        : TLS database with anchor certificate authorities used to validate the server certificate
                        flags: readable, writable
                        Object of type "GTlsDatabase"
  tls-interaction     : A GTlsInteraction object to promt the user for password or certificate
                        flags: readable, writable
                        Object of type "GTlsInteraction"
  do-retransmission   : Ask the server to retransmit lost packets
                        flags: readable, writable
                        Boolean. Default: true
  ntp-time-source     : NTP time source for RTCP packets
                        flags: readable, writable
                        Enum "GstRTSPSrcNtpTimeSource" Default: 0, "ntp"
                           (0): ntp              - NTP time based on realtime clock
                           (1): unix             - UNIX time based on realtime clock
                           (2): running-time     - Running time based on pipeline clock
                           (3): clock-time       - Pipeline clock time
  user-agent          : The User-Agent string to send to the server
                        flags: readable, writable
                        String. Default: "GStreamer/1.14.5"
  max-rtcp-rtp-time-diff: Maximum amount of time in ms that the RTP time in RTCP SRs is allowed to be ahead (-1 disabled)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: 1000
  rfc7273-sync        : Synchronize received streams to the RFC7273 clock (requires clock and offset to be provided)
                        flags: readable, writable
                        Boolean. Default: false
  max-ts-offset-adjustment: The maximum number of nanoseconds per frame that time stamp offsets may be adjusted (0 = no limit).
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
  max-ts-offset       : The maximum absolute value of the time offset in (nanoseconds). Note, if the ntp-sync parameter is set the default value is changed to 0 (no limit)
                        flags: readable, writable
                        Integer64. Range: 0 - 9223372036854775807 Default: 3000000000
  default-rtsp-version: The RTSP version that should be tried first when negotiating version.
                        flags: readable, writable
                        Enum "GstRTSPVersion" Default: 16, "1-0"
                           (0): invalid          - GST_RTSP_VERSION_INVALID
                           (16): 1-0              - GST_RTSP_VERSION_1_0
                           (17): 1-1              - GST_RTSP_VERSION_1_1
                           (32): 2-0              - GST_RTSP_VERSION_2_0
  backchannel         : The type of backchannel to setup. Default is 'none'.
                        flags: readable, writable
                        Enum "GstRTSPBackchannel" Default: 0, "none"
                           (0): none             - No backchannel
                           (1): onvif            - ONVIF audio backchannel

Element Signals:
  "pad-added" :  void user_function (GstElement* object,
                                     GstPad* arg0,
                                     gpointer user_data);
  "pad-removed" :  void user_function (GstElement* object,
                                       GstPad* arg0,
                                       gpointer user_data);
  "no-more-pads" :  void user_function (GstElement* object,
                                        gpointer user_data);
  "handle-request" :  void user_function (GstElement* object,
                                          gpointer arg0,
                                          gpointer arg1,
                                          gpointer user_data);
  "on-sdp" :  void user_function (GstElement* object,
                                  GstSDPMessage* arg0,
                                  gpointer user_data);
  "select-stream" :  gboolean user_function (GstElement* object,
                                             guint arg0,
                                             GstCaps* arg1,
                                             gpointer user_data);
  "new-manager" :  void user_function (GstElement* object,
                                       GstElement* arg0,
                                       gpointer user_data);
  "request-rtcp-key" :  GstCaps * user_function (GstElement* object,
                                                 guint arg0,
                                                 gpointer user_data);
  "accept-certificate" :  gboolean user_function (GstElement* object,
                                                  GTlsConnection* arg0,
                                                  GTlsCertificate* arg1,
                                                  GTlsCertificateFlags arg2,
                                                  gpointer user_data);
  "before-send" :  gboolean user_function (GstElement* object,
                                           GstRTSPMessage* arg0,
                                           gpointer user_data);

Element Actions:
  "push-backchannel-buffer" :  GstFlowReturn user_function (GstElement* object,
                                                            guint arg0,
                                                            GstBuffer* arg1);

3.2 gst-launch-1.0

用于创建及执行一个Pipline，因此通常使用gst-launch先验证相关功能，然后再编写相应应用。
通过上面ogg视频播放的例子，我们已经看到，一个pipeline的多个element之间通过 “!" 分隔，同时可以设置element及Cap的属性。
下面是解析RTSP视频流的pipeline：

gst-launch-1.0 -v rtspsrc location=rtsp://10.201.0.158:8554/vlc ! rtph264depay !  h264parse ! omxh264dec ! nvvidconv ! video/x-raw, width=(int)2048, height=(int)1536, format=(string)BGRx ! videoconvert !  ximagesink sync=false

我们可以通过gst-inspect-1.0工具查看每个plugin的功能和属性，选择合适的插件来构成pipeline。
具体在python实现OpenCV+Gstreamer的方法是：OpenCV提供了cv2.VideoCapture()函数，只需把Gstreamer参数传给该函数即可。
具体代码如下：

def open_cam_rtsp(uri, width, height, latency):
    gst_str = ("rtspsrc location={} latency={} ! rtph264depay ! h264parse ! omxh264dec ! "
               "nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
               "videoconvert ! appsink").format(uri, latency, width, height)
    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
 
def open_cam_usb(dev, width, height):
    # We want to set width and height here, otherwise we could just do:
    #     return cv2.VideoCapture(dev)
    gst_str = ("v4l2src device=/dev/video{} ! "
               "video/x-raw, width=(int){}, height=(int){}, format=(string)RGB ! "
               "videoconvert ! appsink").format(dev, width, height)
    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
 
def open_cam_onboard(width, height):
    # On versions of L4T previous to L4T 28.1, flip-method=2
    # Use Jetson onboard camera
    gst_str = ("nvcamerasrc ! "
               "video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)30/1 ! "
               "nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
               "videoconvert ! appsink").format(width, height)
    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

4. 参考链接

https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html
https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html
https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html