H264 RTP与RTCP SR
NALU 头由一个字节组成, 它的语法如下:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
F: 1 个比特.
forbidden_zero_bit. 在 H.264 规范中规定了这一位必须为 0.
NRI: 2 个比特.
nal_ref_idc. 取 00 ~ 11, 似乎指示这个 NALU 的重要性, 如 00 的 NALU 解码器可以丢弃它而不影响图像的回放. 不过一般情况下不太关心
这个属性.
Type: 5 个比特.
nal_unit_type. 这个 NALU 单元的类型. 简述如下:
0 没有定义
1-23 NAL单元 单个 NAL 单元包.
24 STAP-A 单一时间的组合包
24 STAP-B 单一时间的组合包
26 MTAP16 多个时间的组合包
27 MTAP24 多个时间的组合包
28 FU-A 分片的单元
29 FU-B 分片的单元
30-31 没有定义
2. 打包模式
下面是 RFC 3550 中规定的 RTP 头的结构.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
负载类型 Payload type (PT): 7 bits
序列号 Sequence number (SN): 16 bits
时间戳 Timestamp: 32 bits
H.264 Payload 格式定义了三种不同的基本的负载(Payload)结构. 接收端可能通过 RTP Payload
的第一个字节来识别它们. 这一个字节类似 NALU 头的格式, 而这个头结构的 NAL 单元类型字段
则指出了代表的是哪一种结构,
这个字节的结构如下, 可以看出它和 H.264 的 NALU 头结构是一样的.
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
字段 Type: 这个 RTP payload 中 NAL 单元的类型. 这个字段和 H.264 中类型字段的区别是, 当 type
的值为 24 ~ 31 表示这是一个特别格式的 NAL 单元, 而 H.264 中, 只取 1~23 是有效的值.
24 STAP-A 单一时间的组合包
24 STAP-B 单一时间的组合包
26 MTAP16 多个时间的组合包
27 MTAP24 多个时间的组合包
28 FU-A 分片的单元
29 FU-B 分片的单元
30-31 没有定义
可能的结构类型分别有:
1. 单一 NAL 单元模式
即一个 RTP 包仅由一个完整的 NALU 组成. 这种情况下 RTP NAL 头类型字段和原始的 H.264的
NALU 头类型字段是一样的.
2. 组合封包模式
即可能是由多个 NAL 单元组成一个 RTP 包. 分别有4种组合方式: STAP-A, STAP-B, MTAP16, MTAP24.
那么这里的类型值分别是 24, 25, 26 以及 27.
3. 分片封包模式
用于把一个 NALU 单元封装成多个 RTP 包. 存在两种类型 FU-A 和 FU-B. 类型值分别是 28 和 29.
2.1 单一 NAL 单元模式
对于 NALU 的长度小于 MTU 大小的包, 一般采用单一 NAL 单元模式.
对于一个原始的 H.264 NALU 单元常由 [Start Code] [NALU Header] [NALU Payload] 三部分组成, 其中 Start Code 用于标示这是一个
NALU 单元的开始, 必须是 "00 00 00 01" 或 "00 00 01", NALU 头仅一个字节, 其后都是 NALU 单元内容.
打包时去除 "00 00 01" 或 "00 00 00 01" 的开始码, 把其他数据封包的 RTP 包即可.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| type | |
+-+-+-+-+-+-+-+-+ |
| |
| Bytes 2..n of a Single NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
如有一个 H.264 的 NALU 是这样的:
[00 00 00 01 67 42 A0 1E 23 56 0E 2F ... ]
这是一个序列参数集 NAL 单元. [00 00 00 01] 是四个字节的开始码, 67 是 NALU 头, 42 开始的数据是 NALU 内容.
封装成 RTP 包将如下:
[ RTP Header ] [ 67 42 A0 1E 23 56 0E 2F ]
即只要去掉 4 个字节的开始码就可以了.
2.2 组合封包模式
其次, 当 NALU 的长度特别小时, 可以把几个 NALU 单元封在一个 RTP 包中.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Data |
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 Data |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
2.3 Fragmentation Units (FUs).
而当 NALU 的长度超过 MTU 时, 就必须对 NALU 单元进行分片封包. 也称为 Fragmentation Units (FUs).
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 14. RTP payload format for FU-A
The FU indicator octet has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
The FU header has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|R| Type |
+---------------+
SR: Sender Report RTCP Packet
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
header |V=2|P| RC | PT=SR=200 | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC of sender |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
sender | NTP timestamp, most significant word |
info +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NTP timestamp, least significant word |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sender's packet count |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sender's octet count |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
report | SSRC_1 (SSRC of first source) |
block +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1 | fraction lost | cumulative number of packets lost |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| extended highest sequence number received |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| interarrival jitter |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| last SR (LSR) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| delay since last SR (DLSR) |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
report | SSRC_2 (SSRC of second source) |
block +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
2 : ... :
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| profile-specific extensions |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The sender report packet consists of three sections, possibly
followed by a fourth profile-specific extension section if defined.
The first section, the header, is 8 octets long. The fields have the
following meaning:
version (V): 2 bits
Identifies the version of RTP, which is the same in RTCP packets
as in RTP data packets. The version defined by this specification
is two (2).
padding (P): 1 bit
If the padding bit is set, this individual RTCP packet contains
some additional padding octets at the end which are not part of
the control information but are included in the length field. The
last octet of the padding is a count of how many padding octets
should be ignored, including itself (it will be a multiple of
four). Padding may be needed by some encryption algorithms with
fixed block sizes. In a compound RTCP packet, padding is only
required on one individual packet because the compound packet is
encrypted as a whole for the method in Section 9.1. Thus, padding
MUST only be added to the last individual packet, and if padding
is added to that packet, the padding bit MUST be set only on that
packet. This convention aids the header validity checks described
in Appendix A.2 and allows detection of packets from some early
implementations that incorrectly set the padding bit on the first
individual packet and add padding to the last individual packet.
reception report count (RC): 5 bits
The number of reception report blocks contained in this packet. A
value of zero is valid.
packet type (PT): 8 bits
Contains the constant 200 to identify this as an RTCP SR packet.
length: 16 bits
The length of this RTCP packet in 32-bit words minus one,
including the header and any padding. (The offset of one makes
zero a valid length and avoids a possible infinite loop in
scanning a compound RTCP packet, while counting 32-bit words
avoids a validity check for a multiple of 4.)
SSRC: 32 bits
The synchronization source identifier for the originator of this
SR packet.
The second section, the sender information, is 20 octets long and is
present in every sender report packet. It summarizes the data
transmissions from this sender. The fields have the following
meaning:
NTP timestamp: 64 bits
Indicates the wallclock time (see Section 4) when this report was
sent so that it may be used in combination with timestamps
returned in reception reports from other receivers to measure
round-trip propagation to those receivers. Receivers should
expect that the measurement accuracy of the timestamp may be
limited to far less than the resolution of the NTP timestamp. The
measurement uncertainty of the timestamp is not indicated as it
may not be known. On a system that has no notion of wallclock
time but does have some system-specific clock such as "system
uptime", a sender MAY use that clock as a reference to calculate
relative NTP timestamps. It is important to choose a commonly
used clock so that if separate implementations are used to produce
the individual streams of a multimedia session, all
implementations will use the same clock. Until the year 2036,
relative and absolute timestamps will differ in the high bit so
(invalid) comparisons will show a large difference; by then one
hopes relative timestamps will no longer be needed. A sender that
has no notion of wallclock or elapsed time MAY set the NTP
timestamp to zero.
RTP timestamp: 32 bits
Corresponds to the same time as the NTP timestamp (above), but in
the same units and with the same random offset as the RTP
timestamps in data packets. This correspondence may be used for
intra- and inter-media synchronization for sources whose NTP
timestamps are synchronized, and may be used by media-independent
receivers to estimate the nominal RTP clock frequency. Note that
in most cases this timestamp will not be equal to the RTP
timestamp in any adjacent data packet. Rather, it MUST be
calculated from the corresponding NTP timestamp using the
relationship between the RTP timestamp counter and real time as
maintained by periodically checking the wallclock time at a
sampling instant.
sender's packet count: 32 bits
The total number of RTP data packets transmitted by the sender
since starting transmission up until the time this SR packet was
generated. The count SHOULD be reset if the sender changes its
SSRC identifier.
sender's octet count: 32 bits
The total number of payload octets (i.e., not including header or
padding) transmitted in RTP data packets by the sender since
starting transmission up until the time this SR packet was
generated. The count SHOULD be reset if the sender changes its
SSRC identifier. This field can be used to estimate the average
payload data rate.
The third section contains zero or more reception report blocks
depending on the number of other sources heard by this sender since
the last report. Each reception report block conveys statistics on
the reception of RTP packets from a single synchronization source.
Receivers SHOULD NOT carry over statistics when a source changes its
SSRC identifier due to a collision. These statistics are:
SSRC_n (source identifier): 32 bits
The SSRC identifier of the source to which the information in this
reception report block pertains.
fraction lost: 8 bits
The fraction of RTP data packets from source SSRC_n lost since the
previous SR or RR packet was sent, expressed as a fixed point
number with the binary point at the left edge of the field. (That
is equivalent to taking the integer part after multiplying the
loss fraction by 256.) This fraction is defined to be the number
of packets lost divided by the number of packets expected, as
defined in the next paragraph. An implementation is shown in
Appendix A.3. If the loss is negative due to duplicates, the
fraction lost is set to zero. Note that a receiver cannot tell
whether any packets were lost after the last one received, and
that there will be no reception report block issued for a source
if all packets from that source sent during the last reporting
interval have been lost.
cumulative number of packets lost: 24 bits
The total number of RTP data packets from source SSRC_n that have
been lost since the beginning of reception. This number is
defined to be the number of packets expected less the number of
packets actually received, where the number of packets received
includes any which are late or duplicates. Thus, packets that
arrive late are not counted as lost, and the loss may be negative
if there are duplicates. The number of packets expected is
defined to be the extended last sequence number received, as
defined next, less the initial sequence number received. This may
be calculated as shown in Appendix A.3.
extended highest sequence number received: 32 bits
The low 16 bits contain the highest sequence number received in an
RTP data packet from source SSRC_n, and the most significant 16
bits extend that sequence number with the corresponding count of
sequence number cycles, which may be maintained according to the
algorithm in Appendix A.1. Note that different receivers within
the same session will generate different extensions to the
sequence number if their start times differ significantly.
interarrival jitter: 32 bits
An estimate of the statistical variance of the RTP data packet
interarrival time, measured in timestamp units and expressed as an
unsigned integer. The interarrival jitter J is defined to be the
mean deviation (smoothed absolute value) of the difference D in
packet spacing at the receiver compared to the sender for a pair
of packets. As shown in the equation below, this is equivalent to
the difference in the "relative transit time" for the two packets;
the relative transit time is the difference between a packet's RTP
timestamp and the receiver's clock at the time of arrival,
measured in the same units.
If Si is the RTP timestamp from packet i, and Ri is the time of
arrival in RTP timestamp units for packet i, then for two packets
i and j, D may be expressed as
D(i,j) = (Rj - Ri) - (Sj - Si) = (Rj - Sj) - (Ri - Si)
The interarrival jitter SHOULD be calculated continuously as each
data packet i is received from source SSRC_n, using this
difference D for that packet and the previous packet i-1 in order
of arrival (not necessarily in sequence), according to the formula
J(i) = J(i-1) + (|D(i-1,i)| - J(i-1))/16
Whenever a reception report is issued, the current value of J is
sampled.
The jitter calculation MUST conform to the formula specified here
in order to allow profile-independent monitors to make valid
interpretations of reports coming from different implementations.
This algorithm is the optimal first-order estimator and the gain
parameter 1/16 gives a good noise reduction ratio while
maintaining a reasonable rate of convergence [22]. A sample
implementation is shown in Appendix A.8. See Section 6.4.4 for a
discussion of the effects of varying packet duration and delay
before transmission.
last SR timestamp (LSR): 32 bits
The middle 32 bits out of 64 in the NTP timestamp (as explained in
Section 4) received as part of the most recent RTCP sender report
(SR) packet from source SSRC_n. If no SR has been received yet,
the field is set to zero.
delay since last SR (DLSR): 32 bits
The delay, expressed in units of 1/65536 seconds, between
receiving the last SR packet from source SSRC_n and sending this
reception report block. If no SR packet has been received yet
from SSRC_n, the DLSR field is set to zero.
Let SSRC_r denote the receiver issuing this receiver report.
Source SSRC_n can compute the round-trip propagation delay to
SSRC_r by recording the time A when this reception report block is
received. It calculates the total round-trip time A-LSR using the
last SR timestamp (LSR) field, and then subtracting this field to
leave the round-trip propagation delay as (A - LSR - DLSR). This
is illustrated in Fig. 2. Times are shown in both a hexadecimal
representation of the 32-bit fields and the equivalent floating-
point decimal representation. Colons indicate a 32-bit field
divided into a 16-bit integer part and 16-bit fraction part.
This may be used as an approximate measure of distance to cluster
receivers, although some links have very asymmetric delays.
[10 Nov 1995 11:33:25.125 UTC] [10 Nov 1995 11:33:36.5 UTC]
n SR(n) A=b710:8000 (46864.500 s)
---------------------------------------------------------------->
v ^
ntp_sec =0xb44db705 v ^ dlsr=0x0005:4000 ( 5.250s)
ntp_frac=0x20000000 v ^ lsr =0xb705:2000 (46853.125s)
(3024992005.125 s) v ^
r v ^ RR(n)
---------------------------------------------------------------->
|<-DLSR->|
(5.250 s)
A 0xb710:8000 (46864.500 s)
DLSR -0x0005:4000 ( 5.250 s)
LSR -0xb705:2000 (46853.125 s)
-------------------------------
delay 0x0006:2000 ( 6.125 s)
