【转载】live555学习 --H264数据处理（3）

本文转载自白杨 《live555学习（十五） --H264数据处理（3）》

    H264FUAFragmenter又对数据做了什么呢？

  1: void H264FUAFragmenter::doGetNextFrame() {    
  2:   if (fNumValidDataBytes == 1) {    
  3:       //读取一个新的frame     
  4:     // We have no NAL unit data currently in the buffer.  Read a new one:     
  5:     fInputSource->getNextFrame(&fInputBuffer[1], fInputBufferSize - 1,    
  6:                    afterGettingFrame, this,    
  7:                    FramedSource::handleClosure, this);    
  8:   } else {    
  9:       //     
 10:       //现在buffer中已经存在NALU数据了，需要考虑三种情况     
 11:       //1.一个新的NALU,且足够小能投递给RTP sink。     
 12:       //2.一个新的NALU,但是比RTP sink要求的包大了，投递第一个分片作为一个FU-A packet, 并带上一个额外的头字节。     
 13:       //3.部分NALU数据,投递下一个分片作为一个FU-A packet，并带上2个额外的头字节。     
 14:     // We have NAL unit data in the buffer.  There are three cases to consider:     
 15:     // 1. There is a new NAL unit in the buffer, and it's small enough to deliver     
 16:     //    to the RTP sink (as is).     
 17:     // 2. There is a new NAL unit in the buffer, but it's too large to deliver to     
 18:     //    the RTP sink in its entirety.  Deliver the first fragment of this data,     
 19:     //    as a FU-A packet, with one extra preceding header byte.     
 20:     // 3. There is a NAL unit in the buffer, and we've already delivered some     
 21:     //    fragment(s) of this.  Deliver the next fragment of this data,     
 22:     //    as a FU-A packet, with two extra preceding header bytes.     
 23:     
 24:     
 25:     if (fMaxSize < fMaxOutputPacketSize) { // shouldn't happen     
 26:       envir() << "H264FUAFragmenter::doGetNextFrame(): fMaxSize ("    
 27:           << fMaxSize << ") is smaller than expected\n";    
 28:     } else {    
 29:       fMaxSize = fMaxOutputPacketSize;    
 30:     }    
 31:     
 32:     
 33:     fLastFragmentCompletedNALUnit = True; // by default     
 34:     if (fCurDataOffset == 1) { // case 1 or 2     
 35:       if (fNumValidDataBytes - 1 <= fMaxSize) { // case 1     
 36:           //     
 37:           //情况1, 处理整个NALU     
 38:           //     
 39:     memmove(fTo, &fInputBuffer[1], fNumValidDataBytes - 1);    
 40:     fFrameSize = fNumValidDataBytes - 1;    
 41:     fCurDataOffset = fNumValidDataBytes;    
 42:       } else { // case 2     
 43:           //     
 44:           //情况2，处理NALU的第1个分片。注意，我们添加FU指示符和FU头字节(with S bit)到包的最前面(     
 45:           //重用已经存在的NAL 头字节作为FU的头字节)     
 46:           //     
 47:     // We need to send the NAL unit data as FU-A packets.  Deliver the first     
 48:     // packet now.  Note that we add FU indicator and FU header bytes to the front     
 49:     // of the packet (reusing the existing NAL header byte for the FU header).     
 50:     fInputBuffer[0] = (fInputBuffer[1] & 0xE0) | 28; // FU indicator     
 51:     fInputBuffer[1] = 0x80 | (fInputBuffer[1] & 0x1F); // FU header (with S bit)   重用NALU头字节     
 52:     memmove(fTo, fInputBuffer, fMaxSize);    
 53:     fFrameSize = fMaxSize;    
 54:     fCurDataOffset += fMaxSize - 1;    
 55:     fLastFragmentCompletedNALUnit = False;    
 56:       }    
 57:     } else { // case 3     
 58:         //     
 59:         //情况3，处理非第1个分片。需要添加FU指示符和FU头(我们重用了第一个分片中的字节，但是需要清除S位，     
 60:         //并在最后一个分片中添加E位)     
 61:         //     
 62:         //     
 63:       // We are sending this NAL unit data as FU-A packets.  We've already sent the     
 64:       // first packet (fragment).  Now, send the next fragment.  Note that we add     
 65:       // FU indicator and FU header bytes to the front.  (We reuse these bytes that     
 66:       // we already sent for the first fragment, but clear the S bit, and add the E     
 67:       // bit if this is the last fragment.)     
 68:  
 69:       fInputBuffer[fCurDataOffset-2] = fInputBuffer[0]; // FU indicator     
 70:       fInputBuffer[fCurDataOffset-1] = fInputBuffer[1]&~0x80; // FU header (no S bit)     
 71:       unsigned numBytesToSend = 2 + fNumValidDataBytes - fCurDataOffset;    
 72:       if (numBytesToSend > fMaxSize) {    
 73:     // We can't send all of the remaining data this time:     
 74:     numBytesToSend = fMaxSize;    
 75:     fLastFragmentCompletedNALUnit = False;    
 76:       } else {    
 77:     //     
 78:     //最后一个分片，需要在FU头中设置E标志位     
 79:     // This is the last fragment:     
 80:     fInputBuffer[fCurDataOffset-1] |= 0x40; // set the E bit in the FU header     
 81:     fNumTruncatedBytes = fSaveNumTruncatedBytes;    
 82:       }    
 83:       memmove(fTo, &fInputBuffer[fCurDataOffset-2], numBytesToSend);    
 84:       fFrameSize = numBytesToSend;    
 85:       fCurDataOffset += numBytesToSend - 2;    
 86:     }      
 87:     
 88:     if (fCurDataOffset >= fNumValidDataBytes) {    
 89:       // We're done with this data.  Reset the pointers for receiving new data:     
 90:       fNumValidDataBytes = fCurDataOffset = 1;    
 91:     }    
 92:     
 93:     // Complete delivery to the client:     
 94:     FramedSource::afterGetting(this);    
 95:   }    
 96: }   
 97: 

   当fNumValidDataBytes等于１时，表明buffer(fInputBuffer)中没有Nal Unit数据，那么就读入一个新的．从哪里读呢？还记得前面讲过的吗？H264FUAFragmenter在第一次读数据时代替了H264VideoStreamFramer，同时也与H264VideoStreamFramer还有ByteStreamFileSource手牵着脚，脚牵着手形成了链结构．文件数据从ByteStreamFileSource读入，经H264VideoStreamFramer处理传给H264FUAFragmenter．ByteStreamFileSource返回给H264VideoStreamFramer一段数据，H264VideoStreamFramer返回一个H264FUAFragmenter一个Nal unit ．
H264FUAFragmenter对Nal Unit做了什么呢？先看注释：
当我们有了nal unit，要处理３种情况：
１有一个完整的nal unit，并且它小到能够被打包进rtp包中．
２有一个完整的nal unit，但是它很大，那么就得为它分片传送了，把第一片打入一个FU-A包，此时利用了缓冲中前面的一个字节的头部．
３一个nal unit的已被发送了一部分，那么我们继续按FU-A包发送．此时利用了缓冲中前面的处理中已使用的两个字节的头部．
fNumValidDataBytes是H264FUAFragmenter缓冲fInputBuffer中有效数据的字节数．可以看到fNumValidDataBytes重置时被置为１，为什么不是０呢？因为fInputBuffer的第一个字节一直被留用作AU-A包的头部．如果是single nal打包，则从fInputBuffer的第二字节开始把nal unit复制到输出缓冲fTo，如果是FU-A包，则从fInputBuffer的第一字节开始复制．

结合本文最后面的H.264视频 RTP负载格式，可以很容易地把此段函数看明白。

   最后，来看下parse函数，parse是定义在MPEGVideoStreamParser中的纯虚函数，在子类H264VideoStreamParser中实现。parse主要是从文件的字节流中，分离出一个个的Frame,对于H264而言其实就是对一个个的NALU。*.264文件的格式非常简单，每个NALU以 0x00000001 作为起始符号(中间的NALU也可以以0x000001作为起始符)，顺序存放。

   
   
   
   

    
    
    
    
  1: unsigned H264VideoStreamParser::parse() { 
  2:   try { 
  3:     // 
  4:     //首先找到起始符号, 并跳过。文件流的最开始必需以0x00000001开始，但后续的NALU充许以0x000001(3 bytes)作为分隔 
  5:     // 
  6:     // The stream must start with a 0x00000001: 
  7:     if (!fHaveSeenFirstStartCode) { 
  8:       // Skip over any input bytes that precede the first 0x00000001: 
  9:       u_int32_t first4Bytes; 
 10:       while ((first4Bytes = test4Bytes()) != 0x00000001) { 
 11:     get1Byte(); setParseState(); // ensures that we progress over bad data 
 12:       } 
 13:       skipBytes(4); // skip this initial code 
 14:        
 15:       setParseState(); 
 16:       fHaveSeenFirstStartCode = True; // from now on 
 17:     } 
 18:     // 
 19:     //将起始标志也保存到目的缓冲区中 
 20:     // 
 21:     if (fOutputStartCodeSize > 0) { 
 22:       // Include a start code in the output: 
 23:       save4Bytes(0x00000001);    
 24:     } 
 25:      
 26:     // 
 27:     //保存所有数据，直至遇到起始标志，或者文件结束符。需要注意NALU中的第一个字节，因为它指示了NALU的类型 
 28:     // 
 29:     // Then save everything up until the next 0x00000001 (4 bytes) or 0x000001 (3 bytes), or we hit EOF. 
 30:     // Also make note of the first byte, because it contains the "nal_unit_type":  
 31:     u_int8_t firstByte; 
 32:     if (haveSeenEOF()) { 
 33:        // 
 34:        //已经设置了文件结束标志，将剩下的数据保存也来即可 
 35:        // 
 36:       // We hit EOF the last time that we tried to parse this data, 
 37:       // so we know that the remaining unparsed data forms a complete NAL unit: 
 38:       unsigned remainingDataSize = totNumValidBytes() - curOffset(); 
 39:       if (remainingDataSize == 0) (void)get1Byte(); // forces another read, which will cause EOF to get handled for real this time 
 40: #ifdef DEBUG 
 41:       fprintf(stderr, "This NAL unit (%d bytes) ends with EOF\n", remainingDataSize); 
 42: #endif 
 43:       if (remainingDataSize == 0) return 0; 
 44:       firstByte = get1Byte();   //将第一个字节保存下来，其指示了NALU的类型 
 45:       saveByte(firstByte); 
 46:        
 47:       while (--remainingDataSize > 0) { 
 48:     saveByte(get1Byte()); 
 49:       } 
 50:     } else { 
 51:       u_int32_t next4Bytes = test4Bytes(); 
 52:       firstByte = next4Bytes>>24;   //将第一个字节保存下来 
 53:       // 
 54:       //将下一个起始符号之前的数据都保存下来 
 55:       // 
 56:       while (next4Bytes != 0x00000001 && (next4Bytes&0xFFFFFF00) != 0x00000100) { 
 57:     // We save at least some of "next4Bytes". 
 58:     if ((unsigned)(next4Bytes&0xFF) > 1) {  //一次可以保存4个字节,并不需要一个一个字节对比，除非到了结尾 
 59:       // Common case: 0x00000001 or 0x000001 definitely doesn't begin anywhere in "next4Bytes", so we save all of it: 
 60:       save4Bytes(next4Bytes); 
 61:       skipBytes(4); 
 62:     } else { 
 63:       // Save the first byte, and continue testing the rest: 
 64:       saveByte(next4Bytes>>24); 
 65:       skipBytes(1); 
 66:     } 
 67:     next4Bytes = test4Bytes(); 
 68:       } 
 69:       // Assert: next4Bytes starts with 0x00000001 or 0x000001, and we've saved all previous bytes (forming a complete NAL unit). 
 70:       // Skip over these remaining bytes, up until the start of the next NAL unit: 
 71:       if (next4Bytes == 0x00000001) { 
 72:     skipBytes(4); 
 73:       } else { 
 74:     skipBytes(3); 
 75:       } 
 76:     } 
 77:   
 78:     u_int8_t nal_ref_idc = (firstByte&0x60)>>5; 
 79:     u_int8_t nal_unit_type = firstByte&0x1F; 
 80: #ifdef DEBUG 
 81:     fprintf(stderr, "Parsed %d-byte NAL-unit (nal_ref_idc: %d, nal_unit_type: %d (\"%s\"))\n", 
 82:         curFrameSize()-fOutputStartCodeSize, nal_ref_idc, nal_unit_type, nal_unit_type_description[nal_unit_type]); 
 83: #endif 
 84:  
 85:  
 86:     // 
 87:     //下面根据NALU的类型来作具体的分析 
 88:     // 
 89:     switch (nal_unit_type) { 
 90:       case 6: { // Supplemental enhancement information (SEI) 
 91:     analyze_sei_data(); 
 92:     // Later, perhaps adjust "fPresentationTime" if we saw a "pic_timing" SEI payload??? ##### 
 93:     break; 
 94:       } 
 95:       case 7: { // Sequence parameter set (序列参数集) 
 96:       // 
 97:       //保存一份SPS的副本到H264VideoStreamFramer中，后面的pps也需要保存，sps中可能还包含了帧率信息 
 98:       // 
 99:     // First, save a copy of this NAL unit, in case the downstream object wants to see it: 
100:     usingSource()->saveCopyOfSPS(fStartOfFrame + fOutputStartCodeSize, fTo - fStartOfFrame - fOutputStartCodeSize); 
101:  
102:  
103:     // Parse this NAL unit to check whether frame rate information is present: 
104:     unsigned num_units_in_tick, time_scale, fixed_frame_rate_flag; 
105:     analyze_seq_parameter_set_data(num_units_in_tick, time_scale, fixed_frame_rate_flag); 
106:     if (time_scale > 0 && num_units_in_tick > 0) { 
107:       usingSource()->fFrameRate = time_scale/(2.0*num_units_in_tick);   //sps中包含了帧率信息 
108: #ifdef DEBUG 
109:       fprintf(stderr, "Set frame rate to %f fps\n", usingSource()->fFrameRate); 
110:       if (fixed_frame_rate_flag == 0) { 
111:         fprintf(stderr, "\tWARNING: \"fixed_frame_rate_flag\" was not set\n"); 
112:       } 
113: #endif 
114:     } else {    //sps中不包含帧率信息，则使用source中设置的默认帧率 
115: #ifdef DEBUG 
116:       fprintf(stderr, "\tThis \"Sequence Parameter Set\" NAL unit contained no frame rate information, so we use a default frame rate of %f fps\n", usingSource()->fFrameRate); 
117: #endif 
118:     } 
119:     break; 
120:       } 
121:       case 8: { // Picture parameter set (图像参数集PPS) 
122:     // Save a copy of this NAL unit, in case the downstream object wants to see it: 
123:     usingSource()->saveCopyOfPPS(fStartOfFrame + fOutputStartCodeSize, fTo - fStartOfFrame - fOutputStartCodeSize); 
124:       } 
125:     } 
126:      
127:     usingSource()->setPresentationTime();   //设置当前的时间 
128: #ifdef DEBUG 
129:     unsigned long secs = (unsigned long)usingSource()->fPresentationTime.tv_sec; 
130:     unsigned uSecs = (unsigned)usingSource()->fPresentationTime.tv_usec; 
131:     fprintf(stderr, "\tPresentation time: %lu.%06u\n", secs, uSecs); 
132: #endif 
133:      
134:     // 
135:     //如果这个NALU是一个VCL NALU(即包含的是视频数据)，我们需要扫描下一个NALU的起始符， 
136:     //以判断这个NALU是否是当前的"access unit"(这个"access unit"应该可以理解为一帧图像帧吧)。 
137:     //我们需要根据这个信息去指明何时该增加"fPresentationTime"(RTP 打包时也需要根据这个信息，决定是否设置"M"位)。 
138:     // 
139:     // If this NAL unit is a VCL NAL unit, we also scan the start of the next NAL unit, to determine whether this NAL unit 
140:     // ends the current 'access unit'.  We need this information to figure out when to increment "fPresentationTime". 
141:     // (RTP streamers also need to know this in order to figure out whether or not to set the "M" bit.) 
142:     Boolean thisNALUnitEndsAccessUnit = False; // until we learn otherwise  
143:     if (haveSeenEOF()) { 
144:       // There is no next NAL unit, so we assume that this one ends the current 'access unit': 
145:       thisNALUnitEndsAccessUnit = True; 
146:     } else { 
147:       Boolean const isVCL = nal_unit_type <= 5 && nal_unit_type > 0; // Would need to include type 20 for SVC and MVC ##### 
148:       if (isVCL) { 
149:     u_int32_t first4BytesOfNextNALUnit = test4Bytes(); 
150:     u_int8_t firstByteOfNextNALUnit = first4BytesOfNextNALUnit>>24; 
151:     u_int8_t next_nal_ref_idc = (firstByteOfNextNALUnit&0x60)>>5; 
152:     u_int8_t next_nal_unit_type = firstByteOfNextNALUnit&0x1F; 
153:     if (next_nal_unit_type >= 6) { 
154:         //下一个NALU不是VCL的，当前的"access unit"结束了 
155:       // The next NAL unit is not a VCL; therefore, we assume that this NAL unit ends the current 'access unit': 
156: #ifdef DEBUG 
157:       fprintf(stderr, "\t(The next NAL unit is not a VCL)\n"); 
158: #endif 
159:       thisNALUnitEndsAccessUnit = True; 
160:     } else { 
161:         //下一个NALU也是VCL的，还需要检查一下它们是不是属于同一个"access unit" 
162:       // The next NAL unit is also a VLC.  We need to examine it a little to figure out if it's a different 'access unit'. 
163:       // (We use many of the criteria described in section 7.4.1.2.4 of the H.264 specification.) 
164:       Boolean IdrPicFlag = nal_unit_type == 5; 
165:       Boolean next_IdrPicFlag = next_nal_unit_type == 5; 
166:       if (next_IdrPicFlag != IdrPicFlag) { 
167:         // IdrPicFlag differs in value 
168: #ifdef DEBUG 
169:         fprintf(stderr, "\t(IdrPicFlag differs in value)\n"); 
170: #endif 
171:         thisNALUnitEndsAccessUnit = True; 
172:       } else if (next_nal_ref_idc != nal_ref_idc && next_nal_ref_idc*nal_ref_idc == 0) { 
173:         // nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0 
174: #ifdef DEBUG 
175:         fprintf(stderr, "\t(nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0)\n"); 
176: #endif 
177:         thisNALUnitEndsAccessUnit = True; 
178:       } else if ((nal_unit_type == 1 || nal_unit_type == 2 || nal_unit_type == 5) 
179:              && (next_nal_unit_type == 1 || next_nal_unit_type == 2 || next_nal_unit_type == 5)) { 
180:         // Both this and the next NAL units begin with a "slice_header". 
181:         // Parse this (for each), to get parameters that we can compare: 
182:          
183:         // Current NAL unit's "slice_header": 
184:         unsigned frame_num, pic_parameter_set_id, idr_pic_id; 
185:         Boolean field_pic_flag, bottom_field_flag; 
186:         analyze_slice_header(fStartOfFrame + fOutputStartCodeSize, fTo, nal_unit_type, 
187:                  frame_num, pic_parameter_set_id, idr_pic_id, field_pic_flag, bottom_field_flag); 
188:          
189:         // Next NAL unit's "slice_header": 
190: #ifdef DEBUG 
191:         fprintf(stderr, "    Next NAL unit's slice_header:\n"); 
192: #endif 
193:         u_int8_t next_slice_header[NUM_NEXT_SLICE_HEADER_BYTES_TO_ANALYZE]; 
194:         testBytes(next_slice_header, sizeof next_slice_header); 
195:         unsigned next_frame_num, next_pic_parameter_set_id, next_idr_pic_id; 
196:         Boolean next_field_pic_flag, next_bottom_field_flag; 
197:         analyze_slice_header(next_slice_header, &next_slice_header[sizeof next_slice_header], next_nal_unit_type, 
198:                  next_frame_num, next_pic_parameter_set_id, next_idr_pic_id, next_field_pic_flag, next_bottom_field_flag); 
199:          
200:         if (next_frame_num != frame_num) { 
201:           // frame_num differs in value 
202: #ifdef DEBUG 
203:           fprintf(stderr, "\t(frame_num differs in value)\n"); 
204: #endif 
205:           thisNALUnitEndsAccessUnit = True; 
206:         } else if (next_pic_parameter_set_id != pic_parameter_set_id) { 
207:           // pic_parameter_set_id differs in value 
208: #ifdef DEBUG 
209:           fprintf(stderr, "\t(pic_parameter_set_id differs in value)\n"); 
210: #endif 
211:           thisNALUnitEndsAccessUnit = True; 
212:         } else if (next_field_pic_flag != field_pic_flag) { 
213:           // field_pic_flag differs in value 
214: #ifdef DEBUG 
215:           fprintf(stderr, "\t(field_pic_flag differs in value)\n"); 
216: #endif 
217:           thisNALUnitEndsAccessUnit = True; 
218:         } else if (next_bottom_field_flag != bottom_field_flag) { 
219:           // bottom_field_flag differs in value 
220: #ifdef DEBUG 
221:           fprintf(stderr, "\t(bottom_field_flag differs in value)\n"); 
222: #endif 
223:           thisNALUnitEndsAccessUnit = True; 
224:         } else if (next_IdrPicFlag == 1 && next_idr_pic_id != idr_pic_id) { 
225:           // IdrPicFlag is equal to 1 for both and idr_pic_id differs in value 
226:           // Note: We already know that IdrPicFlag is the same for both. 
227: #ifdef DEBUG 
228:           fprintf(stderr, "\t(IdrPicFlag is equal to 1 for both and idr_pic_id differs in value)\n"); 
229: #endif 
230:           thisNALUnitEndsAccessUnit = True; 
231:         } 
232:       } 
233:     } 
234:       } 
235:     } 
236:      
237:     if (thisNALUnitEndsAccessUnit) { 
238: #ifdef DEBUG 
239:       fprintf(stderr, "*****This NAL unit ends the current access unit*****\n"); 
240: #endif 
241:       usingSource()->fPictureEndMarker = True;  //这里就是设置RTP打包时用到的M标志了 
242:       ++usingSource()->fPictureCount; 
243:  
244:  
245:       // 
246:       //下一个NALU不再属于当前"access　unit""时，才改变时间 
247:       // 
248:       // Note that the presentation time for the next NAL unit will be different: 
249:       struct timeval& nextPT = usingSource()->fNextPresentationTime; // alias　这里是引用　 
250:       nextPT = usingSource()->fPresentationTime; 
251:       double nextFraction = nextPT.tv_usec/1000000.0 + 1/usingSource()->fFrameRate;  
252:       unsigned nextSecsIncrement = (long)nextFraction; 
253:       nextPT.tv_sec += (long)nextSecsIncrement; 
254:       nextPT.tv_usec = (long)((nextFraction - nextSecsIncrement)*1000000); 
255:     } 
256:     setParseState(); 
257:  
258:  
259:     return curFrameSize(); 
260:   } catch (int /*e*/) { 
261: #ifdef DEBUG 
262:     fprintf(stderr, "H264VideoStreamParser::parse() EXCEPTION (This is normal behavior - *not* an error)\n"); 
263: #endif 
264:     return 0;  // the parsing got interrupted 
265:   } 
266: }
   
   
   
   

        H264VideoStreamParser::parse()函数除了取出Frame，还对NALU中的部分参数做了解释工作。对于PPS或者SPS类型的NALU，要保存到H264VideoStreamFramer中。"access unit",在这里可以理解为一副图像,一个"access unit"可以包含多个NALU，很显示这些NALU的时间戳应该是相同的。实际上，很多时候一个"access unit"单元只包含一个NALU，这里简单多了。

H.264 视频 RTP 负载格式

1. 网络抽象层单元类型 (NALU)

NALU 头由一个字节组成, 它的语法如下:

      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |F|NRI|  Type   |
      +---------------+

F: 1 个比特.
  forbidden_zero_bit. 在 H.264 规范中规定了这一位必须为 0.

NRI: 2 个比特.
  nal_ref_idc. 取 00 ~ 11, 似乎指示这个 NALU 的重要性, 如 00 的 NALU 解码器可以丢弃它而不影响图像的回放. 不过一般情况下不太关心

这个属性.

Type: 5 个比特.
  nal_unit_type. 这个 NALU 单元的类型. 简述如下:

  0     没有定义
  1-23  NAL单元  单个 NAL 单元包.
  24    STAP-A   单一时间的组合包
  25    STAP-B   单一时间的组合包
  26    MTAP16   多个时间的组合包
  27    MTAP24   多个时间的组合包
  28    FU-A     分片的单元
  29    FU-B     分片的单元
  30-31 没有定义

2. 打包模式

  下面是 RFC 3550 中规定的 RTP 头的结构.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |V=2|P|X|  CC   |M|     PT      |       sequence number         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                           timestamp                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           synchronization source (SSRC) identifier            |
      +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
      |            contributing source (CSRC) identifiers             |
      |                             ....                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  负载类型 Payload type (PT): 7 bits
  序列号 Sequence number (SN): 16 bits
  时间戳 Timestamp: 32 bits
  
  H.264 Payload 格式定义了三种不同的基本的负载(Payload)结构. 接收端可能通过 RTP Payload 
  的第一个字节来识别它们. 这一个字节类似 NALU 头的格式, 而这个头结构的 NAL 单元类型字段
  则指出了代表的是哪一种结构,

  这个字节的结构如下, 可以看出它和 H.264 的 NALU 头结构是一样的.
      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |F|NRI|  Type   |
      +---------------+
  字段 Type: 这个 RTP payload 中 NAL 单元的类型. 这个字段和 H.264 中类型字段的区别是, 当 type
  的值为 24 ~ 31 表示这是一个特别格式的 NAL 单元, 而 H.264 中, 只取 1~23 是有效的值.
   
  24    STAP-A   单一时间的组合包
  25    STAP-B   单一时间的组合包
  26    MTAP16   多个时间的组合包
  27    MTAP24   多个时间的组合包
  28    FU-A     分片的单元
  29    FU-B     分片的单元
  30-31 没有定义

  可能的结构类型分别有:

  1. 单一 NAL 单元模式
     即一个 RTP 包仅由一个完整的 NALU 组成. 这种情况下 RTP NAL 头类型字段和原始的 H.264的
  NALU 头类型字段是一样的.

  2. 组合封包模式
    即可能是由多个 NAL 单元组成一个 RTP 包. 分别有4种组合方式: STAP-A, STAP-B, MTAP16, MTAP24.
  那么这里的类型值分别是 24, 25, 26 以及 27.

  3. 分片封包模式
    用于把一个 NALU 单元封装成多个 RTP 包. 存在两种类型 FU-A 和 FU-B. 类型值分别是 28 和 29.

2.1 单一 NAL 单元模式

  对于 NALU 的长度小于 MTU 大小的包, 一般采用单一 NAL 单元模式.
  对于一个原始的 H.264 NALU 单元常由 [Start Code] [NALU Header] [NALU Payload] 三部分组成, 其中 Start Code 用于标示这是一个

NALU 单元的开始, 必须是 "00 00 00 01" 或 "00 00 01", NALU 头仅一个字节, 其后都是 NALU 单元内容.
  打包时去除 "00 00 01" 或 "00 00 00 01" 的开始码, 把其他数据封包的 RTP 包即可.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |F|NRI|  type   |                                               |
      +-+-+-+-+-+-+-+-+                                               |
      |                                                               |
      |               Bytes 2..n of a Single NAL unit                 |
      |                                                               |
      |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               :...OPTIONAL RTP padding        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  如有一个 H.264 的 NALU 是这样的:

  [00 00 00 01 67 42 A0 1E 23 56 0E 2F ... ]

  这是一个序列参数集 NAL 单元. [00 00 00 01] 是四个字节的开始码, 67 是 NALU 头, 42 开始的数据是 NALU 内容.

  封装成 RTP 包将如下:

  [ RTP Header ] [ 67 42 A0 1E 23 56 0E 2F ]

  即只要去掉 4 个字节的开始码就可以了.

2.2 组合封包模式

  其次, 当 NALU 的长度特别小时, 可以把几个 NALU 单元封在一个 RTP 包中.

  
       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          RTP Header                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |STAP-A NAL HDR |         NALU 1 Size           | NALU 1 HDR    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         NALU 1 Data                           |
      :                                                               :
      +               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |               | NALU 2 Size                   | NALU 2 HDR    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         NALU 2 Data                           |
      :                                                               :
      |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               :...OPTIONAL RTP padding        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.3 Fragmentation Units (FUs).

  而当 NALU 的长度超过 MTU 时, 就必须对 NALU 单元进行分片封包. 也称为 Fragmentation Units (FUs).
  
       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | FU indicator  |   FU header   |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
      |                                                               |
      |                         FU payload                            |
      |                                                               |
      |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               :...OPTIONAL RTP padding        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Figure 14.  RTP payload format for FU-A

   The FU indicator octet has the following format:

      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |F|NRI|  Type   |
      +---------------+

   The FU header has the following format:

      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |S|E|R|  Type   |
      +---------------+

－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

H264FUAFragmenter中只支持single和FU-A模式，不支持其它模式．

        我们现在还得出一个结论，我们可以看出RTPSink与Source怎样分工：RTPSink只做形成通用RTP包头的工作，各种媒体格式的Source才是实现媒体数据RTP封包的地方，其实按习惯感觉XXXRTPSink才是进行封包的地方．但是，从文件的安排上，H264FUAFragmenter被隐藏在H264VideoRTPSink中，并在程序中暗渡陈仓地把H264VideoStreamFramer替换掉，其实还是按习惯的架构（设计模式）来做的，所以如果把H264FUAFragmenter的工作移到H264VideoRTPSink中也是没问题的．

http://blog.csdn.net/nkmnkm/article/details/6947309

http://blog.csdn.net/gavinr/article/details/7042228