webrtc 代码学习（十九）Audio Nack Track 重传机制

Audio Nack Track 重传机制
作者：LanPZzzz

1. Nack Track 在 modules\audio_coding\neteq\nack_tracker.cc 中

是NetEq 的成员变量 modules\audio_coding\neteq\neteq_impl.cc 484

void NetEqImpl::EnableNack(size_t max_nack_list_size) {
  rtc::CritScope lock(&crit_sect_);
  if (!nack_enabled_) {
    const int kNackThresholdPackets = 2;  创建时这个值=2，是表示有2个seq间隔是就要重传了，很重要
    nack_.reset(NackTracker::Create(kNackThresholdPackets));
    nack_enabled_ = true;
    nack_->UpdateSampleRate(fs_hz_);
  }
  nack_->SetMaxNackListSize(max_nack_list_size);
}

2. insert packet 时候，会调用UpdateLastReceivedPacket 方法，用于判断当前包是否是连续的，有没有需要重传的包，什么时候insert packet，请查看rtp 包的接收

void NackTracker::UpdateLastReceivedPacket(uint16_t sequence_number,
                                           uint32_t timestamp) {
  // Just record the value of sequence number and timestamp if this is the
  // first packet.
  如果是第一个包，就只是保存
  if (!any_rtp_received_) {
    sequence_num_last_received_rtp_ = sequence_number;
    timestamp_last_received_rtp_ = timestamp;
    any_rtp_received_ = true;
    // If no packet is decoded, to have a reasonable estimate of time-to-play
    // use the given values.
    if (!any_rtp_decoded_) {
      sequence_num_last_decoded_rtp_ = sequence_number;
      timestamp_last_decoded_rtp_ = timestamp;
    }
    return;
  }

  如果是上一个包，就不保存了
  if (sequence_number == sequence_num_last_received_rtp_)
    return;

  从nack 的list 中，重传的包到了之后，从nack 的list 中去掉
  // Received RTP should not be in the list.
  nack_list_.erase(sequence_number);

  和上一个包进行对比，如果是false，走下边
  // If this is an old sequence number, no more action is required, return.
  if (IsNewerSequenceNumber(sequence_num_last_received_rtp_, sequence_number))
    return;

  计算samples_per_packet_ 每个包的平均增长数
  UpdateSamplesPerPacket(sequence_number, timestamp);
  
  更新list，很重要
  UpdateList(sequence_number);
  
  保存这次的信息
  sequence_num_last_received_rtp_ = sequence_number;
  timestamp_last_received_rtp_ = timestamp;
  
  根据limit 删除不要nack 的包，及丢帧了
  LimitNackListSize();
}

3. IsNewerSequenceNumber

t1 = last
t2 = current

if t1 > t2 && |t1-t2| == kBreakPoint;
if t1 = max && t2 = 0 return true
if t1 = 0 && t2 = max return false
else
if t1 - t2 < kBreakPoint return true
if t1 - t2 > kBreakPoint return false
总结：t1 < t2 && t1 - t2 < kBreakpoint return false，这个时候t2 是 t1 的下一个包

4. 更新list ，这个很重要

void NackTracker::UpdateList(uint16_t sequence_number_current_received_rtp) {
  // Some of the packets which were considered late, now are considered missing.
  ChangeFromLateToMissing(sequence_number_current_received_rtp);

  if (IsNewerSequenceNumber(sequence_number_current_received_rtp,
                            sequence_num_last_received_rtp_ + 1))
    添加到list 中
    AddToList(sequence_number_current_received_rtp);
}

5. ChangeFromLateToMissing

void NackTracker::ChangeFromLateToMissing(
    uint16_t sequence_number_current_received_rtp) {
  根据2分法，如果有 nack_threshold_packets_ = 2个间隔
  NackList::const_iterator lower_bound =
      nack_list_.lower_bound(static_cast(
          sequence_number_current_received_rtp - nack_threshold_packets_));

  前面的is_missing 就设置成true
  for (NackList::iterator it = nack_list_.begin(); it != lower_bound; ++it)
    it->second.is_missing = true;
}

6. AddToList

void NackTracker::AddToList(uint16_t sequence_number_current_received_rtp) {
  assert(!any_rtp_decoded_ ||
         IsNewerSequenceNumber(sequence_number_current_received_rtp,
                               sequence_num_last_decoded_rtp_));

  // Packets with sequence numbers older than |upper_bound_missing| are
  // considered missing, and the rest are considered late.
  uint16_t upper_bound_missing =
      sequence_number_current_received_rtp - nack_threshold_packets_;

  for (uint16_t n = sequence_num_last_received_rtp_ + 1;
       IsNewerSequenceNumber(sequence_number_current_received_rtp, n); ++n) {
    bool is_missing = IsNewerSequenceNumber(upper_bound_missing, n);
    
    这里计算的时间是什么？？？？？？？？
    uint32_t timestamp = EstimateTimestamp(n);
    NackElement nack_element(TimeToPlay(timestamp), timestamp, is_missing);
    nack_list_.insert(nack_list_.end(), std::make_pair(n, nack_element));
  }
}

7. 丢弃掉已经不能重发的帧

void NackTracker::LimitNackListSize() {
  这里丢弃小于 limit 的数据
  uint16_t limit = sequence_num_last_received_rtp_ -
                   static_cast(max_nack_list_size_) - 1;
  nack_list_.erase(nack_list_.begin(), nack_list_.upper_bound(limit));
}

8. decoder 获取包的时候，to be continue… =======

void NackTracker::UpdateLastDecodedPacket(uint16_t sequence_number,
                                          uint32_t timestamp) {
  if (IsNewerSequenceNumber(sequence_number, sequence_num_last_decoded_rtp_) ||
      !any_rtp_decoded_) {
    sequence_num_last_decoded_rtp_ = sequence_number;
    timestamp_last_decoded_rtp_ = timestamp;
    // Packets in the list with sequence numbers less than the
    // sequence number of the decoded RTP should be removed from the lists.
    // They will be discarded by the jitter buffer if they arrive.
    nack_list_.erase(nack_list_.begin(),
                     nack_list_.upper_bound(sequence_num_last_decoded_rtp_));

    // Update estimated time-to-play.
    for (NackList::iterator it = nack_list_.begin(); it != nack_list_.end();
         ++it)
      it->second.time_to_play_ms = TimeToPlay(it->second.estimated_timestamp);
  } else {
    assert(sequence_number == sequence_num_last_decoded_rtp_);

    // Same sequence number as before. 10 ms is elapsed, update estimations for
    // time-to-play.
    UpdateEstimatedPlayoutTimeBy10ms();

    // Update timestamp for better estimate of time-to-play, for packets which
    // are added to NACK list later on.
    timestamp_last_decoded_rtp_ += sample_rate_khz_ * 10;
  }
  any_rtp_decoded_ = true;
}

9. 获取需要重发的数据，is_missing = true && time_to_play_ms > round_trip_time_ms（包的播放时间要比当前需要的时间要大）

std::vector NackTracker::GetNackList(
    int64_t round_trip_time_ms) const {
  RTC_DCHECK_GE(round_trip_time_ms, 0);
  std::vector sequence_numbers;
  for (NackList::const_iterator it = nack_list_.begin(); it != nack_list_.end();
       ++it) {
    if (it->second.is_missing &&
        it->second.time_to_play_ms > round_trip_time_ms)
      sequence_numbers.push_back(it->first);
  }
  return sequence_numbers;
}

10. 在每次insert packet的时候，都要GetNackList，然后如果有，就调用SendNACK

modules\rtp_rtcp\source\rtp_rtcp_impl.cc 682

// Send a Negative acknowledgment packet.
int32_t ModuleRtpRtcpImpl::SendNACK(const uint16_t* nack_list,
                                    const uint16_t size) {
  保存记录
  for (int i = 0; i < size; ++i) {
    receive_loss_stats_.AddLostPacket(nack_list[i]);
  }

  uint16_t nack_length = size;
  uint16_t start_id = 0;
  int64_t now = clock_->TimeInMilliseconds();
  if (TimeToSendFullNackList(now)) {
    nack_last_time_sent_full_ = now;
    nack_last_time_sent_full_prev_ = now;
  } else {
    如果是已经发送过的Seq，就直接返回
    // Only send extended list.
    if (nack_last_seq_number_sent_ == nack_list[size - 1]) {
      // Last sequence number is the same, do not send list.
      return 0;
    }

     获取新的Seq list 位置，这里只发送新的Seq，已经发送过的，就不在发送了
    // Send new sequence numbers.
    for (int i = 0; i < size; ++i) {
      if (nack_last_seq_number_sent_ == nack_list[i]) {
        start_id = i + 1;
        break;
      }
    }
    nack_length = size - start_id;
  }

  最大重传数目
  // Our RTCP NACK implementation is limited to kRtcpMaxNackFields sequence
  // numbers per RTCP packet.
  if (nack_length > kRtcpMaxNackFields) {
    nack_length = kRtcpMaxNackFields;
  }
  
  记录本次发送的最大重传Seq
  nack_last_seq_number_sen_ = nack_list[start_id + nack_length - 1];
  
  重传
  return rtcp_sender_.SendRTCP(GetFeedbackState(), kRtcpNack, nack_length,
                               &nack_list[start_id]);
}

11. nack 默认是不打开的？？？

nack 在Video 的sdp 上的是打开的，audio 没有打开，所有想使用audio 的nack 需要在sdp 中打开nack 属性。
如下，就是video 的nack 属性

   a=rtpmap:96 VP8/90000\r\n
   a=rtcp-fb:96 goog-remb\r\n
   a=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\n
   a=rtcp-fb:96 nack\r\n
   a=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\n

a=rtcp-fb:111 transport-cc
以上这行说明opus编码支持使用rtcp来控制拥塞，参考https://tools.ietf.org/html/draft-holmer-rmcat-transport-wide-cc-extensions-01a=rtcp-fb
反馈消息，接收方向发送方发送的反馈消息

a=rtcp-fb:98 nack
支持丢包重传，参考rfc4585
a=rtcp-fb:98 nack pli
支持关键帧丢包重传,参考rfc4585

12.audio nack 在sdp 中打开，具体怎么做，需要查看下！！！！

13. 流程？

构造WebRtcAudioSendStream （media\engine\webrtcvoiceengine.cc 746）
-> stream_ = call_->CreateAudioSendStream(config_);
-> AudioSendStream::ConfigureStream （audio\audio_send_stream.cc 195）
-> new_config.rtp.nack.rtp_history_ms 设置这个属性就可以了打开nack
->
->
->
->

WebRtcVoiceMediaChannel::AddRecvStream（media\engine\webrtcvoiceengine.cc 1861）

AudioReceiveStream::ConfigureStream （audio\audio_receive_stream.cc 340）

14. ReceiveNack，接收到Nack消息，从pace 中发送

RTPSender::OnReceivedNack （modules\rtp_rtcp\source\rtp_sender.cc 715）
->… -> PacedSender::InsertPacket （modules\pacing\paced_sender.cc 185）
frame 到 modules\pacing\round_robin_packet_queue.cc 中
PacedSender 中PacedSender::Process() （modules\pacing\paced_sender.cc 262），在线程中进行SendRtp 的data