流媒体学习之路(mediasoup)——simulcast 与 svc 简介（7）
流媒体学习之路(mediasoup)——simulcast 与 svc 简介（7）
——
我正在的github给大家开发一个用于做实验的项目 —— github.com/qw225967/Bifrost
目标：可以让大家熟悉各类Qos能力、带宽估计能力，提供每个环节关键参数调节接口并实现一个json全配置，提供全面的可视化算法观察能力。
欢迎大家使用
——
文章目录
- 流媒体学习之路(mediasoup)——simulcast 与 svc 简介（7）
- 一、Simulcast
-
- 1.1 simulcast简介
- 1.2 mediasoup中的Simulcast
- 1.3 Simulcast的缺陷
- 二、SVC
-
- 2.1 SVC简介
- 2.2 mediasoup中的SVC
- 三、结语
前面的内容解析了拥塞控制模块，它的作用只是将网络的状况进行及时地反馈以及预测，但是如何在弱网环境下进行可靠传输并没有详细的介绍。当网络传输出现问题（例如：带宽下降、丢包上涨、延迟增加等），那么如何让用户在弱网条件下拥有更好的体验呢？答案是我们降低发送的码率，牺牲观看的清晰度并维持流畅的播放是可行的。下面我们将介绍mediasoup中两种从客户端入手进行码率调整的方案：simulcast 和 SVC（Scalable Video Coding），相关的文档可以浏览mediasoup的官方介绍文档。
一、Simulcast
1.1 simulcast简介
simulcast直译过来是多播的意思，其实就是一个客户端发送多条不同码率大小的流，为了达到这样的效果，客户端会对同一帧画面进行不同分辨率的编码来达到降低码率的目的。
例如上图，推流客户端同时推两路大小不一的流随后在服务器对下行网络进行估计，网络正常时下发720P分辨率的流，网络发生拥塞时下发360P分辨率的流，就能实现码率下降来抵抗弱网。
1.2 mediasoup中的Simulcast
simulcast模块集成在 SimulcastConsumer 类中——SimulcastConsumer.hpp 和 SimulcastConsumer.cpp。下面给出该类的头文件：
class SimulcastConsumer : public RTC::Consumer, public RTC::RtpStreamSend::Listener
{
public:
SimulcastConsumer(
const std::string& id,
const std::string& producerId,
RTC::Consumer::Listener* listener,
json& data);
~SimulcastConsumer() override;
public:
void FillJson(json& jsonObject) const override;
void FillJsonStats(json& jsonArray) const override;
void FillJsonScore(json& jsonObject) const override;
void HandleRequest(Channel::Request* request) override;
RTC::Consumer::Layers GetPreferredLayers() const override
{
RTC::Consumer::Layers layers;
layers.spatial = this->preferredSpatialLayer;
layers.temporal = this->preferredTemporalLayer;
return layers;
}
bool IsActive() const override
{
// clang-format off
return (
RTC::Consumer::IsActive() &&
std::any_of(
this->producerRtpStreams.begin(),
this->producerRtpStreams.end(),
[](const RTC::RtpStream* rtpStream)
{
return (rtpStream != nullptr && rtpStream->GetScore() > 0u);
}
)
);
// clang-format on
}
void ProducerRtpStream(RTC::RtpStream* rtpStream, uint32_t mappedSsrc) override;
void ProducerNewRtpStream(RTC::RtpStream* rtpStream, uint32_t mappedSsrc) override;
void ProducerRtpStreamScore(RTC::RtpStream* rtpStream, uint8_t score, uint8_t previousScore) override;
void ProducerRtcpSenderReport(RTC::RtpStream* rtpStream, bool first) override;
uint8_t GetBitratePriority() const override;
uint32_t IncreaseLayer(uint32_t bitrate, bool considerLoss) override;
void ApplyLayers() override;
uint32_t GetDesiredBitrate() const override;
void SendRtpPacket(RTC::RtpPacket* packet) override;
void GetRtcp(RTC::RTCP::CompoundPacket* packet, RTC::RtpStreamSend* rtpStream, uint64_t nowMs) override;
std::vector<RTC::RtpStreamSend*> GetRtpStreams() override
{
return this->rtpStreams;
}
void NeedWorstRemoteFractionLost(uint32_t mappedSsrc, uint8_t& worstRemoteFractionLost) override;
void ReceiveNack(RTC::RTCP::FeedbackRtpNackPacket* nackPacket) override;
void ReceiveKeyFrameRequest(RTC::RTCP::FeedbackPs::MessageType messageType, uint32_t ssrc) override;
void ReceiveRtcpReceiverReport(RTC::RTCP::ReceiverReport* report) override;
uint32_t GetTransmissionRate(uint64_t nowMs) override;
float GetRtt() const override;
private:
void UserOnTransportConnected() override;
void UserOnTransportDisconnected() override;
void UserOnPaused() override;
void UserOnResumed() override;
void CreateRtpStream();
void RequestKeyFrames();
void RequestKeyFrameForTargetSpatialLayer();
void RequestKeyFrameForCurrentSpatialLayer();
void MayChangeLayers(bool force = false);
bool RecalculateTargetLayers(int16_t& newTargetSpatialLayer, int16_t& newTargetTemporalLayer) const;
void UpdateTargetLayers(int16_t newTargetSpatialLayer, int16_t newTargetTemporalLayer);
bool CanSwitchToSpatialLayer(int16_t spatialLayer) const;
void EmitScore() const;
void EmitLayersChange() const;
RTC::RtpStream* GetProducerCurrentRtpStream() const;
RTC::RtpStream* GetProducerTargetRtpStream() const;
RTC::RtpStream* GetProducerTsReferenceRtpStream() const;
/* Pure virtual methods inherited from RtpStreamSend::Listener. */
public:
void OnRtpStreamScore(RTC::RtpStream* rtpStream, uint8_t score, uint8_t previousScore) override;
void OnRtpStreamRetransmitRtpPacket(RTC::RtpStreamSend* rtpStream, RTC::RtpPacket* packet) override;
private:
// Allocated by this.
RTC::RtpStreamSend* rtpStream{ nullptr };
// Others.
std::unordered_map<uint32_t, int16_t> mapMappedSsrcSpatialLayer;
std::vector<RTC::RtpStreamSend*> rtpStreams;
std::vector<RTC::RtpStream*> producerRtpStreams; // Indexed by spatial layer.
bool syncRequired{ false };
RTC::SeqManager<uint16_t> rtpSeqManager;
int16_t preferredSpatialLayer{ -1 };
int16_t preferredTemporalLayer{ -1 };
int16_t provisionalTargetSpatialLayer{ -1 };
int16_t provisionalTargetTemporalLayer{ -1 };
int16_t targetSpatialLayer{ -1 };
int16_t targetTemporalLayer{ -1 };
int16_t currentSpatialLayer{ -1 };
int16_t tsReferenceSpatialLayer{ -1 }; // Used for RTP TS sync.
std::unique_ptr<RTC::Codecs::EncodingContext> encodingContext;
uint32_t tsOffset{ 0u }; // RTP Timestamp offset.
bool keyFrameForTsOffsetRequested{ false };
uint64_t lastBweDowngradeAtMs{ 0u }; // Last time we moved to lower spatial layer due to BWE.
};
这里需要提到成员命名中的空间层和时间层的概念，码率其实是一个数据大小与时间的比值，那么是两个维度的衡量值，因此我们控制发送码率可以通过两个维度去调整，一个是空间层级、一个是时间层级。与空间强相关的就是我们的分辨率——也就是每张图片的大小，而与时间强相关的则是帧率——也就是每秒多少张图，这样我们就可以控制流的大小了。
而代码中关键成员为下列几个（我们可以直接理解空间为分辨率标志，时间为帧率标志）：
空间层级：currentSpatialLayer（当前大小）、targetSpatialLayer（目标大小）、preferredSpatialLayer（最佳大小）、provisionalTargetSpatialLayer（临时大小）、tsReferenceSpatialLayer（参考空间）；
时间层级：preferredTemporalLayer（最佳时间）、provisionalTargetTemporalLayer（参考时间）、targetTemporalLayer（目标时间）。
以上的成员作为 producerRtpStreams 管理的标识来对流进行分辨，当有新的流生成时，会根据流中的参数放到producerRtpStreams中等待切换。具体代码如下：
void SimulcastConsumer::ProducerRtpStream(RTC::RtpStream* rtpStream, uint32_t mappedSsrc)
{
MS_TRACE();
auto it = this->mapMappedSsrcSpatialLayer.find(mappedSsrc);
MS_ASSERT(it != this->mapMappedSsrcSpatialLayer.end(), "unknown mappedSsrc");
int16_t spatialLayer = it->second;
this->producerRtpStreams[spatialLayer] = rtpStream;
}
void SimulcastConsumer::ProducerNewRtpStream(RTC::RtpStream* rtpStream, uint32_t mappedSsrc)
{
MS_TRACE();
auto it = this->mapMappedSsrcSpatialLayer.find(mappedSsrc);
MS_ASSERT(it != this->mapMappedSsrcSpatialLayer.end(), "unknown mappedSsrc");
int16_t spatialLayer = it->second;
this->producerRtpStreams[spatialLayer] = rtpStream;
// Emit the score event.
EmitScore();
if (IsActive())
MayChangeLayers();
}
void SimulcastConsumer::ProducerRtpStreamScore(
RTC::RtpStream* /*rtpStream*/, uint8_t score, uint8_t previousScore)
{
MS_TRACE();
// Emit the score event.
EmitScore();
if (RTC::Consumer::IsActive())
{
// Just check target layers if the stream has died or reborned.
// clang-format off
if (
!this->externallyManagedBitrate ||
(score == 0u || previousScore == 0u)
)
// clang-format on
{
MayChangeLayers();
}
}
}
流存储结束后我们分析一下发送rtp数据的流程。在之前的文章中提到的拥塞控制模块中的TransportCongestionControlClient类是服务端中控制发送的部分。我们直接从feedback然后带宽估计完成后进行解析。当带宽估计完成，我们的目标码率和当前码率出现差异时将会进行下面的函数调用。
void TransportCongestionControlClient::MayEmitAvailableBitrateEvent(uint32_t previousAvailableBitrate)
{
MS_TRACE();
uint64_t nowMs = DepLibUV::GetTimeMsInt64();
bool notify{ false };
// Ignore if first event.
// NOTE: Otherwise it will make the Transport crash since this event also happens
// during the constructor of this class.
if (this->lastAvailableBitrateEventAtMs == 0u)
{
this->lastAvailableBitrateEventAtMs = nowMs;
return;
}
// Emit if this is the first valid event.
if (!this->availableBitrateEventCalled)
{
this->availableBitrateEventCalled = true;
notify = true;
}
// Emit event if AvailableBitrateEventInterval elapsed.
else if (nowMs - this->lastAvailableBitrateEventAtMs >= AvailableBitrateEventInterval)
{
notify = true;
}
// Also emit the event fast if we detect a high BWE value decrease.
else if (this->bitrates.availableBitrate < previousAvailableBitrate * 0.75)
{
MS_WARN_TAG(
bwe,
"high BWE value decrease detected, notifying the listener [now:%" PRIu32 ", before:%" PRIu32
"]",
this->bitrates.availableBitrate,
previousAvailableBitrate);
notify = true;
}
// Also emit the event fast if we detect a high BWE value increase.
else if (this->bitrates.availableBitrate > previousAvailableBitrate * 1.50)
{
MS_DEBUG_TAG(
bwe,
"high BWE value increase detected, notifying the listener [now:%" PRIu32 ", before:%" PRIu32
"]",
this->bitrates.availableBitrate,
previousAvailableBitrate);
notify = true;
}
if (notify)
{
MS_DEBUG_DEV(
"notifying the listener with new available bitrate:%" PRIu32,
this->bitrates.availableBitrate);
this->lastAvailableBitrateEventAtMs = nowMs;
this->listener->OnTransportCongestionControlClientBitrates(this, this->bitrates);
}
}
上述代码中主要干了码率增大减小时是否变更码流的阈值判断，思想就是：当码率在2s内迅速减小到75%时切换小码率、2s内码率增大到150%时切换大码率。意味着在上调码率上更为保守可以保证流畅传输，不至于因为网络传输的波动而导致来回切换。当上述的判断结束后，就会进入传输控制类的码率调整的方法中：
inline void Transport::OnTransportCongestionControlClientBitrates(
RTC::TransportCongestionControlClient* /*tccClient*/,
RTC::TransportCongestionControlClient::Bitrates& bitrates)
{
MS_TRACE();
MS_DEBUG_DEV("outgoing available bitrate:%" PRIu32, bitrates.availableBitrate);
DistributeAvailableOutgoingBitrate();
ComputeOutgoingDesiredBitrate();
// May emit 'trace' event.
EmitTraceEventBweType(bitrates);
}
Transport类管理的所有传输控制的内容，算是一个中介者。在DistributeAvailableOutgoingBitrate函数中对切换流的具体内容进行了控制：
void Transport::DistributeAvailableOutgoingBitrate()
{
MS_TRACE();
MS_ASSERT(this->tccClient, "no TransportCongestionClient");
std::multimap<uint8_t, RTC::Consumer*> multimapPriorityConsumer;
// Fill the map with Consumers and their priority (if > 0).
for (auto& kv : this->mapConsumers)
{
auto* consumer = kv.second;
auto priority = consumer->GetBitratePriority();
if (priority > 0u)
multimapPriorityConsumer.emplace(priority, consumer);
}
// Nobody wants bitrate. Exit.
if (multimapPriorityConsumer.empty())
return;
uint32_t availableBitrate = this->tccClient->GetAvailableBitrate();
this->tccClient->RescheduleNextAvailableBitrateEvent();
MS_DEBUG_DEV("before layer-by-layer iterations [availableBitrate:%" PRIu32 "]", availableBitrate);
// Redistribute the available bitrate by allowing Consumers to increase
// layer by layer. Take into account the priority of each Consumer to
// provide it with more bitrate.
while (availableBitrate > 0u)
{
auto previousAvailableBitrate = availableBitrate;
for (auto it = multimapPriorityConsumer.rbegin(); it != multimapPriorityConsumer.rend(); ++it)
{
auto priority = it->first;
auto* consumer = it->second;
auto bweType = this->tccClient->GetBweType();
// If a Consumer has priority > 1, call IncreaseLayer() more times to
// provide it with more available bitrate to choose its preferred layers.
for (uint8_t i{ 1u }; i <= priority; ++i)
{
uint32_t usedBitrate;
switch (bweType)
{
case RTC::BweType::TRANSPORT_CC:
usedBitrate = consumer->IncreaseLayer(availableBitrate, /*considerLoss*/ false);
break;
case RTC::BweType::REMB:
usedBitrate = consumer->IncreaseLayer(availableBitrate, /*considerLoss*/ true);
break;
}
MS_ASSERT(usedBitrate <= availableBitrate, "Consumer used more layer bitrate than given");
availableBitrate -= usedBitrate;
// Exit the loop fast if used bitrate is 0.
if (usedBitrate == 0u)
break;
}
}
// If no Consumer used bitrate, exit the loop.
if (availableBitrate == previousAvailableBitrate)
break;
}
MS_DEBUG_DEV("after layer-by-layer iterations [availableBitrate:%" PRIu32 "]", availableBitrate);
// Finally instruct Consumers to apply their computed layers.
for (auto it = multimapPriorityConsumer.rbegin(); it != multimapPriorityConsumer.rend(); ++it)
{
auto* consumer = it->second;
consumer->ApplyLayers();
}
}
把待切换的码率通过关联的 consumer 类成员传入 IncreaseLayer。
uint32_t SimulcastConsumer::IncreaseLayer(uint32_t bitrate, bool considerLoss)
{
MS_TRACE();
MS_ASSERT(this->externallyManagedBitrate, "bitrate is not externally managed");
MS_ASSERT(IsActive(), "should be active");
// If already in the preferred layers, do nothing.
// clang-format off
if (
this->provisionalTargetSpatialLayer == this->preferredSpatialLayer &&
this->provisionalTargetTemporalLayer == this->preferredTemporalLayer
)
// clang-format on
{
return 0u;
}
uint32_t virtualBitrate;
if (considerLoss)
{
// Calculate virtual available bitrate based on given bitrate and our
// packet lost.
auto lossPercentage = this->rtpStream->GetLossPercentage();
if (lossPercentage < 2)
virtualBitrate = 1.08 * bitrate;
else if (lossPercentage > 10)
virtualBitrate = (1 - 0.5 * (lossPercentage / 100)) * bitrate;
else
virtualBitrate = bitrate;
}
else
{
virtualBitrate = bitrate;
}
uint32_t requiredBitrate{ 0u };
int16_t spatialLayer{ 0 };
int16_t temporalLayer{ 0 };
auto nowMs = DepLibUV::GetTimeMs();
for (size_t sIdx{ 0u }; sIdx < this->producerRtpStreams.size(); ++sIdx)
{
spatialLayer = static_cast<int16_t>(sIdx);
// If this is higher than current spatial layer and we moved to to current spatial
// layer due to BWE limitations, check how much it has elapsed since then.
if (nowMs - this->lastBweDowngradeAtMs < BweDowngradeConservativeMs)
{
if (this->provisionalTargetSpatialLayer > -1 && spatialLayer > this->currentSpatialLayer)
{
MS_DEBUG_DEV(
"avoid upgrading to spatial layer %" PRIi16 " due to recent BWE downgrade", spatialLayer);
goto done;
}
}
// Ignore spatial layers lower than the one we already have.
if (spatialLayer < this->provisionalTargetSpatialLayer)
continue;
// This can be null.
auto* producerRtpStream = this->producerRtpStreams.at(spatialLayer);
// Producer stream does not exist or it's not good. Ignore.
if (!producerRtpStream || producerRtpStream->GetScore() < StreamGoodScore)
continue;
// If the stream has not been active time enough and we have an active one
// already, move to the next spatial layer.
// clang-format off
if (
spatialLayer != this->provisionalTargetSpatialLayer &&
this->provisionalTargetSpatialLayer != -1 &&
producerRtpStream->GetActiveMs() < StreamMinActiveMs
)
// clang-format on
{
continue;
}
// We may not yet switch to this spatial layer.
if (!CanSwitchToSpatialLayer(spatialLayer))
continue;
temporalLayer = 0;
// Check bitrate of every temporal layer.
for (; temporalLayer < producerRtpStream->GetTemporalLayers(); ++temporalLayer)
{
// Ignore temporal layers lower than the one we already have (taking into account
// the spatial layer too).
// clang-format off
if (
spatialLayer == this->provisionalTargetSpatialLayer &&
temporalLayer <= this->provisionalTargetTemporalLayer
)
// clang-format on
{
continue;
}
requiredBitrate = producerRtpStream->GetLayerBitrate(nowMs, 0, temporalLayer);
// This is simulcast so we must substract the bitrate of the current temporal
// spatial layer if this is the temporal layer 0 of a higher spatial layer.
//
// clang-format off
if (
requiredBitrate &&
temporalLayer == 0 &&
this->provisionalTargetSpatialLayer > -1 &&
spatialLayer > this->provisionalTargetSpatialLayer
)
// clang-format on
{
auto* provisionalProducerRtpStream =
this->producerRtpStreams.at(this->provisionalTargetSpatialLayer);
auto provisionalRequiredBitrate = provisionalProducerRtpStream->GetLayerBitrate(
nowMs, 0, this->provisionalTargetTemporalLayer);
if (requiredBitrate > provisionalRequiredBitrate)
requiredBitrate -= provisionalRequiredBitrate;
else
requiredBitrate = 1u; // Don't set 0 since it would be ignored.
}
MS_DEBUG_DEV(
"testing layers %" PRIi16 ":%" PRIi16 " [virtual bitrate:%" PRIu32
", required bitrate:%" PRIu32 "]",
spatialLayer,
temporalLayer,
virtualBitrate,
requiredBitrate);
// If active layer, end iterations here. Otherwise move to next spatial layer.
if (requiredBitrate)
goto done;
else
break;
}
// If this is the preferred or higher spatial layer, take it and exit.
if (spatialLayer >= this->preferredSpatialLayer)
break;
}
done:
// No higher active layers found.
if (!requiredBitrate)
return 0u;
// No luck.
if (requiredBitrate > virtualBitrate)
return 0u;
// Set provisional layers.
this->provisionalTargetSpatialLayer = spatialLayer;
this->provisionalTargetTemporalLayer = temporalLayer;
MS_DEBUG_DEV(
"setting provisional layers to %" PRIi16 ":%" PRIi16 " [virtual bitrate:%" PRIu32
", required bitrate:%" PRIu32 "]",
this->provisionalTargetSpatialLayer,
this->provisionalTargetTemporalLayer,
virtualBitrate,
requiredBitrate);
if (requiredBitrate <= bitrate)
return requiredBitrate;
else if (requiredBitrate <= virtualBitrate)
return bitrate;
else
return requiredBitrate; // NOTE: This cannot happen.
}
而这个函数是对层级的切换控制函数，主要思想是：通过对流的评分系统来挑选出一个适合的流，然后把流切换到目标中。最后是调用SendRtpPacket函数发送完成。
1.3 Simulcast的缺陷
虽然Simulcast可以解决下行带宽不足时降码率的需求，但是在应用的过程中存在比较致命的缺陷：
1.上行带宽的增加，我们在上行时会多发送多条流，导致大量的带宽浪费。而为了应对几个下行网络较差的用户而牺牲上行用户的带宽资源，这样的做法有待商榷；
2.下行可供的流选择性很少，及时使用3条流同时传输的方式，在移动网络这样复杂多变的网络条件下，上下调整的幅度将会是巨大的，会造成不好的用户体验，相比SVC适用性较差。
二、SVC
2.1 SVC简介
可伸缩视频编码SVC（Scalable Video Coding）技术是H.264标准的一个扩展，最初由JVT在2004年开始制定。H.264 SVC是H.264标准的扩展部分，SVC扩展部分引入了一种传统H.264 AVC不存在的概念——编码流中的层。基本层编码最低层的时域、空域和质量流；增强层以基本层作为起始点，对附加信息进行，从而在解码过程中重构更高层的质量、分辨率和时域层。通过解码基本层和相邻增强层，解码器能生成特定层的视频流。
2.2 mediasoup中的SVC
得益于mediasoup良好的模块分化，SVC模块和上述的simulcast模块的实现类似。
class SvcConsumer : public RTC::Consumer, public RTC::RtpStreamSend::Listener
{
public:
SvcConsumer(
const std::string& id,
const std::string& producerId,
RTC::Consumer::Listener* listener,
json& data);
~SvcConsumer() override;
public:
void FillJson(json& jsonObject) const override;
void FillJsonStats(json& jsonArray) const override;
void FillJsonScore(json& jsonObject) const override;
void HandleRequest(Channel::ChannelRequest* request) override;
RTC::Consumer::Layers GetPreferredLayers() const override
{
RTC::Consumer::Layers layers;
layers.spatial = this->preferredSpatialLayer;
layers.temporal = this->preferredTemporalLayer;
return layers;
}
bool IsActive() const override
{
// clang-format off
return (
RTC::Consumer::IsActive() &&
this->producerRtpStream &&
(this->producerRtpStream->GetScore() > 0u || this->producerRtpStream->HasDtx())
);
// clang-format on
}
void ProducerRtpStream(RTC::RtpStream* rtpStream, uint32_t mappedSsrc) override;
void ProducerNewRtpStream(RTC::RtpStream* rtpStream, uint32_t mappedSsrc) override;
void ProducerRtpStreamScore(RTC::RtpStream* rtpStream, uint8_t score, uint8_t previousScore) override;
void ProducerRtcpSenderReport(RTC::RtpStream* rtpStream, bool first) override;
uint8_t GetBitratePriority() const override;
uint32_t IncreaseLayer(uint32_t bitrate, bool considerLoss) override;
void ApplyLayers() override;
uint32_t GetDesiredBitrate() const override;
void SendRtpPacket(RTC::RtpPacket* packet) override;
void GetRtcp(RTC::RTCP::CompoundPacket* packet, RTC::RtpStreamSend* rtpStream, uint64_t nowMs) override;
std::vector<RTC::RtpStreamSend*> GetRtpStreams() override
{
return this->rtpStreams;
}
void NeedWorstRemoteFractionLost(uint32_t mappedSsrc, uint8_t& worstRemoteFractionLost) override;
void ReceiveNack(RTC::RTCP::FeedbackRtpNackPacket* nackPacket) override;
void ReceiveKeyFrameRequest(RTC::RTCP::FeedbackPs::MessageType messageType, uint32_t ssrc) override;
void ReceiveRtcpReceiverReport(RTC::RTCP::ReceiverReport* report) override;
uint32_t GetTransmissionRate(uint64_t nowMs) override;
float GetRtt() const override;
private:
void UserOnTransportConnected() override;
void UserOnTransportDisconnected() override;
void UserOnPaused() override;
void UserOnResumed() override;
void CreateRtpStream();
void RequestKeyFrame();
void MayChangeLayers(bool force = false);
bool RecalculateTargetLayers(int16_t& newTargetSpatialLayer, int16_t& newTargetTemporalLayer) const;
void UpdateTargetLayers(int16_t newTargetSpatialLayer, int16_t newTargetTemporalLayer);
void EmitScore() const;
void EmitLayersChange() const;
/* Pure virtual methods inherited from RtpStreamSend::Listener. */
public:
void OnRtpStreamScore(RTC::RtpStream* rtpStream, uint8_t score, uint8_t previousScore) override;
void OnRtpStreamRetransmitRtpPacket(RTC::RtpStreamSend* rtpStream, RTC::RtpPacket* packet) override;
private:
// Allocated by this.
RTC::RtpStreamSend* rtpStream{ nullptr };
// Others.
std::vector<RTC::RtpStreamSend*> rtpStreams;
RTC::RtpStream* producerRtpStream{ nullptr };
bool syncRequired{ false };
RTC::SeqManager<uint16_t> rtpSeqManager;
int16_t preferredSpatialLayer{ -1 };
int16_t preferredTemporalLayer{ -1 };
int16_t provisionalTargetSpatialLayer{ -1 };
int16_t provisionalTargetTemporalLayer{ -1 };
std::unique_ptr<RTC::Codecs::EncodingContext> encodingContext;
uint64_t lastBweDowngradeAtMs{ 0u }; // Last time we moved to lower spatial layer due to BWE.
};
与simulcast类似的分层概念也应用到了svc模块中。
而比较重要的点是，mediasoup中的svc只支持vp9的方式进行，在Tools.hpp中已经明确写了：
static bool IsValidTypeForCodec(RTC::RtpParameters::Type type, const RTC::RtpCodecMimeType& mimeType)
{
switch (type)
{
case RTC::RtpParameters::Type::NONE:
{
return false;
}
case RTC::RtpParameters::Type::SIMPLE:
{
return true;
}
case RTC::RtpParameters::Type::SIMULCAST:
{
switch (mimeType.type)
{
case RTC::RtpCodecMimeType::Type::VIDEO:
{
switch (mimeType.subtype)
{
case RTC::RtpCodecMimeType::Subtype::VP8:
case RTC::RtpCodecMimeType::Subtype::H264:
return true;
default:
return false;
}
}
default:
{
return false;
}
}
}
case RTC::RtpParameters::Type::SVC:
{
switch (mimeType.type)
{
case RTC::RtpCodecMimeType::Type::VIDEO:
{
switch (mimeType.subtype)
{
case RTC::RtpCodecMimeType::Subtype::VP9:
return true;
default:
return false;
}
}
default:
{
return false;
}
}
}
case RTC::RtpParameters::Type::PIPE:
{
return true;
}
default:
{
return false;
}
}
}
而SVC的producer会在创建的时候和simulcast一样，根据传入的rtpParameters来创建。随后走层级切换的逻辑来进行传输：
void SvcConsumer::ApplyLayers()
{
MS_TRACE();
MS_ASSERT(this->externallyManagedBitrate, "bitrate is not externally managed");
MS_ASSERT(IsActive(), "should be active");
auto provisionalTargetSpatialLayer = this->provisionalTargetSpatialLayer;
auto provisionalTargetTemporalLayer = this->provisionalTargetTemporalLayer;
// Reset provisional target layers.
this->provisionalTargetSpatialLayer = -1;
this->provisionalTargetTemporalLayer = -1;
if (!IsActive())
return;
// clang-format off
if (
provisionalTargetSpatialLayer != this->encodingContext->GetTargetSpatialLayer() ||
provisionalTargetTemporalLayer != this->encodingContext->GetTargetTemporalLayer()
)
// clang-format on
{
UpdateTargetLayers(provisionalTargetSpatialLayer, provisionalTargetTemporalLayer);
// If this looks like a spatial layer downgrade due to BWE limitations, set member.
// clang-format off
if (
this->rtpStream->GetActiveMs() > BweDowngradeMinActiveMs &&
this->encodingContext->GetTargetSpatialLayer() < this->encodingContext->GetCurrentSpatialLayer() &&
this->encodingContext->GetCurrentSpatialLayer() <= this->preferredSpatialLayer
)
// clang-format on
{
MS_DEBUG_DEV(
"possible target spatial layer downgrade (from %" PRIi16 " to %" PRIi16
") due to BWE limitation",
this->encodingContext->GetCurrentSpatialLayer(),
this->encodingContext->GetTargetSpatialLayer());
this->lastBweDowngradeAtMs = DepLibUV::GetTimeMs();
}
}
}
...
uint32_t SvcConsumer::IncreaseLayer(uint32_t bitrate, bool considerLoss)
{
MS_TRACE();
MS_ASSERT(this->externallyManagedBitrate, "bitrate is not externally managed");
MS_ASSERT(IsActive(), "should be active");
if (this->producerRtpStream->GetScore() == 0u)
return 0u;
// If already in the preferred layers, do nothing.
// clang-format off
if (
this->provisionalTargetSpatialLayer == this->preferredSpatialLayer &&
this->provisionalTargetTemporalLayer == this->preferredTemporalLayer
)
// clang-format on
{
return 0u;
}
uint32_t virtualBitrate;
if (considerLoss)
{
// Calculate virtual available bitrate based on given bitrate and our
// packet lost.
auto lossPercentage = this->rtpStream->GetLossPercentage();
if (lossPercentage < 2)
virtualBitrate = 1.08 * bitrate;
else if (lossPercentage > 10)
virtualBitrate = (1 - 0.5 * (lossPercentage / 100)) * bitrate;
else
virtualBitrate = bitrate;
}
else
{
virtualBitrate = bitrate;
}
uint32_t requiredBitrate{ 0u };
int16_t spatialLayer{ 0 };
int16_t temporalLayer{ 0 };
auto nowMs = DepLibUV::GetTimeMs();
for (; spatialLayer < this->producerRtpStream->GetSpatialLayers(); ++spatialLayer)
{
// If this is higher than current spatial layer and we moved to to current spatial
// layer due to BWE limitations, check how much it has elapsed since then.
if (nowMs - this->lastBweDowngradeAtMs < BweDowngradeConservativeMs)
{
if (this->provisionalTargetSpatialLayer > -1 && spatialLayer > this->encodingContext->GetCurrentSpatialLayer())
{
MS_DEBUG_DEV(
"avoid upgrading to spatial layer %" PRIi16 " due to recent BWE downgrade", spatialLayer);
goto done;
}
}
// Ignore spatial layers lower than the one we already have.
if (spatialLayer < this->provisionalTargetSpatialLayer)
continue;
temporalLayer = 0;
// Check bitrate of every temporal layer.
for (; temporalLayer < this->producerRtpStream->GetTemporalLayers(); ++temporalLayer)
{
// Ignore temporal layers lower than the one we already have (taking into account
// the spatial layer too).
// clang-format off
if (
spatialLayer == this->provisionalTargetSpatialLayer &&
temporalLayer <= this->provisionalTargetTemporalLayer
)
// clang-format on
{
continue;
}
requiredBitrate =
this->producerRtpStream->GetLayerBitrate(nowMs, spatialLayer, temporalLayer);
MS_DEBUG_DEV(
"testing layers %" PRIi16 ":%" PRIi16 " [virtual bitrate:%" PRIu32
", required bitrate:%" PRIu32 "]",
spatialLayer,
temporalLayer,
virtualBitrate,
requiredBitrate);
// If active layer, end iterations here. Otherwise move to next spatial layer.
if (requiredBitrate)
goto done;
else
break;
}
// If this is the preferred or higher spatial layer, take it and exit.
if (spatialLayer >= this->preferredSpatialLayer)
break;
}
done:
// No higher active layers found.
if (!requiredBitrate)
return 0u;
// No luck.
if (requiredBitrate > virtualBitrate)
return 0u;
// Set provisional layers.
this->provisionalTargetSpatialLayer = spatialLayer;
this->provisionalTargetTemporalLayer = temporalLayer;
MS_DEBUG_DEV(
"upgrading to layers %" PRIi16 ":%" PRIi16 " [virtual bitrate:%" PRIu32
", required bitrate:%" PRIu32 "]",
this->provisionalTargetSpatialLayer,
this->provisionalTargetTemporalLayer,
virtualBitrate,
requiredBitrate);
if (requiredBitrate <= bitrate)
return requiredBitrate;
else if (requiredBitrate <= virtualBitrate)
return bitrate;
else
return requiredBitrate; // NOTE: This cannot happen.
}
void SvcConsumer::ApplyLayers()
{
MS_TRACE();
MS_ASSERT(this->externallyManagedBitrate, "bitrate is not externally managed");
MS_ASSERT(IsActive(), "should be active");
auto provisionalTargetSpatialLayer = this->provisionalTargetSpatialLayer;
auto provisionalTargetTemporalLayer = this->provisionalTargetTemporalLayer;
// Reset provisional target layers.
this->provisionalTargetSpatialLayer = -1;
this->provisionalTargetTemporalLayer = -1;
if (!IsActive())
return;
// clang-format off
if (
provisionalTargetSpatialLayer != this->encodingContext->GetTargetSpatialLayer() ||
provisionalTargetTemporalLayer != this->encodingContext->GetTargetTemporalLayer()
)
// clang-format on
{
UpdateTargetLayers(provisionalTargetSpatialLayer, provisionalTargetTemporalLayer);
// If this looks like a spatial layer downgrade due to BWE limitations, set member.
// clang-format off
if (
this->rtpStream->GetActiveMs() > BweDowngradeMinActiveMs &&
this->encodingContext->GetTargetSpatialLayer() < this->encodingContext->GetCurrentSpatialLayer() &&
this->encodingContext->GetCurrentSpatialLayer() <= this->preferredSpatialLayer
)
// clang-format on
{
MS_DEBUG_DEV(
"possible target spatial layer downgrade (from %" PRIi16 " to %" PRIi16
") due to BWE limitation",
this->encodingContext->GetCurrentSpatialLayer(),
this->encodingContext->GetTargetSpatialLayer());
this->lastBweDowngradeAtMs = DepLibUV::GetTimeMs();
}
}
}
三、结语
本文只是对mediasoup的simulcast和SVC的层级进行简单的介绍，后面会对simulcast或者svc整体应用的过程（包括：创建、传输、拥塞控制交互进行分析）。