GStreamer basesink中的时钟同步
播放复杂媒体时，每个audio和video sample必须在特定时间按特定顺序播放。为此，GStreamer提供了一种同步机制，通过使用 GstClock object、buffer timestamps和SEGMENT event来实现：
（1）GstClock：GstPipeline中的所有element的全局clock，pipeline由PAUSED state转换到PLAYING state时候通过gst_element_provide_clock() 获取到一个clock，并下发给pipeline中所以element。
（2）SEGMENT event：element向下游元素发送buffer之前，必须先发送SEGMENT事件。SEGMENT event主要包括了segment的 rate, start/stop等, 其中SEGMENT 的start/stop规定了buffer的有效时间戳范围。
（3）buffer timestamps：只有timestamps处于SEGMENT 的start 和stop 之间的buffers 是可以被显示的，否则将被丢弃或者裁剪。
1 Clock running-time
一台计算机中，有多种源可以用作时间源，例如系统时间、声卡、CPU计数器等。因此，GStreamer有许多可用的GstClock实现。时钟时间不必从0或其他特定值开始，一些时钟从特定的开始日期开始计数，另一些则从上次重新启动开始计数。
在pipeline由PAUSED向PLAYING状态切换时，gst_pipeline_change_state函数会调用gst_element_provide_clock产生一个clock，并将这个clock发布给所有element，这也意味着pipeline中所有element拥有相同的clock。
pipeline选择clock后，clock保持在running_time，running_time表示在PLAYING state下花费的总时间，不同状态下running_time的计算如下：
（1）如果pipeline为NULL/READY，则running_time未定义。
（2）在PAUSED中，running_time为上次PAUSED时的时间。当stream第一次进入PAUSED state时，running_time为0。
（3）在PLAYING中，running_time是absolute_time和base time之间的差值。base time定义为absolute_time减去pipeline设置为PLAYING时的运行时间。
（4）seek后，running_time设置为0。
gstreamer中使用gst_clock_get_time ()来返回一个absolute-time，每次调用该函数获得的absolute-time都是一个新值，且是单调递增的。
base-time是pipeline由PAUSED向PLAYING那一瞬间的系统时间，pipeline由PAUSED向PLAYING状态切换时会调用gst_clock_get_time ()获取到当前的系统时间，并将之作为base-time，设置给所有的element, 因此同一个pipeline中所有的element具有相同的base-time。在PLAYING state时，running-time是指base-time与当前absolute-time 之间的差值。Pipeline 由PAUSED向PLAYING状态切换时获取当前系统时间的code如下：
获取到的当前系统时间now调整后作为base-time 设置给所有element。
由于pipeline中的所有element都有相同的clock和base-time，因此它们都可以根据pipeline时钟计算运行时间。
在PLAYING state时，running_time的计算方法如下：
C.running_time = absolute_time - base_time
C.running_time是通过clock获得的running_time，该值以时钟的速率单调增加。
2 Buffer running-time
上面介绍了clock的running-time计算方法，实际上每个buffer也有一个running-time，同步前需要计算buffer的running-time。要计算buffer running-time，需要使用buffer 的timestamp和buffer前面的SEGMENT事件。首先，需要将SEGMENT事件转换为GstSegment对象，然后使用gst_SEGMENT_to_running_time（）函数来计算buffer的running-time。
buffer的running-time计算前先使用以下符号定义：
B: GstBuffer
B.timestamp = buffer timestamp (GST_BUFFER_PTS or GST_BUFFER_DTS)
S: SEGMENT event preceding the buffers.。
S.start：SEGMENT事件中的开始字段，这是允许的最低时间戳。
S.stop：SEGMENT事件中的停止字段，这是允许的上限时间戳。
S.rate：SEGMENT事件的速率字段，定义播放速率。
S.base：时间的基准时间，先前所有segment的总running_time。
S.offset： S.start或S.stop的偏移量。
有效buffer是B.timestamp在S.start和S.stop之间buffer，应将该范围之外的所有其他buffer会被丢弃或剪裁。GstBuffer timestamps和前面的SEGMENT事件定义了buffer timestamps到running_time的转换如下：
if (S.rate > 0.0)
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
=>
B.timestamp = (B.running_time - S.base) * ABS (S.rate) + S.start + S.offset
else
B.running_time = ((S.stop - S.offset) - B.timestamp) / ABS (S.rate) + S.base
=>
B.timestamp = S.stop - S.offset - ((B.running_time - S.base) * ABS (S.rate))
B.running_time代表根据SEGMENT event和该segment中buffer的timestamp获得的running_time。
正常情况下第一个可显示的buffer的B.running_time值为0，因为B.timestamp=S.start， S.offset = 0以及S.base = 0。
对于S.rate>1.0，timestamp将按比例缩小以提高播放速率。同样，0.0和1.0之间的速率会减慢播放速度。
对于负速率，从S.stop到S.start接收时间戳，使得接收到的第一缓冲区将被转换为B.running_time为0（B.timestamp＝＝S.stop和S.base＝＝0）。
3 Stream time
stream time也称为stream中的position，该值介于0和媒体文件duration之间的值。stream time作用如下：
（1）回报pipeline中的POSITION查询
（2）seek events/queries中使用的位置
（3）用于同步控制器值的位置
stream time计算的时候会使用SEGMENT中的字段：
S.time：SEGMENT事件中的时间字段，是S.start的stream-time。
S.applied_rate：已应用于segment的速率。
stream time是使用buffer times和前面的SEGMENTevent计算的，如下所示：
stream_time = (B.timestamp - S.start) * ABS (S.applied_rate) + S.time
=> B.timestamp = (stream_time - S.time) / ABS(S.applied_rate) + S.start
对于负速率，B.timestamp将从S.stop倒退到S.start，使stream-time倒退：
stream_time = (S.stop - B.timestamp) * ABS(S.applied_rate) + S.time
=> B.timestamp = S.stop - (stream_time - S.time) / ABS(S.applied_rate)
4 Gstreamer 同步用到的各个时间的对应关系
上面介绍了clock-time、running-time，stream-time以及basetime的含义，以下是各个timelines的之间的对应关系。下图表示播放100ms样本并在50ms到100ms之间重复该部分时，pipeline中的不同时间。
从上图中可以看到buffer的running-time总是随着clock time单调递增。当buffer的running-time等于(clock-time) – (base-time)时，就会播放该buffer。stream time表示流中的位置，并在重复时向后跳跃。
5 stream同步
使用clock和element的base_time，可以获得一个clock的running_time:
C.running_time = absolute_time - base_time
使用buffer timestamp和先前的SEGMENT event（假定正播放速率）可以获取一个buffer的running_time：
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
同步的任务是确保当clock达到相同的C.running_time时播放B.running_time的buffer。因此，以下内容必须成立：
B.running_time = C.running_time
应播放B.running_time buffer时的absolute_time用B.sync_time表示，
B.sync_time = B.running_time + base_time
Sink中render buffer前需要等时钟达到B.sync_time，对于多个流，这意味着具有相同running_time的buffer将同时显示，这样就达到了同步的目的。
5.1 buffer running time 计算
每一张video数据render前都会走到gst_base_sink_do_sync这个api中,该函数中首先会调用到gst_base_sink_default_get_times这个api，用来计算buffer 的start timestamp和stop timestamp。Start timestamp是buffer本身的timestamp，stop是buffer timestamp + duration。
Start timestamp和end timestamp计算后会经过gst_segment_clip ，变为clip的 strart position =>cstart，以及clip后的stop postion=> cstop。clip的作用就是保证cstart， cstop在segment之内。
然后cstart， cstop调用gst_segment_to_running_time 将这buffer 的start 和stop timestamp变为running time start 和running time stop。
gst_segment_to_running_time中调用了gst_segment_to_running_time_full，该api中先计算了B.timestamp - (S.start + S.offset) 这一部分：
然后将上面的计算结果除以ABS (S.rate) 再加上S.base:
与之前的提到的buffer running time的计算方法一致。
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
5.2 buffer stream time 计算
cstart， cstop通过调用gst_segment_to_stream_time api获得buffer的stream time start 和stream time stop，需要注意的是stream time并不参与同步。
gst_segment_to_stream_time先获取segment start 和segment time:
start = segment->start;
time = segment->time;
然后根据stream time的计算公式获得stream time。
stream_time = (B.timestamp - S.start) * ABS (S.applied_rate) + S.time
/* add or subtract from segment time based on applied rate */
if (G_LIKELY (segment->applied_rate > 0.0)) {
if (G_LIKELY (position > start)) {
/* bring to uncorrected position in segment */
*stream_time = position - start;
/* correct for applied rate if needed */
if (G_UNLIKELY (abs_applied_rate != 1.0))
*stream_time *= abs_applied_rate;
/* correct for segment time */
*stream_time += time;
res = 1;
} else {
*stream_time = start - position;
if (G_UNLIKELY (abs_applied_rate != 1.0))
*stream_time *= abs_applied_rate;
if (*stream_time > time) {
*stream_time -= time;
res = -1;
} else {
*stream_time = time - *stream_time;
res = 1;
}
}
5.3 buffer running start time调整
获取到rstart(buffer running start time)后需要根据latency 和 timestamp offset调整buffer running start time。
/* adjust for latency */
stime = gst_base_sink_adjust_time (basesink, rstart);
引入latency主要是考虑到buffer从source推送到sink需要花费一定的时间。
timestamp offset主要让timestamp不正确的buffer也可以播放，该值可以通过set property设置给sink。设置负值，buffer将在早于其timestamp render，设置正值会延迟render。
/* with STREAM_LOCK, PREROLL_LOCK, LOCK
* adjust a timestamp with the latency and timestamp offset. This function does
* not adjust for the render delay. */
static GstClockTime
gst_base_sink_adjust_time (GstBaseSink * basesink, GstClockTime time)
{
GstClockTimeDiff ts_offset;
/* don't do anything funny with invalid timestamps */
if (G_UNLIKELY (!GST_CLOCK_TIME_IS_VALID (time)))
return time;
time += basesink->priv->latency;
/* apply offset, be careful for underflows */
ts_offset = basesink->priv->ts_offset;
if (ts_offset < 0) {
ts_offset = -ts_offset;
if (ts_offset < time)
time -= ts_offset;
else
time = 0;
} else
time += ts_offset;
/* subtract the render delay again, which was included in the latency */
if (time > basesink->priv->render_delay)
time -= basesink->priv->render_delay;
else
time = 0;
return time;
}
5.4 wait clock time 到达buffer sync time
拿到了buffer running start time后就可以通过gst_base_sink_wait_clock这api来等clock running time 到达buffer running start time，该函数会一直block到clock running time 到达buffer running start time。
/* This function will return immediately if start == -1, no clock
* or sync is disabled with GST_CLOCK_BADTIME. */
status = gst_base_sink_wait_clock (basesink, stime, &jitter);
gst_base_sink_wait_clock中先会将base-time加给buffer running start time，从而获得buffer的sync time（前面有介绍buffer sync time的含义以及计算方法：B.sync_time = B.running_time + base_time）。
GstClockReturn
gst_base_sink_wait_clock (GstBaseSink * sink, GstClockTime time,
GstClockTimeDiff * jitter)
{
GstClockReturn ret;
GstClock *clock;
GstClockTime base_time;
if (G_UNLIKELY (!GST_CLOCK_TIME_IS_VALID (time)))
goto invalid_time;
GST_OBJECT_LOCK (sink);
if (G_UNLIKELY (!sink->sync))
goto no_sync;
if (G_UNLIKELY ((clock = GST_ELEMENT_CLOCK (sink)) == NULL))
goto no_clock;
base_time = GST_ELEMENT_CAST (sink)->base_time;
GST_LOG_OBJECT (sink,
"time %" GST_TIME_FORMAT ", base_time %" GST_TIME_FORMAT,
GST_TIME_ARGS (time), GST_TIME_ARGS (base_time));
/* add base_time to running_time to get the time against the clock */
time += base_time;
拿到buffer的sync time会通过调用gst_clock_entry_reinit api 将 buffer 的sync time 传递给global clock。
/* WARNING : Does not modify the refcount
* WARNING : Do not use if a pending clock operation is happening on that entry */
static gboolean
gst_clock_entry_reinit (GstClock * clock, GstClockEntry * entry,
GstClockTime time, GstClockTime interval, GstClockEntryType type)
{
g_return_val_if_fail (entry->status != GST_CLOCK_BUSY, FALSE);
g_return_val_if_fail (gst_clock_id_uses_clock ((GstClockID) entry, clock),
FALSE);
entry->type = type;
entry->time = time;
entry->interval = interval;
entry->status = GST_CLOCK_OK;
entry->unscheduled = FALSE;
entry->woken_up = FALSE;
return TRUE;
}
然后gst_base_sink_wait_clock 会调用到gst_system_clock_id_wait_jitter_unlocked这个api，该api会调用gst_clock_get_time ()获取当前clock 的absolute-time，并计算absolute-time和buffer sync time之间的diff，该函数最后会调用gst_pthread_cond_wait_until等待diff时长后返回。
/* synchronously wait on the given GstClockEntry.
*
* We do this by blocking on the entry specifically rather than a global
* condition variable so that each possible thread may be woken up
* individually. This ensures that we don't wake up possibly multiple threads
* when unscheduling an entry.
*
* Entries that arrive too late are simply not waited on and a
* GST_CLOCK_EARLY result is returned.
*
* This is called with the ENTRY_LOCK but not SYSTEM_CLOCK_LOCK!
*
* MT safe.
*/
static GstClockReturn
gst_system_clock_id_wait_jitter_unlocked (GstClock * clock,
GstClockEntry * entry, GstClockTimeDiff * jitter, gboolean restart)
{
GstClockTime entryt, now;
GstClockTimeDiff diff;
GstClockReturn status;
gint64 mono_ts;
status = GST_CLOCK_ENTRY_STATUS (entry);
if (G_UNLIKELY (status == GST_CLOCK_UNSCHEDULED)) {
return GST_CLOCK_UNSCHEDULED;
}
/* need to call the overridden method because we want to sync against the time
* of the clock, whatever the subclass uses as a clock. */
now = gst_clock_get_time (clock);
mono_ts = g_get_monotonic_time ();
/* get the time of the entry */
entryt = GST_CLOCK_ENTRY_TIME (entry);
/* the diff of the entry with the clock is the amount of time we have to
* wait */
diff = GST_CLOCK_DIFF (now, entryt);
if (G_LIKELY (jitter))
*jitter = -diff;
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock, "entry %p"
" time %" GST_TIME_FORMAT
" now %" GST_TIME_FORMAT
" diff (time-now) %" G_GINT64_FORMAT,
entry, GST_TIME_ARGS (entryt), GST_TIME_ARGS (now), diff);
if (G_LIKELY (diff > CLOCK_MIN_WAIT_TIME)) {
#ifdef WAIT_DEBUGGING
GstClockTime final;
#endif
while (TRUE) {
gboolean waitret;
#ifdef HAVE_CLOCK_NANOSLEEP
if (diff <= 500 * GST_USECOND) {
/* In order to provide more accurate wait, we will use BLOCKING
clock_nanosleep for any deadlines at or below 500us */
struct timespec end;
GST_TIME_TO_TIMESPEC (mono_ts * 1000 + diff, end);
GST_SYSTEM_CLOCK_ENTRY_UNLOCK ((GstClockEntryImpl *) entry);
waitret =
clock_nanosleep (CLOCK_MONOTONIC, TIMER_ABSTIME, &end, NULL) == 0;
GST_SYSTEM_CLOCK_ENTRY_LOCK ((GstClockEntryImpl *) entry);
} else {
if (diff < 2 * GST_MSECOND) {
/* For any deadline within 2ms, we first use the regular non-blocking
wait by reducing the diff accordingly */
diff -= 500 * GST_USECOND;
}
#endif
/* now wait on the entry, it either times out or the cond is signalled.
* The status of the entry is BUSY only around the wait. */
waitret =
GST_SYSTEM_CLOCK_ENTRY_WAIT_UNTIL ((GstClockEntryImpl *) entry,
mono_ts * 1000 + diff);
#ifdef HAVE_CLOCK_NANOSLEEP
}
#endif
/* get the new status, mark as DONE. We do this so that the unschedule
* function knows when we left the poll and doesn't need to wakeup the
* poll anymore. */
status = GST_CLOCK_ENTRY_STATUS (entry);
/* we were unscheduled, exit immediately */
if (G_UNLIKELY (status == GST_CLOCK_UNSCHEDULED))
break;
if (G_UNLIKELY (status != GST_CLOCK_BUSY))
GST_CAT_ERROR_OBJECT (GST_CAT_CLOCK, clock,
"unexpected status %d for entry %p", status, entry);
GST_CLOCK_ENTRY_STATUS (entry) = GST_CLOCK_DONE;
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock,
"entry %p unlocked, status %d", entry, status);
if (G_UNLIKELY (status == GST_CLOCK_UNSCHEDULED)) {
goto done;
} else {
if (waitret) {
/* some other id got unlocked */
if (!restart) {
/* this can happen if the entry got unlocked because of an async
* entry was added to the head of the async queue. */
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock,
"wakeup waiting for entry %p", entry);
goto done;
}
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock,
"entry %p needs to be restarted", entry);
} else {
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock,
"entry %p unlocked after timeout", entry);
}
/* reschedule if gst_cond_wait_until returned early or we have to reschedule after
* an unlock*/
mono_ts = g_get_monotonic_time ();
now = gst_clock_get_time (clock);
diff = GST_CLOCK_DIFF (now, entryt);
if (diff <= CLOCK_MIN_WAIT_TIME) {
/* timeout, this is fine, we can report success now */
GST_CLOCK_ENTRY_STATUS (entry) = status = GST_CLOCK_OK;
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock,
"entry %p finished, diff %" G_GINT64_FORMAT, entry, diff);
#ifdef WAIT_DEBUGGING
final = gst_system_clock_get_internal_time (clock);
GST_CAT_DEBUG (GST_CAT_CLOCK, "Waited for %" G_GINT64_FORMAT
" got %" G_GINT64_FORMAT " diff %" G_GINT64_FORMAT
" %g target-offset %" G_GINT64_FORMAT " %g", entryt, now,
now - entryt,
(double) (GstClockTimeDiff) (now - entryt) / GST_SECOND,
(final - target),
((double) (GstClockTimeDiff) (final - target)) / GST_SECOND);
#endif
goto done;
} else {
GST_CAT_DEBUG_OBJECT (GST_CAT_CLOCK, clock,
"entry %p restart, diff %" G_GINT64_FORMAT, entry, diff);
/* we are going to poll again, set status back to busy */
GST_CLOCK_ENTRY_STATUS (entry) = GST_CLOCK_BUSY;
}
}
}
} else {
/* we are right on time or too late */
if (G_UNLIKELY (diff == 0)) {
GST_CLOCK_ENTRY_STATUS (entry) = status = GST_CLOCK_OK;
} else {
GST_CLOCK_ENTRY_STATUS (entry) = status = GST_CLOCK_EARLY;
}
}
done:
return status;
}
gst_base_sink_do_sync函数执行完毕后会调用render函数来render frame，gst_base_sink_do_sync主要是等待clock time 到到达buffer的 sync time（或者说clock的running_time到达buffer的running_time），以此来实现stream的同步。
6 参考
1 Clocks and synchronization in GStreamer
2 Synchronisation (gstreamer.freedesktop.org)