HEVC核心编码技术之六.环路滤波，SAO及其它
Overview of the High Efficiency Video Coding(HEVC) Standard之四
K. 环内滤波
In-Loop Filters
In HEVC, two processing steps, namely a deblocking
filter (DBF) followed by an SAO filter, are applied to the
reconstructed samples before writing them into the decoded
picture buffer in the decoder loop. The DBF is intended
to reduce the blocking artifacts due to block-based coding.
在HEVC中，对要写入解码环的DPB的重建像素使用了两个处理：
先SAO滤波，再去块效应滤波；
DBF的目的是为了减少基于块编码而人工产生的块效应；
The DBF is similar to the DBF of the H.264/MPEG-4 AVC
standard, whereas SAO is newly introduced in HEVC. While
the DBF is only applied to the samples located at block
boundaries, the SAO filter is applied adaptively to all samples
satisfying certain conditions, e.g., based on gradient. During
the development of HEVC, it had also been considered to
operate a third processing step called the adaptive loop filter
(ALF) after the SAO filter; however, the ALF feature was not
included in the final design.
HEVC的DBF和H.264/MPEG-4 AVC相似，而SAO则是新引入的；
因为DBF只用于处理块边界上的像素，
而SAO滤波器可用于满足某种统计条件(如基于梯度图)的所有像素；
1) 去块效应滤波
Deblocking Filter:
The deblocking filter is applied to
all samples adjacent to a PU or TU boundary except the case
when the boundary is also a picture boundary, or when deblocking
is disabled across slice or tile boundaries (which is an
option that can be signaled by the encoder). It should be noted
that both PU and TU boundaries should be considered since
PU boundaries are not always aligned with TU boundaries in
some cases of interpicture-predicted CBs. Syntax elements in
the SPS and slice headers control whether the deblocking filter
is applied across the slice and tile boundaries.
除了当边界位于图像边界，或对跨片，或瓦片的去块效应滤波被关闭，
去块效应滤波被用于所有PU或TU边界相邻的像素；
应当注意的是，PU和TU边界都应考虑，
因为PU边界在帧间预测CB中并不是总和TU边界对齐的；
在SPS和片头中有语法元素提示去块效应滤波是否用于跨片和瓦片的边界；
Unlike H.264/MPEG-4 AVC, where the deblocking filter
is applied on a 4×4 sample grid basis, HEVC only applies
the deblocking filter to the edges that are aligned on an 8×8
sample grid, for both the luma and chroma samples. This
restriction reduces the worst-case computational complexity
without noticeable degradation of the visual quality. It also
improves parallel-processing operation by preventing cascading
interactions between nearby filtering operations.
和H.264/MPEG-4 AVC只对4x4像素块边界用去块效应滤波不同，
HEVC只对8x8像素块的亮度和色度的边界使用DBF；
这种限制减少了计算复杂度且并没有对图像质量有很大的降低；
而且能提高并处处理效率；
The strength of the deblocking filter is controlled by the
values of several syntax elements similar to the scheme in
H.264/MPEG-4 AVC, but only three strengths are used rather
than five. Given that P and Q are two adjacent blocks with a
common 8×8 grid boundary, the filter strength of 2 is assigned
when one of the blocks is intrapicture predicted. Otherwise,
the filter strength of 1 is assigned if any of the following
conditions is satisfied.
去块效率的滤波强度参数控制和H.264/MPEG-4 AVC相邻，
但参数个数减少到了3个，
计算方式如下：
1) P or Q has at least one nonzero transform coefficient.
P或Q至少有一个非零变换系数
2) The reference indices of P and Q are not equal.
P和Q的参考索引不相等
3) The motion vectors of P and Q are not equal.
P和Q的MV不相等
4) The difference between a motion vector component of
P and Q is greater than or equal to one integer sample.
P和Q的MV分量的差值大于或等于一个整像素
If none of the above conditions is met, the filter strength of 0
is assigned, which means that the deblocking process is not
applied.
如果上述的条件都不满足，则滤波强度为0，意味着不需要进行去块效应处理；
According to the filter strength and the average quantization
parameter of P and Q, two thresholds, tC and β, are determined
from predefined tables. For luma samples, one of three cases,
no filtering, strong filtering, and weak filtering, is chosen based
on β. Note that this decision is shared across four luma rows
or columns using the first and the last rows or columns to
reduce the computational complexity.
依据滤波强度，P和Q的平均量化参数，可以在预定义的表中确定两个阈值：tC and β；
对于亮度像素，基于β选择下列三种情况中的一种：
不滤波，
强滤波，
弱滤波。
并且这个选择可以跨过四个亮度行和列，在第一和最后一行或列共用，以减少计算复杂度；
There are only two cases, no filtering and normal filtering,
for chroma samples. Normal filtering is applied only when the
filter strength is greater than one. The filtering process is then
performed using the control variables tC and β.
对于色度像素，只有不滤波和正常滤波两种情况；
正常滤波只有在滤波强度大于1时使用，滤波处理使用控制变量tC and β来处理；
Fig. 10. Four gradient patterns used in SAO. Sample labeled “p” indicates
a center sample to be considered. Two samples labeled “n0” and “n1”
specify two neighboring samples along the
(a) horizontal (sao_eo_class = 0),
(b) vertical (sao_eo_class = 1),
(c) 135° diagonal (sao_eo_class = 2), and
(d) 45° (sao_eo_class = 3) gradient patterns.
In HEVC, the processing order of the deblocking filter is
defined as horizontal filtering for vertical edges for the entire
picture first, followed by vertical filtering for horizontal edges.
在HEVC中，DBF的处理顺序是
先对整个图像的垂直边进行水平滤波；
再对水平边界进行垂直滤波；
This specific order enables either multiple horizontal filtering
or vertical filtering processes to be applied in parallel threads,
or can still be implemented on a CTB-by-CTB basis with only
a small processing latency.
这种方式是很方便在并行处理架构同时进行多水平行或多垂直列的滤波；
也可以逐个CTB进行来减少时延；
2) SAO:
SAO is a process that modifies the decoded
samples by conditionally adding an offset value to each sample
after the application of the deblocking filter, based on values
in look-up tables transmitted by the encoder. SAO filtering is
performed on a region basis, based on a filtering type selected
per CTB by a syntax element sao_type_idx. A value of 0 for
sao_type_idx indicates that the SAO filter is not applied to
the CTB, and the values 1 and 2 signal the use of the band
offset and edge offset filtering types, respectively.
In the band offset mode specified by sao_type_idx equal
to 1, the selected offset value directly depends on the sample
amplitude. In this mode, the full sample amplitude range is
uniformly split into 32 segments called bands, and the sample
values belonging to four of these bands (which are consecutive
within the 32 bands) are modified by adding transmitted values
denoted as band offsets, which can be positive or negative. The
main reason for using four consecutive bands is that in the
smooth areas where banding artifacts can appear, the sample
amplitudes in a CTB tend to be concentrated in only few of
the bands. In addition, the design choice of using four offsets
is unified with the edge offset mode of operation which also
uses four offset values.
SAO是在去块效应之后，基于由编码器传输的查找表的值，
依据条件给每个解码后的像素加一个偏移值来改变像素值的处理；
SAO滤波基于每个CTB的语法元素sao_type_idx来选择滤波类型，并且基于区域执行；
sao_type_idx 等于 0时，表示不对CTB使用SAO滤波器；
sao_type_idx 等于 1时，表示使用频带偏移滤波类型，偏移值的选择依靠于像素幅值。
sao_type_idx 等于 2时，表示使用边界偏移滤波类型；
在频带偏移滤波类型中，事个像素幅度的范围被统一划分成32个段，称为频带，
如果像素值属于这些频带中的四个，则加上变换值，称为频带偏移，来修改，
这个偏移值可以是正或负；
使用四个连续频带的理由是在平滑的区域，频带痕迹就会出现，
在CTB中的像素幅值会趋向于集中在少数几个频带；
In the edge offset mode specified by sao_type_idx equal
to 2, a syntax element sao_eo_class with values from 0 to 3
signals whether a horizontal, vertical or one of two diagonal
gradient directions is used for the edge offset classification
in the CTB. Fig. 10 depicts the four gradient patterns used
for the respective sao_eo_class in this mode. Each sample in
the CTB is classified into one of five EdgeIdx categories by
comparing the sample value p located at some position with
the values n0 and n1 of two samples located at neighboring
positions as shown in Table IV. This classification is done for
each sample based on decoded sample values, so no additional
signaling is required for the EdgeIdx classification. Depending
on the EdgeIdx category at the sample position, for EdgeIdx
categories from 1 to 4, an offset value from a transmitted
look-up table is added to the sample value. The offset values
are always positive for categories 1 and 2 and negative for
categories 3 and 4 - thus the filter generally has a smoothing
effect in the edge offset mode.
sao_type_idx等于2时，使用边界偏移模式；
语法元素sao_eo_class等于0到3，表示对CTB使用水平，垂直，还是对角线方向的边界偏移；
TABLE IV
Sample EdgeIdx Categories in SAO Edge Classes
EdgeIdx Condition Meaning
0 Cases not listed below Monotonic area
1 p < n0 and p < n1 Local min
2 p < n0 and p = n1 or p < n1 and p = n0 Edge
3 p > n0 and p = n1 or p > n1 and p = n0 Edge
4 p > n0 and p > n1 Local max
Thus, for SAO types 1 and 2, a total of four amplitude
offset values are transmitted to the decoder for each CTB. For
type 1, the sign is also encoded. The offset values and related
syntax elements such as sao_type_idx and sao_eo_class are
determined by the encoder - typically using criteria that
optimize rate-distortion performance. The SAO parameters can
be indicated to be inherited from the left or above CTB using
a merge flag to make the signaling efficient. In summary,
SAO is a nonlinear filtering operation which allows additional
refinement of the reconstructed signal, and it can enhance the
signal representation in both smooth areas and around edges.
L. 特殊编码模式
Special Coding Modes
HEVC defines three special coding modes, which can be
invoked at the CU level or the TU level.
HEVC定义三种特殊编码模式，可以用在CU级和TU级；
1) In I_PCM mode,
the prediction, transform, quantization
and entropy coding are bypassed, and the samples are
directly represented by a pre-defined number of bits. Its
main purpose is to avoid excessive consumption of bits
when the signal characteristics are extremely unusual
and cannot be properly handled by hybrid coding (e.g.,
noise-like signals).
预测，变换，量化和熵编码都旁路，像素直接表示成预定义的比特数；
它主要的目的是避免当信号特性(如噪声信号)完全不能用混合编码时的情形；
2) 无损模式
In lossless mode,
the transform, quantization, and other
processing that affects the decoded picture (SAO and
deblocking filters) are bypassed, and the residual signal
from inter- or intrapicture prediction is directly fed into
the entropy coder (using the same neighborhood contexts
that would usually be applied to the quantized transform
coefficients). This allows mathematically lossless
reconstruction, which is achieved without defining any
additional coding tools.
变换，量化和其它影响解码图像(SAO和DBF)的处理都被旁路,
帧内或帧间预测的残差信号被直接送到熵编码器；
这种方式实现了算术上的无损重建；
3) In transform skipping mode,
only the transform is bypassed.
This primarily improves compression for certain
types of video content such as computer-generated images
or graphics mixed with camera-view content (e.g.,
scrolling text). This mode can be applied to TBs of 4×4
size only.
只有变换被旁路，它主要用来压缩一些特定类型的视频内容，
如计算生成的图像或有摄像角度内容的图形。
这种模式只能用在4x4变换尺寸；
SAO and deblocking filtering are not applied to lossless mode
regions, and a flag controls whether they are applied to I_PCM
regions.
SAO和DBF不能用于无损模式，
有标志位来控制它们是否能用于I_PCM模式；
用六，利永贞。
【白话】“用六”这一爻，利于永远保持中正。
《象》曰：用六“永贞”，以大终也。
【白话】《象辞》说：用六的爻辞说“利于永远保持中正”，即是指阴盛到了极点就会向阳转化。
K. 环内滤波
In-Loop Filters
In HEVC, two processing steps, namely a deblocking
filter (DBF) followed by an SAO filter, are applied to the
reconstructed samples before writing them into the decoded
picture buffer in the decoder loop. The DBF is intended
to reduce the blocking artifacts due to block-based coding.
在HEVC中，对要写入解码环的DPB的重建像素使用了两个处理：
先SAO滤波，再去块效应滤波；
DBF的目的是为了减少基于块编码而人工产生的块效应；
The DBF is similar to the DBF of the H.264/MPEG-4 AVC
standard, whereas SAO is newly introduced in HEVC. While
the DBF is only applied to the samples located at block
boundaries, the SAO filter is applied adaptively to all samples
satisfying certain conditions, e.g., based on gradient. During
the development of HEVC, it had also been considered to
operate a third processing step called the adaptive loop filter
(ALF) after the SAO filter; however, the ALF feature was not
included in the final design.
HEVC的DBF和H.264/MPEG-4 AVC相似，而SAO则是新引入的；
因为DBF只用于处理块边界上的像素，
而SAO滤波器可用于满足某种统计条件(如基于梯度图)的所有像素；
1) 去块效应滤波
Deblocking Filter:
The deblocking filter is applied to
all samples adjacent to a PU or TU boundary except the case
when the boundary is also a picture boundary, or when deblocking
is disabled across slice or tile boundaries (which is an
option that can be signaled by the encoder). It should be noted
that both PU and TU boundaries should be considered since
PU boundaries are not always aligned with TU boundaries in
some cases of interpicture-predicted CBs. Syntax elements in
the SPS and slice headers control whether the deblocking filter
is applied across the slice and tile boundaries.
除了当边界位于图像边界，或对跨片，或瓦片的去块效应滤波被关闭，
去块效应滤波被用于所有PU或TU边界相邻的像素；
应当注意的是，PU和TU边界都应考虑，
因为PU边界在帧间预测CB中并不是总和TU边界对齐的；
在SPS和片头中有语法元素提示去块效应滤波是否用于跨片和瓦片的边界；
Unlike H.264/MPEG-4 AVC, where the deblocking filter
is applied on a 4×4 sample grid basis, HEVC only applies
the deblocking filter to the edges that are aligned on an 8×8
sample grid, for both the luma and chroma samples. This
restriction reduces the worst-case computational complexity
without noticeable degradation of the visual quality. It also
improves parallel-processing operation by preventing cascading
interactions between nearby filtering operations.
和H.264/MPEG-4 AVC只对4x4像素块边界用去块效应滤波不同，
HEVC只对8x8像素块的亮度和色度的边界使用DBF；
这种限制减少了计算复杂度且并没有对图像质量有很大的降低；
而且能提高并处处理效率；
The strength of the deblocking filter is controlled by the
values of several syntax elements similar to the scheme in
H.264/MPEG-4 AVC, but only three strengths are used rather
than five. Given that P and Q are two adjacent blocks with a
common 8×8 grid boundary, the filter strength of 2 is assigned
when one of the blocks is intrapicture predicted. Otherwise,
the filter strength of 1 is assigned if any of the following
conditions is satisfied.
去块效率的滤波强度参数控制和H.264/MPEG-4 AVC相邻，
但参数个数减少到了3个，
计算方式如下：
1) P or Q has at least one nonzero transform coefficient.
P或Q至少有一个非零变换系数
2) The reference indices of P and Q are not equal.
P和Q的参考索引不相等
3) The motion vectors of P and Q are not equal.
P和Q的MV不相等
4) The difference between a motion vector component of
P and Q is greater than or equal to one integer sample.
P和Q的MV分量的差值大于或等于一个整像素
If none of the above conditions is met, the filter strength of 0
is assigned, which means that the deblocking process is not
applied.
如果上述的条件都不满足，则滤波强度为0，意味着不需要进行去块效应处理；
According to the filter strength and the average quantization
parameter of P and Q, two thresholds, tC and β, are determined
from predefined tables. For luma samples, one of three cases,
no filtering, strong filtering, and weak filtering, is chosen based
on β. Note that this decision is shared across four luma rows
or columns using the first and the last rows or columns to
reduce the computational complexity.
依据滤波强度，P和Q的平均量化参数，可以在预定义的表中确定两个阈值：tC and β；
对于亮度像素，基于β选择下列三种情况中的一种：
不滤波，
强滤波，
弱滤波。
并且这个选择可以跨过四个亮度行和列，在第一和最后一行或列共用，以减少计算复杂度；
There are only two cases, no filtering and normal filtering,
for chroma samples. Normal filtering is applied only when the
filter strength is greater than one. The filtering process is then
performed using the control variables tC and β.
对于色度像素，只有不滤波和正常滤波两种情况；
正常滤波只有在滤波强度大于1时使用，滤波处理使用控制变量tC and β来处理；
Fig. 10. Four gradient patterns used in SAO. Sample labeled “p” indicates
a center sample to be considered. Two samples labeled “n0” and “n1”
specify two neighboring samples along the
(a) horizontal (sao_eo_class = 0),
(b) vertical (sao_eo_class = 1),
(c) 135° diagonal (sao_eo_class = 2), and
(d) 45° (sao_eo_class = 3) gradient patterns.
In HEVC, the processing order of the deblocking filter is
defined as horizontal filtering for vertical edges for the entire
picture first, followed by vertical filtering for horizontal edges.
在HEVC中，DBF的处理顺序是
先对整个图像的垂直边进行水平滤波；
再对水平边界进行垂直滤波；
This specific order enables either multiple horizontal filtering
or vertical filtering processes to be applied in parallel threads,
or can still be implemented on a CTB-by-CTB basis with only
a small processing latency.
这种方式是很方便在并行处理架构同时进行多水平行或多垂直列的滤波；
也可以逐个CTB进行来减少时延；
2) SAO:
SAO is a process that modifies the decoded
samples by conditionally adding an offset value to each sample
after the application of the deblocking filter, based on values
in look-up tables transmitted by the encoder. SAO filtering is
performed on a region basis, based on a filtering type selected
per CTB by a syntax element sao_type_idx. A value of 0 for
sao_type_idx indicates that the SAO filter is not applied to
the CTB, and the values 1 and 2 signal the use of the band
offset and edge offset filtering types, respectively.
In the band offset mode specified by sao_type_idx equal
to 1, the selected offset value directly depends on the sample
amplitude. In this mode, the full sample amplitude range is
uniformly split into 32 segments called bands, and the sample
values belonging to four of these bands (which are consecutive
within the 32 bands) are modified by adding transmitted values
denoted as band offsets, which can be positive or negative. The
main reason for using four consecutive bands is that in the
smooth areas where banding artifacts can appear, the sample
amplitudes in a CTB tend to be concentrated in only few of
the bands. In addition, the design choice of using four offsets
is unified with the edge offset mode of operation which also
uses four offset values.
SAO是在去块效应之后，基于由编码器传输的查找表的值，
依据条件给每个解码后的像素加一个偏移值来改变像素值的处理；
SAO滤波基于每个CTB的语法元素sao_type_idx来选择滤波类型，并且基于区域执行；
sao_type_idx 等于 0时，表示不对CTB使用SAO滤波器；
sao_type_idx 等于 1时，表示使用频带偏移滤波类型，偏移值的选择依靠于像素幅值。
sao_type_idx 等于 2时，表示使用边界偏移滤波类型；
在频带偏移滤波类型中，事个像素幅度的范围被统一划分成32个段，称为频带，
如果像素值属于这些频带中的四个，则加上变换值，称为频带偏移，来修改，
这个偏移值可以是正或负；
使用四个连续频带的理由是在平滑的区域，频带痕迹就会出现，
在CTB中的像素幅值会趋向于集中在少数几个频带；
In the edge offset mode specified by sao_type_idx equal
to 2, a syntax element sao_eo_class with values from 0 to 3
signals whether a horizontal, vertical or one of two diagonal
gradient directions is used for the edge offset classification
in the CTB. Fig. 10 depicts the four gradient patterns used
for the respective sao_eo_class in this mode. Each sample in
the CTB is classified into one of five EdgeIdx categories by
comparing the sample value p located at some position with
the values n0 and n1 of two samples located at neighboring
positions as shown in Table IV. This classification is done for
each sample based on decoded sample values, so no additional
signaling is required for the EdgeIdx classification. Depending
on the EdgeIdx category at the sample position, for EdgeIdx
categories from 1 to 4, an offset value from a transmitted
look-up table is added to the sample value. The offset values
are always positive for categories 1 and 2 and negative for
categories 3 and 4 - thus the filter generally has a smoothing
effect in the edge offset mode.
sao_type_idx等于2时，使用边界偏移模式；
语法元素sao_eo_class等于0到3，表示对CTB使用水平，垂直，还是对角线方向的边界偏移；
TABLE IV
Sample EdgeIdx Categories in SAO Edge Classes
EdgeIdx Condition Meaning
0 Cases not listed below Monotonic area
1 p < n0 and p < n1 Local min
2 p < n0 and p = n1 or p < n1 and p = n0 Edge
3 p > n0 and p = n1 or p > n1 and p = n0 Edge
4 p > n0 and p > n1 Local max
Thus, for SAO types 1 and 2, a total of four amplitude
offset values are transmitted to the decoder for each CTB. For
type 1, the sign is also encoded. The offset values and related
syntax elements such as sao_type_idx and sao_eo_class are
determined by the encoder - typically using criteria that
optimize rate-distortion performance. The SAO parameters can
be indicated to be inherited from the left or above CTB using
a merge flag to make the signaling efficient. In summary,
SAO is a nonlinear filtering operation which allows additional
refinement of the reconstructed signal, and it can enhance the
signal representation in both smooth areas and around edges.
L. 特殊编码模式
Special Coding Modes
HEVC defines three special coding modes, which can be
invoked at the CU level or the TU level.
HEVC定义三种特殊编码模式，可以用在CU级和TU级；
1) In I_PCM mode,
the prediction, transform, quantization
and entropy coding are bypassed, and the samples are
directly represented by a pre-defined number of bits. Its
main purpose is to avoid excessive consumption of bits
when the signal characteristics are extremely unusual
and cannot be properly handled by hybrid coding (e.g.,
noise-like signals).
预测，变换，量化和熵编码都旁路，像素直接表示成预定义的比特数；
它主要的目的是避免当信号特性(如噪声信号)完全不能用混合编码时的情形；
2) 无损模式
In lossless mode,
the transform, quantization, and other
processing that affects the decoded picture (SAO and
deblocking filters) are bypassed, and the residual signal
from inter- or intrapicture prediction is directly fed into
the entropy coder (using the same neighborhood contexts
that would usually be applied to the quantized transform
coefficients). This allows mathematically lossless
reconstruction, which is achieved without defining any
additional coding tools.
变换，量化和其它影响解码图像(SAO和DBF)的处理都被旁路,
帧内或帧间预测的残差信号被直接送到熵编码器；
这种方式实现了算术上的无损重建；
3) In transform skipping mode,
only the transform is bypassed.
This primarily improves compression for certain
types of video content such as computer-generated images
or graphics mixed with camera-view content (e.g.,
scrolling text). This mode can be applied to TBs of 4×4
size only.
只有变换被旁路，它主要用来压缩一些特定类型的视频内容，
如计算生成的图像或有摄像角度内容的图形。
这种模式只能用在4x4变换尺寸；
SAO and deblocking filtering are not applied to lossless mode
regions, and a flag controls whether they are applied to I_PCM
regions.
SAO和DBF不能用于无损模式，
有标志位来控制它们是否能用于I_PCM模式；
用六，利永贞。
【白话】“用六”这一爻，利于永远保持中正。
《象》曰：用六“永贞”，以大终也。
【白话】《象辞》说：用六的爻辞说“利于永远保持中正”，即是指阴盛到了极点就会向阳转化。