x264 nnz patch
Oops
Filed under: bugs,CABAC,chroma,stupidity,x264 ::
Apparently my nnz patch managed to break lossless mode and it took us nearly 2 weeks to realize it! Fortunately, I managed to fix it with this patch only 30 minutes
after it was reported and it was soon after put into the official git repository. To begin with, to understand this, one has to understand what my nnz patch did.
CAVLC entropy encoding needs to know the number of nonzero coefficients in a DCT block to choose its variable-length codes properly (among other things, it adapts
which VLC table to use based on the characteristics of the DCT block). This is a slightly intensive process; its an extra step to take on every single DCT block to
count the number of nonzero values. Its not too slow, but in CABAC its completely unnecessary, so the purpose of the patch was to move all nnz calculations to CAVLC
and in the macroblock encoding only calculate a single thing: whether the array had coefficients or not, which is very fast and trivial to calculate.
CAVLC熵编码必须知道一个DCT块中的非零系数的个数，来选择正确的变长码表（根据DCT块的特点来选择不同的码表是CAVLC编码的特性之一）。每个DCT块的编码都必须执行这一步。虽然它不
太费时间，但是对于CABAC编码，这一步却是多余的。所以这个补丁就是为了把所有的nnz（非零系数）的计算过程移到CAVLC编码中，从而使得在宏块编码中只计算宏块是否有非零系数。
But in DCT arrays of AC coefficients only, such as in i16x16 and 8×8 chroma blocks (see this post for more info), we need to ignore the first coefficient, the DC
coefficient, when doing this math. So the simplest way to do this is to just zero the DC coefficient at the same time we write the coefficients to memory so it doesn’
t cause issues later.
But I forgot about lossless mode and forgot to set the DC coefficient to zero in that case. The end result? Well, it broke!
但是在只有AC系数的DCT块中，如i16x16和8x8色度块，我们计算非零系数个数时必须忽略第一个系数，即DC系数。所以最简单的做法就是把系数写入内存的时候就把DC系数置0。
但是我在无损模式时忘了这点。结果，完蛋了！