Trained Ternary Quantization

Approach

This method has very little accuracy degradation and can even improve the accuracy of some models. On ImageNet, our model outperforms full-precision AlexNet model by 0.3% of Top-1 accuracy and outperforms previous ternary models by 3%.

First, we normalize the full-precision weights to the range [-1, +1] by dividing each weight by the maximum weight. Next, we quantize the intermediate full-resolution weights to {-1, 0, +1} by thresholding. The threshold factor t is a hyper-parameter that is the same across all the layers in order to reduce the search space. Finally, we perform trained quantization by back propagating two gradients, as shown in the dashed lines in Figure 1. We back-propagate gradient 1 to the full-resolution weights and gradient 2 to the scaling coefficients. The former enables learning the ternary assignments , and the latter enables learning the ternary values.

Experiment

References:
Trained Ternary Quantization, chenzhuozhu, songhan, 2017, International Conference on Learning Representations