keras activity_regularizer含义及作用
In Keras, there are now three types of regularizers for a layer: kernel_regularizer, bias_regularizer, activity_regularizer.
When to use which?
Usually, if you have no prior on the distribution that you wish to model, you would only use the kernel regularizer, since a large enough network can still model your function even if the regularization on the weights are big.
If you want the output function to pass through (or have an intercept closer to) the origin, you can use the bias regularizer.
If you want the output to be smaller (or closer to 0), you can use the activity regularizer.
L1 versus L2 regularization
Now, for the L1 versus L2 loss for weight decay (not to be confused with the outputs loss function).
- L2 loss is defined as w2
- L1 loss is defined as |w|.
where ww is a component of the matrix W.
- The gradient of L2 will be: 2w
- The gradient of L1 will be: sign(w)
Thus, for each gradient update with a learning rate a, in L2 loss, the weights will be subtracted by aW, while in L1 loss they will be subtracted by a⋅sign(W).
The effect of L2 loss on the weights is a reduction of large components in the matrix W, while L1 loss will make the weights matrix sparse, with many zero values. The same applies to the bias and output respectively using the bias and activity regularizer.
activity_regularizer 参数是keras api自己提出的，作用在层输出，使用户可控制在层输出前面或者后面添加正则。
例如，
激活函数后添加：
model.add(Dense(32, activation='relu', activity_regularizer=l1(0.001)))
激活函数前添加：
model.add(Dense(32, activation='linear', activity_regularizer=l1(0.001)))
model.add(Activation('relu'))
常用于autoencoder和encoder-decoder中，学习稀疏的表示。也可用于常规网络层。
Activity regularization is specified on a layer in Keras.
The regularizer is applied to the output of the layer, but you have control over what the “output” of the layer actually means. Specifically, you have flexibility as to whether the layer output means that the regularization is applied before or after the ‘activation‘ function.
Although activity regularization is most often used to encourage sparse learned representations in autoencoder and encoder-decoder models, it can also be used directly within normal neural networks to achieve the same effect and improve the generalization of the model.
reference
neural networks - What is the difference between kernel, bias, and activity regulizers, and when to use which? - Cross Validated
https://machinelearningmastery.com/how-to-reduce-generalization-error-in-deep-neural-networks-with-activity-regularization-in-keras/