神经网络regularization技巧 dropbock

本博客主要解释game公式
详细的论文内容解析见
https://blog.csdn.net/qq_14845119/article/details/85103503
注意其中的maxpool操作，实现周围的变成1 不知道代码是否是这么实现的！

大纲

dropblock：A regularization method for convolutional networks
Abstract
Deep neural networks often work well when they are over-parameterized and
trained with a massive amount of noise and regularization, such as weight decay
and dropout. Although dropout is widely used as a regularization technique for
fully connected layers, it is often less effective for convolutional layers. This
lack of success of dropout for convolutional layers is perhaps due to the fact that
activation units in convolutional layers are spatially correlated so information
can still flow through convolutional networks despite dropout. Thus a structured
form of dropout is needed to regularize convolutional networks. In this paper, we
introduce DropBlock, a form of structured dropout, where units in a contiguous
region of a feature map are dropped together. We found that applying DropbBlock
in skip connections in addition to the convolution layers increases the accuracy.
Also, gradually increasing number of dropped units during training leads to better
accuracy and more robust to hyperparameter choices. Extensive experiments
show that DropBlock works better than dropout in regularizing convolutional
networks. On ImageNet classification, ResNet-50 architecture with DropBlock
achieves 78:13% accuracy, which is more than 1:6% improvement on the baseline.
On COCO detection, DropBlock improves Average Precision of RetinaNet from
36:8% to 38:4%.

1 introduction
2related work
3dropblock
类似于原本的dropout
相邻的区域被dropout掉
setting the value of block_size
setting the value of gama
这个就是用来控制drop的比例的
scheduled dropblock
4experiments
4.1imagenet classification
implementation details
4.1.1 dropblock in resnet-50
where to apply dropblock
dropblock vs dropout
dropblock vs spatialdropout
comparison with droppath
comparison with cutout
comparison with other regularization techniques
4.1.2 dropblock in amoebatnet
4.2 experimental analysis
dropblock drops more semantic information
model trained with dropblock is more robust
dropblock learns spatial distributed representations
4.3 object detection in COCO
where to apply dropblock to retinanet model
training object detector from random intialization
impementation details
4.4 semantic segmentation in PASCAL VOC

5 discussion
In this work, we introduce DropBlock to regularize training CNNs. DropBlock is a form of structured
dropout that drops spatially correlated information. We demonstrate DropBlock is a more effective
regularizer compared to dropout in ImageNet classification and COCO detection. DropBlock consistently
outperforms dropout in an extensive experiment setup. We conduct an analysis to show that
model trained with DropBlock is more robust and has the benefits of model trained with dropout.
The class activation mapping suggests the model can learn more spatially distributed representations
regularized by DropBlock.
Our experiments show that applying DropBlock in skip connections in addition to the convolution
layers increases the accuracy. Also, gradually increasing number of dropped units during training
leads to better accuracy and more robust to hyperparameter choices.

补充理解

发现

更多

既然dropblock有，两个超参数，因此可以用神经网络搜索！来实现进一步的改进！
毕竟文中实验的只是几种参数的选择！

但是 可以料想的是 使用神经玩过搜索得到更优的参数，改进也不会太大！