ImageNet Classification with Deep ConvolutionalNeural Networks

AlexNet网络实现：https://blog.csdn.net/weixin_43912621/article/details/127757396

ImageNet Classification with Deep ConvolutionalNeural Networks

Abstract

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.

这篇文章描述了一个成功的大型卷积神经网络在ImageNet LSVRC-2010比赛中的训练过程，取得了top-1和top-5错误率分别为37.5％和17.0％的成绩。该神经网络具有6000万个参数和65万个神经元，包括五个卷积层，一些后面跟着最大池化层，以及三个全连接层，最后使用1000-way softmax进行分类。为了加快训练速度，使用了非饱和神经元和卷积的GPU实现。为了减少全连接层中的过拟合，作者采用了一种名为“dropout”的技术。

1 Introduction

Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the currentbest error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4].But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al [21]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.

目前物体识别的方法基本上都使用机器学习方法。为了提高它们的性能，我们可以收集更大的数据集、学习更强大的模型，并使用更好的技术来防止过拟合。直到最近，标记图像的数据集相对较小，通常只有数万张图像（例如，NORB [16]、Caltech-101/256 [8, 9]和CIFAR-10/100 [12]）。使用这样大小的数据集可以很好地解决简单的识别任务，特别是如果它们使用保留标签的转换进行增强。例如，MNIST数字识别任务的当前最佳错误率（<0.3％）接近人类表现[4]。但是，在现实场景中，物体展现出相当大的变异性，因此为了学习识别它们，必须使用更大的训练集。实际上，小型图像数据集的缺点已经得到了广泛认识（例如，Pinto等人[21]），但直到最近才有可能收集到数百万张标记的数据集。新的大型数据集包括LabelMe [23]，它包含数十万个完全分割的图像，以及ImageNet [6]，它包含超过1500万个标记的高分辨率图像，涵盖了22000多个类别。

To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies).Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

为了从数百万张图像中学习成千上万种对象，我们需要一个具有大学习能力的模型。然而，对象识别任务的巨大复杂性意味着即使是像ImageNet这样大型的数据集也无法完全涵盖所有情况，因此我们的模型还应该具备许多先前的知识，以弥补我们没有的所有数据。卷积神经网络（CNN）就是这样一类模型 [16, 11, 13, 18, 15, 22, 26]。它们的容量可以通过改变其深度和宽度来控制，并且它们对图像的本质做出了强大而且大多数正确的假设（即统计的静态性和像素依赖性的局部性）。因此，与类似大小的标准前馈神经网络相比，CNN具有更少的连接和参数，因此更易于训练，而它们的理论最佳性能可能只略微劣于标准前馈神经网络。

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.

尽管CNN具有吸引人的特点，并且尽管它们的本地架构相对高效，但将它们大规模应用于高分辨率图像仍然代价高昂。幸运的是，当前的GPU与高度优化的2D卷积实现配对，足以促进训练足够大的CNN，并且最近的数据集（如ImageNet）包含足够的标记示例，以在不严重过拟合的情况下训练此类模型。

The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1 . Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance.

本文的具体贡献如下：我们在ILSVRC-2010和ILSVRC-2012比赛中使用ImageNet子集之一，训练了迄今为止最大的卷积神经网络之一，并在这些数据集上取得了迄今为止最好的结果。我们编写了高度优化的GPU 2D卷积实现以及训练卷积神经网络中所有其他操作，这些操作我们公开提供。我们的网络包含许多新的和不寻常的特性，可以提高其性能并缩短训练时间，在第3节中有详细说明。即使有120万个标记训练示例，我们的网络规模仍然导致过拟合成为一个重要问题，因此我们使用了几种有效的技术来防止过拟合，这些技术在第4节中描述。我们的最终网络包含五个卷积层和三个全连接层，这种深度似乎很重要：我们发现删除任何卷积层（每个卷积层包含不超过模型参数的1％）都会导致较差的性能。

In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

最后，网络的大小主要受到当前GPU上可用内存的限制和我们愿意忍受的训练时间的限制。我们的网络需要在两个GTX 580 3GB GPU上训练五到六天。所有实验都表明，我们的结果可以通过等待更快的GPU和更大的数据集变得更好。

2 The Dataset

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.

ImageNet是一个包含超过1500万张高分辨率图像的数据集，涵盖了大约22,000个类别。这些图像是从网络上收集而来，并使用Amazon的Mechanical Turk众包工具由人类标注者进行了标注。从2010年开始，作为Pascal视觉对象挑战的一部分，每年都会举行一个名为ImageNet大规模视觉识别挑战（ILSVRC）的竞赛。ILSVRC使用ImageNet的一个子集，每个子集包含大约1000个类别中的1000张图像。总共有大约120万个训练图像、5万个验证图像和15万个测试图像。

ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.

ILSVRC-2010是唯一公开测试集标签的ILSVRC版本，因此我们主要在这个版本上进行了大部分的实验。由于我们还在ILSVRC-2012竞赛中提交了我们的模型，因此在第6节中我们也会报告我们在该数据集版本上的结果，但该版本的测试集标签是不公开的。在ImageNet上，通常会报告两个错误率：top-1和top-5，其中top-5错误率是模型最有可能的五个标签中不包括正确标签的测试图像的比例。

ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256 × 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

ImageNet数据集包含尺寸不同的图片，但是我们的系统需要输入具有固定维度。因此，我们将图片缩小到了一个固定的分辨率256 x 256。对于一个矩形图片，我们首先将其缩放到短边长度为256，然后从结果图像中裁剪出中心的256 x 256的区域。除了从每个像素中减去训练集的平均活动值之外，我们没有以其他方式预处理图像。因此，我们的网络是在（中心化的）原始RGB像素值上进行训练的。

3 The Architecture

The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.

我们的网络结构如图2所示。它包含了8个可学习的层，其中有5个卷积层和3个全连接层。下面，我们将描述我们网络结构中一些新颖或不同寻常的特点。第3.1-3.4节按照我们估计的重要性排序，其中最重要的排在最前面。

3.1 ReLU Nonlinearity

The standard way to model a neuron’s output f as a function of its input x is with f(x) = tanh(x) or f(x) = (1 + e −x ) −1 . In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0, x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.

传统上，建模神经元的输出 f 作为其输入 x 的函数的标准方式是 f(x) = tanh(x) 或 f(x) = (1 + e^−x)−1。在使用梯度下降进行训练时，这些饱和非线性要比非饱和非线性 f(x) = max(0, x) 慢得多。根据 Nair 和 Hinton [20] 的说法，具有这种非线性的神经元被称为修正线性单元 (ReLU)。使用 ReLU 的深度卷积神经网络训练速度比使用 tanh 单元的网络快几倍。这在图1中得到了证明，图中显示了在 CIFAR-10 数据集上达到25%训练错误所需的迭代次数，对于一个特定的四层卷积网络。这个图表显示，如果我们使用传统的饱和神经元模型，我们将无法对这样的大型神经网络进行实验。

We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al [11] claim that the nonlinearity f(x) = |tanh(x)| works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.

这并不是我们第一个考虑在CNN中采用传统神经元模型的替代方法。例如，Jarrett等人声称，非线性函数f（x）= |tanh（x）| 与他们在Caltech-101数据集上的对比度归一化后跟随局部平均池化的类型非常搭配。但是，在这个数据集上，主要关注的是防止过度拟合，所以他们观察到的效果与我们使用ReLUs时报告的加速适应训练集的能力是不同的。更快的学习对于在大型数据集上训练的大型模型的性能有很大的影响。

3.2 Training on Multiple GPUs

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation

一个GTX 580 GPU只有3GB的内存，这限制了可以在其上训练的网络的最大大小。事实证明，120万个训练样本足以训练网络，这些网络过大无法适应单个GPU。因此，我们将网络分布在两个GPU上。当前的GPU非常适合跨GPU并行处理，因为它们能够直接读取和写入彼此的内存，而不必通过主机机器的内存。我们采用的并行化方案基本上是将每个GPU的核（或神经元）的一半放在其中，还有一个额外的技巧：GPU只在特定层中进行通信。这意味着，例如，第三层的核将从第二层的所有核映射中获取输入。但是，第四层的核仅从属于同一GPU的第三层的那些核映射中获取输入。选择连接模式是交叉验证的一个问题，但这使我们能够精确调整通信量，直到它成为计算量的可接受部分。

The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cire¸san et al [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net².

我们所得到的架构与Cire¸san等人[5]所采用的“柱状”CNN有些相似，只是我们的列不是独立的（见图2）。这种方案将我们的top-1和top-5错误率分别降低了1.7%和1.2%，与一个在一个GPU上训练的每个卷积层具有一半内核的网络相比。两个GPU的网络训练所需时间略少于一个GPU的网络。

3.3 Local Response Normalization

ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by aⁱ_x,y the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity bⁱ_x,y is given by the expression

where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k, n, α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10⁻⁴ , and β = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).

ReLU（整流线性单元）具有一种理想的特性，即它们不需要输入归一化来防止它们饱和。如果至少有一些训练样本产生正的输入到 ReLU，那么该神经元将会进行学习。然而，我们仍然发现以下本地归一化方案有助于泛化。设 aⁱ_x,y 表示通过应用位于位置 (x,y) 的核 i 并应用 ReLU 非线性函数计算的神经元的活动，那么响应归一化的活动 bⁱ_x,y 由该式给出，其中总和在相同空间位置的 n 个“相邻”核映射上运行，N 是层中总核的数量。核映射的排序当然是任意的，并且在训练开始之前确定。这种响应归一化实现了一种受真实神经元类型启发的侧向抑制形式，使不同核计算的神经元输出之间产生大活动的竞争。常数 k、n、α 和 β 是超参数，其值是使用验证集确定的。我们使用了 k = 2、n = 5、α = 10⁻⁴ 和 β = 0.75。我们在某些层应用了这种归一化（参见第 3.5 节）。

This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization.

这个方案与Jarrett等人的局部对比度归一化方案有些相似之处，但我们的更正确地称为“亮度归一化”，因为我们没有减去平均活动值。响应归一化使我们的top-1和top-5错误率分别降低了1.4％和1.2％。我们还在CIFAR-10数据集上验证了这种方案的有效性：一个四层的CNN，在没有归一化的情况下达到13％的测试错误率，在归一化的情况下降至11％。

3.4 Overlapping Pooling

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z × z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

在卷积神经网络中，池化层总结了同一卷积核图中相邻神经元组的输出。传统上，由相邻池化单元总结的邻域不重叠(例如[17, 11, 4])。更准确地说，可以将池化层视为由一组间隔为s个像素的池化单元构成的网格，每个池化单元总结一个以池化单元位置为中心的大小为z×z的邻域。如果设置s = z，则获得常用于CNN的传统局部池化。如果设置s < z，则获得重叠池化。我们在整个网络中都使用这种方案，其中s = 2，z = 3。与产生等效尺寸输出的不重叠方案s = 2，z = 2相比，这种方案将top-1和top-5错误率分别降低了0.4％和0.3％。我们通常观察到，在训练过程中，具有重叠池化的模型更难过拟合。

3.5 Overall Architecture

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fullyconnected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.

现在我们准备描述我们的CNN的总体架构。如图2所示，该网络包含8层权重，前5层是卷积层，剩下的3层是全连接层。最后一个全连接层的输出被送入一个1000路softmax，产生一个分布，覆盖了1000个类标签。我们的网络最大化多项逻辑回归目标，这相当于在预测分布下最大化正确标签的对数概率在训练案例中的平均值。

The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fullyconnected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.

第二、第四和第五个卷积层的卷积核仅连接到前一层中与其位于同一GPU上的卷积核图（见图2）。第三个卷积层的卷积核连接到第二层中的所有卷积核图。全连接层中的神经元连接到前一层中的所有神经元。响应归一化层跟随第一和第二个卷积层。最大池化层（在第3.4节中描述）跟随响应归一化层以及第五个卷积层。ReLU非线性函数应用于每个卷积层和全连接层的输出。

The first convolutional layer filters the 224×224×3 input image with 96 kernels of size 11×11×3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 × 5 × 48.The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 × 3 × 192 , and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192. The fully-connected layers have 4096 neurons each.

第一个卷积层使用96个大小为11×11×3的卷积核，步长为4个像素（这是卷积核映射中相邻神经元的感受野中心之间的距离），对224×224×3的输入图像进行滤波。第二个卷积层的输入为第一个卷积层的（响应归一化和池化）输出，并使用256个大小为5×5×48的卷积核进行滤波。第三、第四和第五个卷积层相互连接，没有任何中间的池化或归一化层。第三个卷积层具有384个大小为3×3×256的卷积核，连接到第二个卷积层的（归一化，池化）输出。第四个卷积层有384个大小为3×3×192的卷积核，第五个卷积层有256个大小为3×3×192的卷积核。全连接层每个有4096个神经元。

4 Reducing Overfitting

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting

我们的神经网络架构具有6000万个参数。尽管ILSVRC的1000个类别使每个训练示例对从图像到标签的映射施加了10个位的约束，但这证明在没有相当严重的过拟合的情况下无法学习如此多的参数。下面，我们描述了我们应对过拟合的两种主要方式。

4.1 Data Augmentation

The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk.In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.

在图像数据上减少过拟合最简单和最常见的方法是使用保留标签的转换来人为地扩大数据集（例如，[25, 4, 5]）。我们使用两种不同形式的数据增强，两种方法都允许从原始图像产生经过变换的图像，而几乎不需要进行计算，因此这些变换后的图像无需存储在磁盘上。在我们的实现中，变换后的图像是在CPU上的Python代码中生成的，而GPU则在处理前一个图像批次的训练。因此，这些数据增强方案实际上是计算上免费的。

The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the 256×256 images and training our network on these extracted patches4 . This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.

第一种形式的数据增强包括生成图像平移和水平翻转。我们通过从256×256图像中提取随机的224×224块（及其水平反转）来实现此目的，并在这些提取的块上训练我们的网络4。这将训练集的大小增加了2048倍，但所得到的训练样本当然高度相互依赖。如果不使用此方案，我们的网络将遭受严重的过拟合，这将迫使我们使用更小的网络。在测试时，网络通过提取五个224×224块（四个角落块和中心块）以及它们的水平反转（因此总共有十个块），并对网络的softmax层在这十个块上进行的预测进行平均来进行预测。

The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components,with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy = [I R xy, IG xy, IB xy] T we add the following quantity:

where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over 1%.

第二种数据增强方法是修改训练图像中 RGB 通道的强度。具体来说，我们对整个 ImageNet 训练集中的 RGB 像素值执行 PCA。对于每个训练图像，我们添加找到的主成分的倍数，其大小与相应的特征值成比例，乘以从均值为零，标准差为0.1的高斯分布中绘制的随机变量。因此，对于每个 RGB 图像像素 I_xy = [I^R_xy，I^G_xy，I^B_xy] T，我们添加以下数量：

其中p_i和λ_i分别是 RGB 像素值的 3 × 3 协方差矩阵的第 i 个特征向量和特征值，α_i是上述随机变量。对于每个特定的训练图像，每个α_i仅绘制一次，直到该图像再次用于训练，然后重新绘制。该方案大致捕捉了自然图像的一个重要属性，即物体的身份对照明强度和颜色的变化是不变的。该方案将 top-1 错误率降低了1%以上。

4.2 Dropout

Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.

将许多不同模型的预测结合起来是减少测试误差的非常成功的方法[1, 3]，但对于已经需要数天训练的大型神经网络来说，这似乎太昂贵了。然而，有一种非常高效的模型组合版本，在训练期间只需要大约两倍的成本。最近介绍的技术称为“dropout”[10]，它包括将每个隐藏神经元的输出以0.5的概率设置为零。以这种方式“dropout”的神经元不参与前向传递，也不参与反向传播。因此，每次输入被呈现时，神经网络会随机采样不同的体系结构，但所有这些体系结构共享权重。该技术减少了神经元的复杂协同作用，因为神经元不能依赖于特定其他神经元的存在。因此，它被迫学习更加稳健的特征，这些特征与其他神经元的许多不同随机子集结合起来是有用的。在测试时，我们使用所有神经元，但将它们的输出乘以0.5，这是对由指数多个dropout网络产生的预测分布的几何平均值的合理近似。

We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.

我们在图2的前两个全连接层中使用了dropout。如果没有dropout，我们的网络会出现明显的过拟合。使用dropout后，大致需要加倍的迭代次数才能收敛。

5 Details of learning

We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model’s training error. The update rule for weight w was

where i is the iteration index, v is the momentum variable, is the learning rate, and D ∂L ∂w wi E Di is the average over the ith batch Di of the derivative of the objective with respect to w, evaluated at wi

Alexnet网络亮点

1.深度：AlexNet使用了8层神经网络，其中包括5层卷积层和3层全连接层。这是当时最深的神经网络之一，因此引起了巨大的轰动。

2.重叠池化：在CNN中使用重叠的最大池化。此前CNN中普遍使用平均池化，AlexNet全部使用最大池化，避免平均池化的模糊化效果。并且AlexNet中提出让步长比池化核的尺寸小，这样池化层的输出之间会有重叠和覆盖，提升了特征的丰富性

3.非线性激活函数：AlexNet使用ReLU（修正线性单元）作为其激活函数，这是一种非线性函数，可以提高网络的非线性表达能力。相比于传统的Sigmoid和Tanh函数，ReLU的收敛速度更快，训练时间更短。

4.Dropout：AlexNet使用Dropout技术来减轻过拟合问题，这种技术会随机关闭一些神经元，这样可以防止网络对某些特定的特征过度拟合。

5.LRN：提出了LRN层，对局部神经元的活动创建竞争机制，使得其中响应比较大的值变得相对更大，并抑制其他反馈较小的神经元，增强了模型的泛化能力。。

6.并行计算：AlexNet使用了两个GPU进行并行计算，这使得网络训练的速度得到了大幅提升，同时也为后来的神经网络训练提供了一个范本。