ImageNet Classification with Deep Convolutional Neural Networks
使用深度卷积神经网络对图像进行分类
Alex Krizhevsky University of Toronto [email protected]
Alex Krizhevsky 多伦多大学
Ilya Sutskever University of Toronto [email protected]
Alex Krizhevsky 多伦多大学
GeoffreyE. Hinton University of Toronto [email protected]
GeoffreyE. Hinton 多伦多大学（AI教父）
Abstract
摘要
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
我们训练了一个大而深的卷积神经网络来对120万张高分辨率图片进行1000分类，这些图片来自ImageNet LSVRC-2010竞赛。在测试集中，我们的top-1和top-5错误率分别到达了37.5%和17.0%，这无疑比之前最先进的技术还要好，该神经网络有六千万个参数和65万个神经元，包含着5个卷积层，3个全连接层。其中有些卷积层后面跟着最大池化层，三个全连接层之后是使用softmax函数进行1000分类。为了加速训练，我们使用了不饱和神经元和非常高效的GPU进行卷积运算。我们使用了最近开发的正则化方法------dropput来防止在全连接层过拟合，这起到了很好的效果.我们也使用了该模型的变种参加了ILSVRC-2012竞赛，相比于第二名的26.2%的top-5错误率，该模型top-5错误率达到了15.3%。
1、 Introduction
Current approaches to object recognition make essential use of machine learning methods. To im- prove their performance, we can collect larger datasets, learn more powerful models, and use bet- ter techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current- best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to col- lect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.
1、介绍
当下，进行图像分类的手段一般是充分利用机器学习方法。为了提升图像分类的准确率，我们使用了更大数量的训练集、更有效的模型、以及使用更先进的技术来防止过拟合。直到最近，带标签的数据集仍然相对较小，大约成千上万张图片，比如---NORB数据集，Caltech数据集，CIFAR10/100数据集。使用这些数据量的数据集，一些简单的识别任务会被很好的解决，尤其是他们图像转换等标签不变的数据增强方法。例如，在手写数字识别任务中，当下最好的错误率（<0.3%）达到了人类的表现水平。但是在现实世界中存在的对象有着可变性，如果我们想要识别它，就必须使用更大数量的训练集。因此，小容量数据集的缺点就我们所认知了。（就像Pinto et al数据集）但是，直到现在，采取数百万张带标签的数据集才成为可能。这些新的带标签的数据集包括Labelme，它包含了数十万张完全分割的照片；ImageNet，它包含了1500万张带标签的图片，这些图片被分为22000个类别。
注释：NORB ----3D物体识别数据集
To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.
要在数百万张数据中学习一千多分类的，我们需要一个有着很强的学习能力的模型。然而这个无比复杂的图像分类任务意味着即使在ImageNet这样大的数据集中也无法得到很好的解决，因此我们的模型需要有很多的先验知识来弥补我们没有的数据。卷积神经网络（CNNs）就构建了这样的一个模型[16, 11, 13, 18, 15, 22, 26]。可以改变网络的广度和深度来控制模型的学习能力。网络可以对图像的本质做出强大而又正确的判别（即统计的稳定性和像素位置的依赖性）。因此，相比于相似大小的标准前馈神经网络，CNNs有着更少的连接和参数量，因此他很容易训练，尽管他们理论上的最佳表现可能稍微差一点。
Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.
尽管CNNs有很吸引人的特性和更加有效的局部结构，当他们应用在大规模的高分辨率图像上时，消耗的资源仍然过多。幸运的是，目前的GPU能够进行高度优化的二维卷积操作，有足够的能力促进大规模的神经网络训练。像ImageNet等最近的数据集包含着足够的标签，能够训练出不会严重过拟合的模型。
The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance.
这篇论文独特的贡献如下：
我们在应用于ILSVRC-2010和ILSVRC-2012竞赛的ImageNet数据集的子集上训练了最大的卷积神经网络之一，并取得了这些数据集上有报道以来的最好成绩。我们使用了高度优化的GPU进行二维卷积和其他固有的操作来训练卷积神经网络，这些我们都已经公开。我们的卷积神经网络包含大量新的，不同寻常的特征，这些特征提高了模型表现并且减少了训练时间，在第三节会有详细说明。即使我们使用120万个带标签的数据集来训练模型，由于我们的模型很大，因此我们将过拟合看作一个很重要的问题，我们使用了不同的有效的方法来防止过拟合，在第四节会有详细表述。我们最终的网络包含了5个卷积层和三个全连接层，这个网络深度是至关重要的：我们发现。去掉网络中的任何一个卷积层（每个模型的参数少于总参数的1%），网络的性能也会下降。
In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.
最后，该网络的大小受限于GPU的显存以及我们愿意忍受的训练时间。我们的神经网络模型花费了5-6天的时间在GTX-580 3GB的显卡上，实验表明，如果有更加快速的GPU和更大规模的数据集，模型性能可以进一步提升。
2、The Dataset
ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Ama- zon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.
2、数据集
ImageNet数据集是一个超过1500万张高分辨率，并且已经被标注好的数据集，它被分为22000个类别。数据集中的图片全部在网上搜集，通过使用了亚马逊的Mechanical Turk众包工具实现人工标注。从2010年开始，作为Pascal视觉对象挑战赛的一部分------ImageNet大规模视觉挑战赛（ILSVRC）每年都会举办一次。ILSVRC使用ImageNet数据集的子集，ImageNet数据集大约有1000个类别，每个类别大约有1000张图像。总的来说，大约有120万张训练集，50000张验证集，以及150000张测试集。
注：
训练集：用来训练的样本
验证集：模型训练过程中单独留出的样本集，用于调整模型的超参数和对模型的能力进行初步的评估，用以验证当前模型的繁华能力，以决定是否停止继续训练。
测试集：用来评估最终模型的泛化能力。但是不能作为调参、特征选择等的依据。
ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.
ILSVRC-2010是ILSVRC竞赛中唯一测试集公开的版本，因此我们的实验大多在这个版本的数据集上进行，我们也将模型应用到ILSVRC-2012竞赛中，ILSVRC-2012中的测试集标签是不可用的，在第六节我们会列出模型的表现结果。我们习惯于列出两种错误率：top-1和top-5，top5错误率是模型认为最有可能的五个标签中正确标签不在其中的测试图像的分数。
ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256 × 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.
ImageNet包含各类高分辨率的图像，我们系统的输入需要一个不变的维度。因此，我们将图像进行了下采样，将图像的像素修改为256*256.给我们一个矩形图像，我们首先将图像的短边修改为256大小的尺寸，然后将图片中心的256 * 256的区域裁剪出来。除了将图像减去训练集的图像均值（训练集和测试集都减去图像的均值），本文不做任何其他图像的处理。因此本文直接在每个像素的原始RGB值上进行训练。
3、The Architecture
The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.
本文网络结构详见图2，它包含8个学习层------5个卷积层和三个全连接层。下面，我们将描述该网络结构的一些创新和其他新的特征。3.1节和3.4节根据我们估计的重要性来对他们进行排序，第一节是最重要的。
Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.
图表2: 该卷积神经网络的图例描述，明确显示了两个GPU之间的职责。一个GPU运行图上方的层，另一个运行图下方的层。两个GPU仅在特定的层进行通信。网络的输入是150，528维的，网络剩余层中的神经元数量分别是253440，186624，64896，64896，43264，4096，4096，1000
3.1 ReLu Nonlinearity
The standard way to model a neuron’s output f as a function of its input x is with f(x) = tanh(x) or f(x) = (1 + e−x)−1. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0,x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural net- works with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional net- work. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.
通常使用使用一个关于输入x的函数模拟神经元的输出f，这种标准函数是 或者是 在梯度下降的训练时间上，这些饱和的非线性函数比不饱和的非线性函数f(x)=max(0, x)更慢，根据Nair和Hinton[20],我们将具有这三种非线性特征的神经元成为修正线性单元。使用ReLus的深度卷积神经网络训练速度比同样情况下使用tanh单元的速度快好几倍。图一表示使用特定的四层卷积神经网络在数据集CIFAR-10数据集上达到错误率为25%的迭代轮数。这个图表明如果我们使用传统的饱和神经元模型，我们不可能利用这么大的神经网络对本文工作进行实验。
We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity f (x) = |tanh(x)| works particularly well with their type of contrast nor- malization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the pri- mary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.
我们并不是首个提出在卷积神经网络（CNNs）中首次提出替换传统神经模型的。例如, Jarrett等[11]考虑使用非线性激活函数$f(x)=|tanh(x)|，在数据集Caltech-101数据集上，与基于平均池化的对比归一化结合，取得了很好的结果。然而，在这个数据集上他们主要关心的就是防止过拟合，他们观察的结果并不是拟合训练数据的加速能力，而我们使用ReLus函数的目的正在于此。因为快速学习对由大规模数据集上训练出大模型的性能有相当大的影响。
Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each network were chosen independently to make train- ing as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consis- tently learn several times faster than equivalents with saturating neurons.
图表1:使用ReLu激活函数的卷积神经网络（实线）达到25%的错误率，这个训练过程所使用的时间是使用tanh激活函数的卷积神经网络（虚线）的六分之一。为了尽可能的让训练速度更快，每一个网络的学习率都是独立选择的。任何一种都没有经过正则化。这里展示的效果的量级随着网络的结构而变化，但是利用ReLUs的网络始终比同等条件下使用饱和神经元的学习速度快很多倍。
3.2 Training on Multiple GPUs
3.2 多GPU训练
A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.
单个GTX 580GPU只有3GB的内存，从而限制了能由它训练出网络的最大规模。实验表明，使用120万训练样本训练网络已经足够，但是这个任务对一个GPU来说太大了。因此，本文中的网络使用两个GPU。当前的GPU都能很方便的进行交叉GPU并行，因为他们可以直接相互之间读写内存，而不用使用主机内存。我们采用的并行模式本质上就是在每一个GPU上放二分之一的核（或者神经元），我们还使用了其他方法：只有在某些特定的层才能进行GPU之间的交互。这就意味着，例如，第三层的输入为第二层的所有特征图，但是第四层的输入是仅仅是第三层在同一GPU上的特征图。在交叉验证时，连接模式的选择是一个问题，而这个也恰好允许我们精确地调整通信的数量，直到他占计算数量的一个合适比例。
The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cires ̧an et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2.
最终的结构有点像Ciresan等[5]采用的柱状卷积神经网络，但是本文的列不是独立的（见图二）。这种组合让本文的top-1和top-5错误率分别下降了1.7%和1.2%。作为对照，我们让网络模型减少一半的卷积核，并使用单GPU进行训练。结果显示使用双GPU训练时间会更少。
The one-GPU net actually has the same number of kernels as the two-GPU net in the final convolutional layer. This is because most of the net’s parameters are in the first fully-connected layer, which takes the last convolutional layer as input. So to make the two nets have approximately the same number of parameters, we did not halve the size of the final convolutional layer (nor the fully-conneced layers which follow). Therefore this comparison is biased in favor of the one-GPU net, since it is bigger than “half the size” of the two-GPU net.
单GPU结构和双GPU结构在最后一个卷积层有着相同数量的卷积核，因为绝大多数的网络参数都在第一个全连接层，使用最后一个卷积层的输出作为输入。这样就使两种结构的网络有着大约相同数量的参数。我们没有在最后一个卷积层将单GPU结构的参数减半（在紧跟着的全连接层也没有）。因此，在这个对比中，单GPU的网络结构是比双GPU网络减少一半规模的结构要略微大一些。
3.3Local Response Normalization
3.3局部响应归一化
ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by aix,y the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity bix,y is given by the expression
where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k, n, α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10−4, and β = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).
RELUs函数的好处就是，它不需要对输入进行标准化来防止饱和。只要一些训练样本产生一个正输入给一个RELU，那么在那个神经元中学习就会开始。但是，我们还发现如下的局部标准化方案有助于增加泛化性能。aix,y表示使用核i作用于（x，y）然后再采用RELU非线形激活函数计算得到的活跃度。那么响应标准化活跃bix,y由以下公式算出
这里，对同一个空间位置的n个邻接核特征图（kernel maps）求和，N是该层的核的总数目。核特征图的顺序显然是任意的，并且在训练之前就已经决定了的。这种响应归一化实现了侧抑制的一种形式，侧抑制受启发于一种真实神经中发现的形式，对利用不同核计算得到的神经输出之间的大的活跃度生成竞争。常数k, n, α, β是超参数，他们的值使用一个验证集来确定。本文使用k=2, n=5, α=10-4, β=0.75. 本文在某些特定的层中，使用RELUs非线性函数之后应用了归一化。（见3.5节）
This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3.
这个方案与Jarrett等[11]的局部对比度归一化有些类似，但是本文更加准确的称为“亮度归一化”（因为本文没有减去平均活跃度）。响应归一化将我们的top-1和top-5错误率从1.4%降到1.2%。本文也在CIFAR-10数据集上验证了这个方案的有效性：一个四层的CNN网络在未归一化的错误率是13%，在归一化的情况下错误率是11%。
Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z×z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.
3.4 重叠池化
CNNs中的池化层归纳了同一个核特征图中相邻神经元组的输出。通常，由邻接池化单元归纳的邻域并不重叠（例如，[17, 11, 4]）。更确切地说，一个池化层可以看作是包含了每间隔s个像素的池化单元的珊格组成。每一个都归纳了以池化单元为中心，大小为z*z的邻域。如果令S=Z，将会得到CNNs通常采用的局部池化。如果设置S Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution. 现在，我们可以来描述本文CNN的整体结构。正如图2描述所示，这个网络包含八个有权值的层：前五层是卷积层，剩下的三层是全连接层，最后一个全连接层的输出将被喂入1000分类的softmax中，产生1000个类标签的分布。本文的网络最大化多项逻辑回归的结果，也就是最大化训练集中这个预测分布下的正确标签的对数概率的均值。 The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully- connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. 第二、四、五层卷积层的核只和同一个GPU上的前层的核特征图相连（见图二）。第三层卷积核和第二层所有的核特征图相连接。全连接层中的神经元和前一层中的所有神经元相连接。响应归一化层跟着第一和第二层卷积层。最大池化层，3.4中有所描述，既跟着响应归一化层，也跟着第五层卷积层。ReLUs非线性变换应用于每一个卷积核全连接层的输出。 The first convolutional layer filters the 224 × 224 × 3 input image with 96 kernels of size 11 × 11 × 3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboringneurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 × 5 × 48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 × 3 × 192 , and the fifth convolutional layer has 256 kernels of size 3 × 3 × 192. The fully-connected layers have 4096 neurons each. 第一个卷积层使用96个大小为11* 11*3的卷积核对224x224x3的输入图像以4个像素为步长（这是核特征图中相邻感受野中心之间的距离）进行卷积。第二层卷积层将第一层卷积层的输出（经过响应归一化核池化）作为输入，并使用256个大小为5x5x48的核对它进行滤波。第三层，第四层，第五层的卷积层在没有任何池化或者归一化的干预下进行连接。第三个卷积层有384个大小为3x3x256的核，与第二层卷积层（已经归一化和池化）的输出相连。第四个卷积层有384个3x3x192的卷积核，第五个卷积层有256个3x3x192大小的卷积核。每个全连接层有4096个神经元。 Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting. 本文的网络结构有六千万个参数，尽管ILSVRC的1000个类别使得每个训练样本使用10个比特就可以将图像映射到标签上。如果没有大量的过拟合，是不足以学习这些参数的。接下来，我们描述两种主要的方法来对抗过拟合。 The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free. 降低图像数据过拟合的最简单的方法就是利用标签转换人为地增大数据集（例如[25,4,5]）。本文采取两种不同的数据增强方式，这两种方式只需要少量的资源就可以从原图中产生转换图像，因此转换图像不需要存入磁盘。本文利用GPU训练先前一批图像的同时，使用CPU运行Python代码生成转换图像。因此这些数据增强方法实际上是不用消耗计算资源的。 The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random 224×224 patches(and their horizontal reflections) from the 256×256 images and training our network on these extracted patches 4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches. 第一种数据增强的形式包括生成平移图像和水平翻转图像。做法就是从256x256的图像中提取随机的224x224大小的块（以及它们的水平翻转）然后基于这些提取的块来训练我们的网络。这种方法让我们的训练集增加了2048倍（256-224）x2x2=2048，尽管产生的数据是高度相互依赖的。但是如果不使用这个方法，本文的网络会有大量的过拟合，这必将迫使我们使用更小的网络。在测试时，网络通过提取5个224x224块（四个边角块核一个中心块）以及它们的水平翻转（因此共十个块）来做预测，然后网络的softmax层对这十个块做出的预测取均值。 The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy = [IRxy,IGxy,IBxy]T we add the following quantity: where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over 1%. 第二个数据增强的方法包括改变训练图像的RGB通道强度。特别的，本文对整个ImageNet训练集的RGB图像的像素值进行了PCA。对每一幅图像，本文加上多倍的主成分，倍数的值为相应的特征值乘以一个均值为0标准差为0.1的高斯函数产生的随机变量。因此对每一个RGB图像像素图像像素Ixy=[IRxy, IGxy,IBxy]T加上如下的量 这里Pi, λi分别是RGB像素值的3x3协方差矩阵的第i个特征向量和特征值，αi是前述的随机变量，每一个αi的值对一幅特定的训练图像的所有像素是不变的，直到这幅图像再次用于训练，此时才又赋予αi新的值，这个方法得到了图像的一个重要性质，那就是，随着光照的强度和颜色的改变，目标的特性是不变的。这个方案将top-1错误率降低了1%。 Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks. 结合多种不同的模型的预测结果是一种可以降低测试错误率的有效方法[1,3],但是这对于已经要花很多天来训练的大规模神经网络来说显得太耗费时间了，但是，有一种非常有效的模型结合方法，训练时间只需要原来时间的两倍。最新研究的技术，叫做“dropout[10]”,它将每一个隐藏神经元的输出以50%的概率设置为0。这些以这种方式被“踢出”的神经元不会参加前向传递，也不会参加反向传播。因此每次有输入时，神经网络采样一个不同的结构，但是这些结构共享权重。这个技术降低了神经元之间复杂的适应性，因此一个神经元不是依赖于其他特定的神经元。这样就迫使神经元学习更加有效的特征，在和其他神经网络的某些部分相连接时，能够起到更好的作用。在测试时，本文所使用了所有的神经元，在输出时都乘以了0.5，对采用多指数dropout网络生成的预测分布的几何平均数来说这是一个合理的近似。 We use dropout in the first two fully-connected layers of Figure2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge. 本文在图二的全连接的前两个层运用了dropout方法。如果不采用dropout方法，本文的神经网络将会出现大量的过拟合。dropout大概将达到收敛的迭代次数增加了一倍。 We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model’s training error. The update rule for weight w was is the average over the ith batch Di of the derivative of the objective with respect to w, evaluated at wi. We initialized the weights in each layer from a zero-mean Gaussian distribution with standard de- viation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0. We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs. 本文使用随机梯度下降来训练模型，同时设置batch_size大小为128,0.9倍动量以及0.0005的权值衰减。我们发现这个很小的权值衰减对于模型训练相当重要。换句话说，这里的权重衰减不只是一个正则化矩阵，他降低了模型的训练错误率。权重w的更新规则是： 是在点ωi，目标对ω求得的导数的第i个batch:Di的均值。 本文对每一层权重值使用均值为0、标准差为0.01的高斯分布进行初始化。在第二层，第四层和第五层卷积层以及全连接的隐藏层使用常数1初始化神经元偏置项。这个初始化通过给ReLUs激活函数提供正项的输入加快了初始阶段的学习。本文对剩余的层使用常数0初始化神经元偏置项。 我们在所有的层都适用相等的学习率，这个可以在整个学习过程中手动调整来实现。当神经网络使用某一学习率进行学习时，一旦验证集的错误率不再上升，我们就将当前的学习速率除以10，这是一种启发式的方法。学习速率一开始被设置为 0.01，在终止训练之前减少了三次。本文训练该网络对120万的图像训练了大约90个周期，使用了两个英伟达 GTX 580显卡 每个显卡有3GB的内存，花费了五到六天的时间。 Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%5. The best performance achieved during the ILSVRC- 2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best pub- lished results are 45.7% and 25.7% with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24]. 本文的在ILSVRC-2010上的结果见表1。本文网络的测试集top-1和top-5的错误率分别为37.5%和17.0%。在ILSVRC-2010比赛中最好的结果是47.1%和28.2%，采用的方法是对六个基于不同特征训练得到的稀疏编码模型的预测结果求平均数[2]，此后最好的结果是45.7%和25.7%，方法是基于从两种密集采样特征计算得到的Fisher向量（FVs），训练得到两个分类器，所得的结果求平均数。 Table 1: Comparison of results on ILSVRC2010 test set. In italics are best results achieved by others. 表1 基于ILSVRC-2010测试集的结果对比。斜体字是其他方法获得的最好结果 We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1%(see Table 2). The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth con- volutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of 15.3%. The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7]. 我们也以我们的模型参加了ILSVRC-2012比赛，表2中是我们的结果。由于ILSVRC-2012测试集标签不是公开的，我们不能报告我们尝试的所有模型的测试错误率。在本段接下来的部分，我们交换着使用验证错误率和测试错误率因为根据我们的经验，他们不会有超过0.1%的不同（见表2）。本文中所描述的CNN的top-5错误率是18.2%。五个相似的CNN的平均预测结果的错误率是16.4%。在最后一个池化层上增加第六个卷积层，使用整个ImageNet Fall 2011的数据（15M图像，22000种类别）作为分类数据预训练得到的一个CNN，再经过微调，用ILSVRC-2012对该CNN进行测试得到的错误率为16.6%。对上述的五个在整个Fall 2011数据集上预训练过的CNN，得到的预测求平均得到的错误率结果为15.3%。当时第二的队伍得到的错误率为26.2%，使用的方法是对基于从多种密集采样特征计算得到的FVs，训练得到多个分类器的预测值求平均[7]。 Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best results achieved by others. Models with an asterisk* were “pre-trained” to classify the entire ImageNet 2011 Fall release. See Section 6 for details. 表2 基于ILSVRC-2012的验证集和测试集的错误率比较。斜体字是其他方法取得的最好结果。带星号*的模型预先用整个ImageNet 2011 Fall数据集训练过。详见第6节。 Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are 67.4% and 40.9%, attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].. 最后，我们也汇报了我们的基于包含10184钟类别890万张图像的Fall 2009的错误率。对于该数据集我们遵从文献中的约定：一半为训练集一半为测试集。由于没有确定的测试集，我们的分割必然与前面作者使用的不一样，但是这并不会明显地影响结果。我们在该数据集上得到top-1和top-5错误率分别为67.4%和40.9%，这个结果是由在上述网络的最后一个池化层加了第六层卷积层所得到的。之前在这个数据集上最好的结果是78.1%和60.9%[19]。 Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs). Figure 3: 96 convolutional kernels of size 11×11×3 learned by the first convolutional layer on the 224×224×3 input images. The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details. 图三：第一层卷积层是对224x224x3的输入图像使用96个11x11x3的卷积核进行学习，上面48个卷积核在GPU1上进行学习，下面的48个卷积核在GPU2上学习，详见6.1节 Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar(if it happens to be in the top5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image. 图4 （左）8幅ILSVRC-2010测试图像和五个根据本文模型得到的最可能的标签。每一幅图像下方写着正确的标签，正确标签的可能性大小也用红色条表示了出来（如果其恰巧在前5个）。（右）第一列是五幅ILSVRC-2010测试图像。剩余列表示6幅训练图像，这些训练图像在最后一层隐藏层得到的特征向量与测试图像的特征向量有最小的欧式距离。 In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph. 图4的左边我们通过计算8幅测试图像的top-5预测结果定性地评估了网络学习到了什么。注意到即使偏离中心的目标，例如左上方的小虫，也可以被网络识别。大多数top-5标签都显得很合理。例如，只有别的种类的猫被似是而非贴上豹子的标签。在一些情况下（窗格、樱桃）会存在对照片的意图的判断的含糊不清。 Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material. 另一种探讨网络的视觉知识的方法就是考虑最终图像在最后4096维隐藏层产生的特征激活度。如果两幅图像产生的特征激活向量的欧氏距离很小，我们就可以说神经网络的更高层认为它们是相似的（根据了特征激活向量的欧式距离，这种测度跟视觉感官上的相似度是不同的）。图4显示了根据这种测度下的五幅测试集图像和六幅跟他们最相似的训练集图像。注意到在像素水平，第二列中检索到的训练图像一般地不会和第一列的查询图像相近。例如，检索到的狗和大象以多种姿势出现。我们在补充材料中展示更多测试图像的结果。 Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying auto- encoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar. 使用欧氏距离计算4096维、实值向量之间的相似度效率较低，但是可以通过训练一个自动编码器来将这些向量压缩为短的二进制编码而提高效率。这将产生一种比对原始像素应用自动编码器更好的图像检索方法[14]，该方法不使用图像标签，因此有检索具有相似边缘模式的图像的趋势，无论它们在语义上是否相似。 Our results show that a large, deep convolutional neural network is capable of achieving record- breaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results. 本文的结果表明一个大规模深度卷积神经网络在具有高度挑战性的数据集上仅用监督学习就能够获得破纪录的好结果。值得注意的是如果一个卷积层被移除则本文的网络性能会降低。例如，移除任一个中间层，网络的top-1性能会降低大约2%。因此深度对本文的结果真的很重要。 To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images. 为了简化本文的实验，我们没有使用任何非监督预训练即使我们认为它会起作用，尤其是我们可以在标签数据没有发生相应增长的情况下，获得足够的计算资源来增大我们网络的大小，能够有足够的计算能力去显著地增加网络的大小。迄今，由于我们使用了更大的网络，训练了更长的时间，本文的结果已经有所提高，但我们仍然有很多需求来进行时空下人类视觉系统的研究。最终我们想要将非常大规模地深度卷积网络应用于视频序列的处理，视频序列中的时间结构提供了许多有用的信息，而这些信息在静态图中丢失了或者不是很明显。 总体结构： 论文的创新点： 1、使用了不饱和非线性单元ReLUs 2、多GPU训练 3、局部响应归一化 4、重叠池化 5、防止过拟合的措施 （1）数据增强 （2）dropout方法 将最后的隐藏层输出的4096维的向量和其他图片输出的4096维图片做欧式距离，距离较近的，他们的语义信息也很相近。 展望：数据集的大小，卷积层的深度对网络很重要，以后会处理视频等序列化的内容，考虑视频中时间序列的信息，对视频识别很重要，而图片识别恰好忽略了这一点。3.5 Overall Architecture
3.5 总体结构
4 Reducing Overfitting
4 减少过拟合
4.1 Data Augmentation
4.1 数据增强
[P1,P2,P3] [α1λ1, α2λ2, α3λ3]T
4.2 Dropout
4.2 Dropout舍弃
5、Details of learning
5、模型细节
6 Results
6 结果
6.1 Qualitative Evaluations
7 Discussion
7 讨论
总结