Deep learning:三十七(Deep learning中的优化方法)

原文地址为: Deep learning:三十七(Deep learning中的优化方法)

 

  内容:

  本文主要是参考论文:On optimization methods for deep learning,文章内容主要是笔记SGD(随机梯度下降),LBFGS(受限的BFGS),CG(共轭梯度法)三种常见优化算法的在deep learning体系中的性能。下面是一些读完的笔记。

  SGD优点:实现简单,当训练样本足够多时优化速度非常快。

  SGD缺点:需要人为调整很多参数,比如学习率,收敛准则等。另外,它是序列的方法,不利于GPU并行或分布式处理。

  各种deep learning中常见方法(比如说Autoencoder,RBM,DBN,ICA,Sparse coding)的区别是:目标函数形式不同。这其实才是最本质的区别,由于目标函数的不同导致了对其优化的方法也可能会不同,比如说RBM中目标函数跟网络能量有关,采用CD优化的,而Autoencoder目标函数为理论输出和实际输出的MSE,由于此时的目标函数的偏导可以直接被计算,所以可以用LBFGS,CG等方法优化,其它的类似。所以不能单从网络的结构来判断其属于Deep learning中的哪种方法,比如说我单独给定64-100的2层网络,你就无法知道它属于deep learning中的哪一种方法,因为这个网络既可以用RBM也可以用Autoencoder来训练。

  作者通过实验得出的结论是:不同的优化算法有不同的优缺点,适合不同的场合,比如LBFGS算法在参数的维度比较低(一般指小于10000维)时的效果要比SGD(随机梯度下降)和CG(共轭梯度下降)效果好,特别是带有convolution的模型。而针对高维的参数问题,CG的效果要比另2种好。也就是说一般情况下,SGD的效果要差一些,这种情况在使用GPU加速时情况一样,即在GPU上使用LBFGS和CG时,优化速度明显加快,而SGD算法优化速度提高很小。在单核处理器上,LBFGS的优势主要是利用参数之间的2阶近视特性来加速优化,而CG则得得益于参数之间的共轭信息,需要计算器Hessian矩阵。

  不过当使用一个大的minibatch且采用线搜索的话,SGD的优化性能也会提高。

  在单核上比较SGD,LBFGS,CG三种算法的优化性能,当针对Autoencoder模型。结果如下:

   Deep learning:三十七(Deep learning中的优化方法)_第1张图片

  可以看出,SGD效果最差。

  同样的情况下,训练的是Sparse autoencoder模型的比较情况如下:

   Deep learning:三十七(Deep learning中的优化方法)_第2张图片

  这时SGD的效果更差。这主要原因是LBFGS和CG能够使用大的minibatch数据来估算每个节点的期望激发值,这个值是可以用来约束该节点的稀疏特性的,而SGD需要去估计噪声信息。

  当然了作者还有在GUP,convolution上也做了不少实验。

  最后,作者训练了一个2隐含层(这2层不算pooling层)的Sparse autocoder网络,并应用于MNIST上,其识别率结果如下:

   Deep learning:三十七(Deep learning中的优化方法)_第3张图片

  作者网站上给出了一些code,见deep autoencoder with L-BFGS。看着标题本以为code会实现deep convolution autoencoder pre-training和fine-tuning的,因为作者paper里面用的是convolution,阅读完code后发现其实现就是一个普通二层的autoencoder。看来还是得到前面博文第二个问题的答案:Deep learning:三十六(关于构建深度卷积SAE网络的一点困惑)

 

  下面是作者code主要部分的一些注释:

optimizeAutoencoderLBFGS.m(实现deep autoencoder网络的参数优化过程):

function [] = optimizeAutoencoderLBFGS(layersizes, datasetpath, ...
finalObjective)
% train a deep autoencoder with variable hidden sizes
% layersizes : the sizes of the hidden layers. For istance, specifying layersizes =
% [200 100] will create a network looks like input -> 200 -> 100 -> 200
% -> output (same size as input). Notice the mirroring structure of the
% autoencoders. Default layersizes = [2*3072 100]
% datasetpath: the path to the CIFAR dataset (where we find the *.mat
% files). see loadData.m
% finalObjective: the final objective that you use to compare to
% terminate your optimization. To qualify, the objective
% function on the entire training set must be below this
% value.
%
% Author: Quoc V. Le ([email protected])
%
%% Handle default parameters
if nargin < 3 || isempty(finalObjective)
finalObjective
= 70; % i am just making this up, the evaluation objective
% will be much lower
end
if nargin < 2 || isempty(datasetpath)
datasetpath
= '.';
end
if nargin < 1 || isempty(layersizes)
layersizes
= [2*3072 100];
layersizes
= [200 100];
end

%% Load data
loadData
%traindata 3072*10000的,每一列表示一个向量

%% Random initialization
initializeWeights;
%看作者对应该部分的code,也没有感觉出convolution和pooling的影响啊,怎么它就连接起来了呢

%% Optimization: minibatch L-BFGS
% Q.V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, A.Y. Ng.
% On optimization methods for deep learning. ICML, 2011

addpath minFunc
/
options.Method
= 'lbfgs';
options.maxIter
= 20;
options.display
= 'on';
options.TolX
= 1e-3;

perm
= randperm(size(traindata,2));
traindata
= traindata(:,perm);% 将训练样本随机排列
batchSize
= 1000;%因为总共样本数为10000个,所以分成了10个批次
maxIter
= 20;
for i=1:maxIter
startIndex
= mod((i-1) * batchSize, size(traindata,2)) + 1;
fprintf(
'startIndex = %d, endIndex = %d\n', startIndex, startIndex + batchSize-1);
data
= traindata(:, startIndex:startIndex + batchSize-1);
[theta, obj]
= minFunc( @deepAutoencoder, theta, options, layersizes, ...
data);
if obj <= finalObjective % use the minibatch obj as a heuristic for stopping
% because checking the entire dataset is very
% expensive
% yes, we should check the objective for the entire training set
trainError
= deepAutoencoder(theta, layersizes, traindata);
if trainError <= finalObjective
% now your submission is qualified
break
end
end
end

%% write to text files so that we can test your program
writeToTextFiles;

 

deepAutoencoder.m:(深度网络代价函数及其导数的求解函数):

function [cost,grad] = deepAutoencoder(theta, layersizes, data)
% cost and gradient of a deep autoencoder
% layersizes is a vector of sizes of hidden layers, e.g.,
% layersizes[2] is the size of layer 2
% this does not count the visible layer
% data is the input data, each column is an example
% the activation function of the last layer is linear, the activation
% function of intermediate layers is the hyperbolic tangent function

% WARNING: the code is optimized for ease of implemtation and
% understanding, not speed nor space

%% FORCING THETA TO BE IN MATRIX FORMAT FOR EASE OF UNDERSTANDING
% Note that this is not optimized for space, one can just retrieve W and b
% on the fly during forward prop and backprop. But i do it here so that the
% readers can understand what's going on
layersizes = [size(data,1) layersizes];
l
= length(layersizes);
lnew
= 0;
for i=1:l-1
lold
= lnew + 1;
lnew
= lnew + layersizes(i) * layersizes(i+1);
W{i}
= reshape(theta(lold:lnew), layersizes(i+1), layersizes(i));
lold
= lnew + 1;
lnew
= lnew + layersizes(i+1);
b{i}
= theta(lold:lnew);
end
% handle tied-weight stuff
j
= 1;
for i=l:2*(l-1)
lold
= lnew + 1;
lnew
= lnew + layersizes(l-j);
W{i}
= W{l - j}'; %直接用encoder中对应的转置即可
b{i} = theta(lold:lnew);
j
= j + 1;
end
assert(lnew == length(theta), 'Error: dimensions of theta and layersizes do not match\n')


%% FORWARD PROP
for i=1:2*(l-1)-1
if i==1
[h{i} dh{i}]
= tanhAct(bsxfun(@plus, W{i}*data, b{i}));
else
[h{i} dh{i}]
= tanhAct(bsxfun(@plus, W{i}*h{i-1}, b{i}));
end
end
h{i
+1} = linearAct(bsxfun(@plus, W{i+1}*h{i}, b{i+1}));

%% COMPUTE COST
diff
= h{i+1} - data;
M
= size(data,2);
cost
= 1/M * 0.5 * sum(diff(:).^2);% 纯粹标准的autoencoder,不加其它比如sparse限制

%% BACKPROP
if nargout > 1
outderv
= 1/M * diff;
for i=2*(l-1):-1:2
Wgrad{i}
= outderv * h{i-1}';
bgrad{i} = sum(outderv,2);
outderv
= (W{i}' * outderv) .* dh{i-1};
end
Wgrad{
1} = outderv * data';
bgrad{1} = sum(outderv,2);

% handle tied-weight stuff
j
= 1;
for i=l:2*(l-1)
Wgrad{l
-j} = Wgrad{l-j} + Wgrad{i}';
j = j + 1;
end
% dump the results to the grad vector
grad
= zeros(size(theta));
lnew
= 0;
for i=1:l-1
lold
= lnew + 1;
lnew
= lnew + layersizes(i) * layersizes(i+1);
grad(lold:lnew)
= Wgrad{i}(:);
lold
= lnew + 1;
lnew
= lnew + layersizes(i+1);
grad(lold:lnew)
= bgrad{i}(:);
end
j
= 1;
for i=l:2*(l-1)
lold
= lnew + 1;
lnew
= lnew + layersizes(l-j);
grad(lold:lnew)
= bgrad{i}(:);
j
= j + 1;
end
end
end

%% USEFUL ACTIVATION FUNCTIONS
function [a da] = sigmoidAct(x)

a
= 1 ./ (1 + exp(-x));
if nargout > 1
da
= a .* (1-a);
end
end

function [a da] = tanhAct(x)
a
= tanh(x);
if nargout > 1
da
= (1-a) .* (1+a);
end
end

function [a da] = linearAct(x)
a
= x;
if nargout > 1
da
= ones(size(a));
end
end

 

initializeWeights.m(参数初始化赋值,虽然是随机,但是有一定要求):

%% Random initialization
% X. Glorot, Y. Bengio.
% Understanding the dif铿乧ulty of training deep feedforward neural networks.
% AISTATS 2010.
% QVL: this initialization method appears to perform better than
% theta = randn(d,1);
s0
= size(traindata,1);% s0涓烘牱鏈殑缁存暟
layersizes
= [s0 layersizes];%输入层-hidden1-hidden2,这里是3072-6144-100
l
= length(layersizes);%缃戠粶涓殑灞傛暟锛屼笉鍖呭惈瑙g爜閮ㄥ垎锛屽鏋滄槸2涓殣鍚眰鐨勮瘽锛岃繖閲宭=3
lnew
= 0;
for i=1:l-1%1到3之间
lold
= lnew + 1;
lnew
= lnew + layersizes(i) * layersizes(i+1);
r
= sqrt(6) / sqrt(layersizes(i+1)+layersizes(i));
A
= rand(layersizes(i+1), layersizes(i))*2*r - r; %reshape(theta(lold:lnew), layersizes(i+1), layersizes(i));
theta(lold:lnew)
= A(:); %相当于权值W的赋值
lold
= lnew + 1;
lnew
= lnew + layersizes(i+1);
A
= zeros(layersizes(i+1),1);
theta(lold:lnew)
= A(:);%相当于偏置值b的赋值
end %以上是encoder部分
j
= 1;
for i=l:2*(l-1) %1到4之间,下面开始decoder部分
lold
= lnew + 1;
lnew
= lnew + layersizes(l-j);
theta(lold:lnew)
= zeros(layersizes(l-j),1);
j
= j + 1;
end
theta
= theta';
layersizes = layersizes(2:end); %去除输入层

 

 

  参考资料:

  Le, Q. V., et al. (2011). On optimization methods for deep learning. Proc. of ICML.

     deep autoencoder with L-BFGS

      Deep learning:三十六(关于构建深度卷积SAE网络的一点困惑)

 

 

 

 


转载请注明本文地址: Deep learning:三十七(Deep learning中的优化方法)

你可能感兴趣的:(方法,deep,learning,机器学习,优化)