用java写bp神经网络（四）
接上篇。
在(一)和(二)中，程序的体系是Net,Propagation,Trainer,Learner,DataProvider。这篇重构这个体系。
Net
首先是Net，在上篇重新定义了激活函数和误差函数后，内容大致是这样的：
List<DoubleMatrix> weights = new ArrayList<DoubleMatrix>();
List<DoubleMatrix> bs = new ArrayList<>();
List<ActivationFunction> activations = new ArrayList<>();
CostFunction costFunc;
CostFunction accuracyFunc;
int[] nodesNum;
int layersNum;
public CompactDoubleMatrix getCompact(){
return new CompactDoubleMatrix(this.weights,this.bs);
}
函数getCompact()生成对应的超矩阵。
DataProvider
DataProvider是数据的提供者。
public interface DataProvider {
DoubleMatrix getInput();
DoubleMatrix getTarget();
}
如果输入为向量，还包含一个向量字典。
public interface DictDataProvider extends DataProvider {
public DoubleMatrix getIndexs();
public DoubleMatrix getDict();
}
每一列为一个样本。getIndexs()返回输入向量在字典中的索引。
我写了一个有用的类BatchDataProviderFactory来对样本进行批量分割，分割成minibatch。
int batchSize;
int dataLen;
DataProvider originalProvider;
List<Integer> endPositions;
List<DataProvider> providers;
public BatchDataProviderFactory(int batchSize, DataProvider originalProvider) {
super();
this.batchSize = batchSize;
this.originalProvider = originalProvider;
this.dataLen = this.originalProvider.getTarget().columns;
this.initEndPositions();
this.initProviders();
}
public BatchDataProviderFactory(DataProvider originalProvider) {
this(4, originalProvider);
}
public List<DataProvider> getProviders() {
return providers;
}
batchSize指明要分多少批，getProviders返回生成的minibatch，被分的原始数据为originalProvider。
Propagation
Propagation负责对神经网络的正向传播过程和反向传播过程。接口定义如下：
public interface Propagation {
public PropagationResult propagate(Net net,DataProvider provider);
}
传播函数propagate用指定数据对指定网络进行传播操作，返回执行结果。
BasePropagation实现了该接口，实现了简单的反向传播：
public class BasePropagation implements Propagation{
// 多个样本。
protected ForwardResult forward(Net net,DoubleMatrix input) {
ForwardResult result = new ForwardResult();
result.input = input;
DoubleMatrix currentResult = input;
int index = -1;
for (DoubleMatrix weight : net.weights) {
index++;
DoubleMatrix b = net.bs.get(index);
final ActivationFunction activation = net.activations
.get(index);
currentResult = weight.mmul(currentResult).addColumnVector(b);
result.netResult.add(currentResult);
// 乘以导数
DoubleMatrix derivative = activation.derivativeAt(currentResult);
result.derivativeResult.add(derivative);
currentResult = activation.valueAt(currentResult);
result.finalResult.add(currentResult);
}
result.netResult=null;// 不再需要。
return result;
}
// 多个样本梯度平均值。
protected BackwardResult backward(Net net,DoubleMatrix target,
ForwardResult forwardResult) {
BackwardResult result = new BackwardResult();
DoubleMatrix output = forwardResult.getOutput();
DoubleMatrix outputDerivative = forwardResult.getOutputDerivative();
result.cost = net.costFunc.valueAt(output, target);
DoubleMatrix outputDelta = net.costFunc.derivativeAt(output, target).muli(outputDerivative);
if (net.accuracyFunc != null) {
result.accuracy=net.accuracyFunc.valueAt(output, target);
}
result.deltas.add(outputDelta);
for (int i = net.layersNum - 1; i >= 0; i--) {
DoubleMatrix pdelta = result.deltas.get(result.deltas.size() - 1);
// 梯度计算，取所有样本平均
DoubleMatrix layerInput = i == 0 ? forwardResult.input
: forwardResult.finalResult.get(i - 1);
DoubleMatrix gradient = pdelta.mmul(layerInput.transpose()).div(
target.columns);
result.gradients.add(gradient);
// 偏置梯度
result.biasGradients.add(pdelta.rowMeans());
// 计算前一层delta，若i=0，delta为输入层误差，即input调整梯度，不作平均处理。
DoubleMatrix delta = net.weights.get(i).transpose().mmul(pdelta);
if (i > 0)
delta = delta.muli(forwardResult.derivativeResult.get(i - 1));
result.deltas.add(delta);
}
Collections.reverse(result.gradients);
Collections.reverse(result.biasGradients);
//其它的delta都不需要。
DoubleMatrix inputDeltas=result.deltas.get(result.deltas.size()-1);
result.deltas.clear();
result.deltas.add(inputDeltas);
return result;
}
@Override
public PropagationResult propagate(Net net, DataProvider provider) {
ForwardResult forwardResult=this.forward(net, provider.getInput());
BackwardResult backwardResult=this.backward(net, provider.getTarget(), forwardResult);
PropagationResult result=new PropagationResult(backwardResult);
result.output=forwardResult.getOutput();
return result;
}
我们定义的PropagationResult略为：
public class PropagationResult{
DoubleMatrix output;// 输出结果矩阵：outputLen*sampleLength
DoubleMatrix cost;// 误差矩阵：1*sampleLength
DoubleMatrix accuracy;// 准确度矩阵：1*sampleLength
private List<DoubleMatrix> gradients;// 权重梯度矩阵
private List<DoubleMatrix> biasGradients;// 偏置梯度矩阵
DoubleMatrix inputDeltas;//输入层delta矩阵：inputLen*sampleLength
public CompactDoubleMatrix getCompact(){
return new CompactDoubleMatrix(gradients,biasGradients);
}
}
另一个实现了该接口的类为MiniBatchPropagation。他在内部用并行方式对样本进行传播，然后对每个minipatch结果进行综合，内部用到了BatchDataProviderFactory类和BasePropagation类。
Trainer
Trainer接口定义为：
public interface Trainer {
public void train(Net net,DataProvider provider);
}
简单的实现类为：
public class CommonTrainer implements Trainer {
int ecophs;
Learner learner;
Propagation propagation;
List<Double> costs = new ArrayList<>();
List<Double> accuracys = new ArrayList<>();
public void trainOne(Net net, DataProvider provider) {
PropagationResult propResult = this.propagation
.propagate(net, provider);
learner.learn(net, propResult, provider);
Double cost = propResult.getMeanCost();
Double accuracy = propResult.getMeanAccuracy();
if (cost != null)
costs.add(cost);
if (accuracy != null)
accuracys.add(accuracy);
}
@Override
public void train(Net net, DataProvider provider) {
for (int i = 0; i < this.ecophs; i++) {
System.out.println("echops:"+i);
this.trainOne(net, provider);
}
}
}
简单的迭代echops此，没有智能停止功能，每次迭代用Learner调节权重。
Learner
Learner根据每次传播结果对网络权重进行调整，接口定义如下：
public interface Learner<N extends Net,P extends DataProvider> {
public void learn(N net,PropagationResult propResult,P provider);
}
一个简单的根据动量因子-自适应学习率进行调整的实现类为：
public class MomentAdaptLearner<N extends Net, P extends DataProvider>
implements Learner<N, P> {
double moment = 0.7;
double lmd = 1.05;
double preCost = 0;
double eta = 0.01;
double currentEta = eta;
double currentMoment = moment;
CompactDoubleMatrix preGradient;
public MomentAdaptLearner(double moment, double eta) {
super();
this.moment = moment;
this.eta = eta;
this.currentEta = eta;
this.currentMoment = moment;
}
public MomentAdaptLearner() {
}
@Override
public void learn(N net, PropagationResult propResult, P provider) {
if (this.preGradient == null)
init(net, propResult, provider);
double cost = propResult.getMeanCost();
this.modifyParameter(cost);
System.out.println("current eta:" + this.currentEta);
System.out.println("current moment:" + this.currentMoment);
this.updateGradient(net, propResult, provider);
}
public void updateGradient(N net, PropagationResult propResult, P provider) {
CompactDoubleMatrix netCompact = this.getNetCompact(net, propResult,
provider);
CompactDoubleMatrix gradCompact = this.getGradientCompact(net,
propResult, provider);
gradCompact = gradCompact.mul(currentEta * (1 - currentMoment)).addi(
preGradient.mul(currentMoment));
netCompact.subi(gradCompact);
this.preGradient = gradCompact;
}
public CompactDoubleMatrix getNetCompact(N net,
PropagationResult propResult, P provider) {
return net.getCompact();
}
public CompactDoubleMatrix getGradientCompact(N net,
PropagationResult propResult, P provider) {
return propResult.getCompact();
}
public void modifyParameter(double cost) {
if (this.currentEta > 10) {
this.currentEta = 10;
} else if (this.currentEta < 0.0001) {
this.currentEta = 0.0001;
} else if (cost < this.preCost) {
this.currentEta *= 1.05;
this.currentMoment = moment;
} else if (cost < 1.04 * this.preCost) {
this.currentEta *= 0.7;
this.currentMoment *= 0.7;
} else {
this.currentEta = eta;
this.currentMoment = 0.1;
}
this.preCost = cost;
}
public void init(Net net, PropagationResult propResult, P provider) {
PropagationResult pResult = new PropagationResult(net);
preGradient = pResult.getCompact().dup();
}
}
在上面的代码中，我们可以看到CompactDoubleMatrix类对权重自变量的封装，使代码更加简洁，它在此表现出来的就是一个超矩阵，超向量，完全忽略了内部的结构。
同时，其子类实现了同步更新字典的功能，代码也很简洁，只是简单的把需要调整的矩阵append到超矩阵中去即可，在父类中会统一对其进行调整：
public class DictMomentLearner extends
MomentAdaptLearner<Net, DictDataProvider> {
public DictMomentLearner(double moment, double eta) {
super(moment, eta);
}
public DictMomentLearner() {
super();
}
@Override
public CompactDoubleMatrix getNetCompact(Net net,
PropagationResult propResult, DictDataProvider provider) {
CompactDoubleMatrix result = super.getNetCompact(net, propResult,
provider);
result.append(provider.getDict());
return result;
}
@Override
public CompactDoubleMatrix getGradientCompact(Net net,
PropagationResult propResult, DictDataProvider provider) {
CompactDoubleMatrix result = super.getGradientCompact(net, propResult,
provider);
result.append(DictUtil.getDictGradient(provider, propResult));
return result;
}
@Override
public void init(Net net, PropagationResult propResult,
DictDataProvider provider) {
DoubleMatrix preDictGradient = DoubleMatrix.zeros(
provider.getDict().rows, provider.getDict().columns);
super.init(net, propResult, provider);
this.preGradient.append(preDictGradient);
}
}