ML—AdaBoost(二)—MATLAB代码
华电北风吹
天津大学认知计算与应用重点实验室
修改日期：2015/7/27
在网上看了几篇AdaBoost的介绍后，感觉网上介绍的都不好，不能够让人完全理解，因此就下载了一个外国人写的代码，总算透彻的理解了AdaBoost，可以向Transfer开进了，现在分享一下代码：
主函数代码
clear;clc;
%
% DEMONSTRATION OF ADABOOST_tr and ADABOOST_te
%
% Just type "demo" to run the demo.
%
% Using adaboost with linear threshold classifier
% for a two class classification problem.
%
% Bug Reporting: Please contact the author for bug reporting and comments.
%
% Cuneyt Mertayak
% email: [email protected]
% version: 1.0
% date: 21/05/2007
% Creating the training and testing sets
%
tr_n = 200;
te_n = 200;
weak_learner_n = 20;
tr_set = abs(rand(tr_n,2))*100;
te_set = abs(rand(te_n,2))*100;
tr_labels = (tr_set(:,1)-tr_set(:,2) > 0) + 1;
te_labels = (te_set(:,1)-te_set(:,2) > 0) + 1;
% Displaying the training and testing sets
figure;
subplot(2,2,1);
hold on; axis square;
indices = tr_labels==1;
plot(tr_set(indices,1),tr_set(indices,2),'b*');
indices = ~indices;
plot(tr_set(indices,1),tr_set(indices,2),'r*');
title('Training set');
subplot(2,2,2);
hold on; axis square;
indices = te_labels==1;
plot(te_set(indices,1),te_set(indices,2),'b*');
indices = ~indices;
plot(te_set(indices,1),te_set(indices,2),'r*');
title('Testing set');
% Training and testing error rates
tr_error = zeros(1,weak_learner_n);
te_error = zeros(1,weak_learner_n);
for i=1:weak_learner_n
adaboost_model = ADABOOST_tr(@threshold_tr,@threshold_te,tr_set,tr_labels,i);
% 训练样本测试
[L_tr,hits_tr] = ADABOOST_te(adaboost_model,@threshold_te,tr_set,tr_labels);
tr_error(i) = (tr_n-hits_tr)/tr_n;
% 测试样本测试
[L_te,hits_te] = ADABOOST_te(adaboost_model,@threshold_te,te_set,te_labels);
te_error(i) = (te_n-hits_te)/te_n;
end
subplot(2,2,3);
plot(1:weak_learner_n,tr_error);
axis([1,weak_learner_n,0,1]);
title('Training Error');
xlabel('weak classifier number');
ylabel('error rate');
grid on;
subplot(2,2,4); axis square;
plot(1:weak_learner_n,te_error);
axis([1,weak_learner_n,0,1]);
title('Testing Error');
xlabel('weak classifier number');
ylabel('error rate');
grid on;调用的分类器训练函数代码：
function model = threshold_tr(train_set, sample_weights, labels)
%
% TRAINING THRESHOLD CLASSIFIER
%
% Training of the basic linear classifier where seperation hyperplane
% is perpedicular to one dimension.
%
% model = threshold_tr(train_set, sample_weights, labels)
% train_set: an NxD-matrix, each row is a training sample in the D dimensional feature
% space.
% sample_weights: an Nx1-vector, each entry is the weight of the corresponding training sample
% labels: Nx1 dimensional vector, each entry is the corresponding label (either 1 or 2)
%
% model: the ouput model. It consists of
% 1) min_error: training error
% 2) min_error_thr: threshold value
% 3) pos_neg: whether up-direction shows the positive region (label:2, 'pos') or
% the negative region (label:1, 'neg')
%
% Bug Reporting: Please contact the author for bug reporting and comments.
%
% Cuneyt Mertayak
% email: [email protected]
% version: 1.0
% date: 21/05/2007
model = struct('min_error',[],'min_error_thr',[],'pos_neg',[],'dim',[]);
sample_n = size(train_set,1);
min_error = sum(sample_weights);
min_error_thr = 0;
pos_neg = 'pos';
% for each dimension
for dim=1:size(train_set,2)
sorted = sort(train_set(:,dim),1,'ascend');
% for each interval in the specified dimension
for i=1:(sample_n+1)
if(i==1)
thr = sorted(1)-0.5;
elseif(i==sample_n+1)
thr = sorted(sample_n)+0.5;
else
thr = (sorted(i-1)+sorted(i))/2;
end
ind1 = train_set(:,dim) < thr;
ind2 = ~ind1;
tmp_err = sum(sample_weights((labels.*ind1)==2))+sum(sample_weights((labels.*ind2)==1));
if(tmp_err < min_error)
min_error = tmp_err;
min_error_thr = thr;
pos_neg = 'pos';
model.dim = dim;
end
ind1 = train_set(:,dim) < thr;
ind2 = ~ind1;
tmp_err = sum(sample_weights((labels.*ind1)==1))+sum(sample_weights((labels.*ind2)==2));
if(tmp_err < min_error)
min_error = tmp_err;
min_error_thr = thr;
pos_neg = 'neg';
model.dim = dim;
end
end
end
model.min_error = min_error;
model.min_error_thr = min_error_thr;
model.pos_neg = pos_neg;分类器的输入输出就不说了，分类器是最简单的与坐标轴垂直的超平面，模型从所有的dim*(sample_n+1)个超平面中，选择加权分类错误率最小的超平面，作为当前权重的最优超平面，并输出结果
调用的分类器测试函数：
function [L,hits,error_rate] = threshold_te(model,test_set,sample_weights,true_labels)
%
% TESTING THRESHOLD CLASSIFIER
%
% Testing of the basic linear classifier where seperation hyperplane is
% perpedicular to one dimension.
%
% [L,hits,error_rate] = threshold_te(model,test_set,sample_weights,true_labels)
%
% model: the model that is outputed from threshold_tr. It consists of
% 1) min_error: training error
% 2) min_error_thr: threshold value
% 3) pos_neg: whether up-direction shows the positive region (label:2, 'pos') or
% the negative region (label:1, 'neg')
% test_set: an NxD-matrix, each row is a testing sample in the D dimensional feature
% space.
% sample_weights: an Nx1-vector, each entry is the weight of the corresponding test sample
% true_labels: Nx1 dimensional vector, each entry is the corresponding label (either 1 or 2)
%
% L: an Nx2-matrix showing likelihoods of each class
% hits: the number of hits
% error_rate: the error rate with the sample weights
%
%
% Bug Reporting: Please contact the author for bug reporting and comments.
%
% Cuneyt Mertayak
% email: [email protected]
% version: 1.0
% date: 21/05/2007
feat = test_set(:,model.dim);
if(strcmp(model.pos_neg,'pos'))
ind = (feat>model.min_error_thr)+1;
else
ind = (feat模型训练函数就是从当前模型训练输入的数据，得到错误率等指标，这个跟模型训练函数对应，看懂那个这里就很简单，从训练的模型中，找出模型需要的那一纬数据，分类，不说了。
调用的AdaBoost训练函数：
function adaboost_model = ADABOOST_tr(tr_func_handle,te_func_handle,train_set,labels,no_of_hypothesis)
%
% ADABOOST TRAINING: A META-LEARNING ALGORITHM
% adaboost_model = ADABOOST_tr(tr_func_handle,te_func_handle,
% train_set,labels,no_of_hypothesis)
%
% 'tr_func_handle' and 'te_func_handle' are function handles for
% training and testing of a weak learner, respectively. The weak learner
% has to support the learning in weighted datasets. The prototypes
% of these functions has to be as follows.
%
% model = train_func(train_set,sample_weights,labels)
% train_set: a TxD-matrix where each row is a training sample in
% a D dimensional feature space.
% sample_weights: a Tx1 dimensional vector, the i-th entry
% of which denotes the weight of the i-th sample.
% labels: a Tx1 dimensional vector, the i-th entry of which
% is the label of the i-th sample.
% model: the output model of the training phase, which can
% consists of parameters estimated.
%
% [L,hits,error_rate] = test_func(model,test_set,sample_weights,true_labels)
% model: the output of train_func
% test_set: a KxD dimensional matrix, each of whose row is a
% testing sample in a D dimensional feature space.
% sample_weights: a Dx1 dimensional vector, the i-th entry
% of which denotes the weight of the i-th sample.
% true_labels: a Dx1 dimensional vector, the i-th entry of which
% is the label of the i-th sample.
% L: a Dx1-array with the predicted labels of the samples.
% hits: number of hits, calculated with the comparison of L and
% true_labels.
% error_rate: number of misses divided by the number of samples.
%
%
% 'train_set' contains the samples for training and it is NxD matrix
% where N is the number of samples and D is the dimension of the
% feature space. 'labels' is an Nx1 matrix containing the class
% labels of the samples. 'no_of_hypothesis' is the number of weak
% learners to be used.
%
% The output 'adaboost_model' is a structure with the fields
% - 'weights': 1x'no_of_hypothesis' matrix specifying the weights
% of the resulted weighted majority voting combination
% - 'parameters': 1x'no_of_hypothesis' structure matrix specifying
% the special parameters of the hypothesis that is
% created at the corresponding iteration of
% learning algorithm
%
% Specific Properties That Must Be Satisfied by The Function pointed
% by 'func_handle'
% ------------------------------------------------------------------
%
% Note: Labels must be positive integers from 1 upto the number of classes.
% Node-2: Weighting is done as specified in AIMA book, Stuart Russell et.al. (sec edition)
%
% Bug Reporting: Please contact the author for bug reporting and comments.
%
% Cuneyt Mertayak
% email: [email protected]
% version: 1.0
% date: 21/05/2007
%
adaboost_model = struct('weights',zeros(1,no_of_hypothesis),'parameters',[]); %cell(1,no_of_hypothesis));
sample_n = size(train_set,1);
samples_weight = ones(sample_n,1)/sample_n;
for turn=1:no_of_hypothesis
model=tr_func_handle(train_set,samples_weight,labels);
adaboost_model.parameters{turn} =model;
[L,hits,error_rate]=te_func_handle(adaboost_model.parameters{turn},train_set,samples_weight,labels);
if(error_rate==1)
error_rate=1-eps;
elseif(error_rate==0)
error_rate=eps;
end
% The weight of the turn-th weak classifier
adaboost_model.weights(turn) = log10((1-error_rate)/error_rate);
C=likelihood2class(L);
t_labeled=(C==labels); % true labeled samples
% Importance of the true classified samples is decreased for the next weak classifier
samples_weight(t_labeled) = samples_weight(t_labeled)*((error_rate)/(1-error_rate));
% Normalization
samples_weight = samples_weight/sum(samples_weight);
end
% Normalization
adaboost_model.weights=adaboost_model.weights/sum(adaboost_model.weights);根据输入的迭代次数，迭代，得到新模型，计算新模型权重，更新样本权重，迭代。。。。。。
调用的AdaBoost测试函数：
function [L,hits] = ADABOOST_te(adaboost_model,te_func_handle,test_set,true_labels)
%
% ADABOOST TESTING
%
% [L,hits] = ADABOOST_te(adaboost_model,te_func_handle,train_set,
% true_labels)
%
% 'te_func_handle' is a handle to the testing function of a
% learning (weak) algorithm whose prototype is shown below.
%
% [L,hits,error_rate] = test_func(model,test_set,sample_weights,true_labels)
% model: the output of train_func
% test_set: a KxD dimensional matrix, each of whose row is a
% testing sample in a D dimensional feature space.
% sample_weights: a Dx1 dimensional vector, the i-th entry
% of which denotes the weight of the i-th sample.
% true_labels: a Dx1 dimensional vector, the i-th entry of which
% is the label of the i-th sample.
% L: a Dx1-array with the predicted labels of the samples.
% hits: number of hits, calculated with the comparison of L and
% true_labels.
% error_rate: number of misses divided by the number of samples.
%
% It is the corresponding testing
% module of the function that is specified in the training phase.
% 'test_set' is a NxD matrix where N is the number of samples
% in the test set and D is the dimension of the feature space.
% 'true_labels' is a Nx1 matrix specifying the class label of
% each corresponding sample's features (each row) in 'test_set'.
% 'adaboost_model' is the model that is generated by the function
% 'ADABOOST_tr'.
%
% 'L' is the likelihoods that are assigned by the 'ADABOOST_te'.
% 'hits' is the number of correctly predicted labels.
%
% Specific Properties That Must Be Satisfied by The Function pointed
% by 'func_handle'
% ------------------------------------------------------------------
%
% Notice: Labels must be positive integer values from 1 upto the number classes.
%
% Bug Reporting: Please contact the author for bug reporting and comments.
%
% Cuneyt Mertayak
% email: [email protected]
% version: 1.0
% date: 21/05/2007
%
hypothesis_n = length(adaboost_model.weights);
sample_n = size(test_set,1);
class_n = length(unique(true_labels));
temp_L = zeros(sample_n,class_n,hypothesis_n); % likelihoods for each weak classifier
% for each weak classifier, likelihoods of test samples are collected
for i=1:hypothesis_n
[temp_L(:,:,i),hits,error_rate] = te_func_handle(adaboost_model.parameters{i},test_set,ones(sample_n,1),true_labels);
temp_L(:,:,i) = temp_L(:,:,i)*adaboost_model.weights(i);
end
L = sum(temp_L,3);
hits = sum(likelihood2class(L)==true_labels);一个结果辅助转换函数：
function classes = likelihood2class(likelihoods)
%
% LIKELIHOODS TO CLASSES
%
% classes = likelihood2class(likelihoods)
%
% Find the class assignment of the samples from the likelihoods
% 'likelihoods' an NxD matrix where N is the number of samples and
% D is the dimension of the feature space. 'likelihoods(i,j)' is
% the i-th samples likelihood of belonging to class-j.
%
% 'classes' contains the class index of the each sample maximum likelihood
%
% Bug Reporting: Please contact the author for bug reporting and comments.
%
% Cuneyt Mertayak
% email: [email protected]
% version: 1.0
% date: 21/05/2007
%
[sample_n,class_n] = size(likelihoods);
maxs = (likelihoods==repmat(max(likelihoods,[],2),[1,class_n]));
classes=zeros(sample_n,1);
for i=1:sample_n
classes(i) = find(maxs(i,:),1);
end这个也不说了，就是把结果转化成矩阵，这个作用是什么，我也懒得看了，看别人的代码，不用看这么细，没必要。抓住精髓就好了。休息。
------------------