(数据科学学习手札17)线性判别分析的原理简介&Python与R实现

之前数篇博客我们比较了几种具有代表性的聚类算法,但现实工作中,最多的问题是分类与定性预测,即通过基于已标注类型的数据的各显著特征值,通过大量样本训练出的模型,来对新出现的样本进行分类,这也是机器学习中最多的问题,而本文便要介绍分类算法中比较古老的线性判别分析:

线性判别

最早提出合理的判别分析法者是R.A.Fisher(1936),Fisher提出将线性判别函数用于花卉分类上,将花卉的各种特征利用线性组合方法变成单变量值,即将高维数据利用线性判别函数进行线性变化投影到一条直线上,再利用单值比较方法来对新样本进行分类,主要步骤如下:

  Step1:求线性判别函数;

  Step2:计算判别界值;

  Step3:建立判别标准(这里与模糊分类中的隶属度有些相似,即离哪一类的投影中心最近,就将样本判别为哪一类)

下面分别利用Python,R,基于著名的花卉分类数据集iris进行演示:

Python

我们利用sklearn包中封装的LinearDiscriminantAnalysis对iris构建线性判别模型,因为LDA实际上是将高维数据尽可能分开的投影到一条直线上,因此LDA也可以对特定数据进行降维转换:

'''Fisher线性判别分析'''
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from matplotlib.pyplot import style
from sklearn.model_selection import train_test_split

style.use('ggplot')

iris = datasets.load_iris()

X = iris.data
y = iris.target

'''展示LDA的降维功能'''
target_names = iris.target_names

'''设置压缩到1维'''
lda = LinearDiscriminantAnalysis(n_components=1)

'''利用线性判别函数将四维的样本数据压缩到一条直线上'''
X_r2 = lda.fit(X,y).transform(X)
X_Zero = np.zeros(X_r2.shape)
'''绘制降维效果图'''
for c,i,target_names in zip('ryb',[0,1,2],target_names):
    plt.scatter(X_r2[y == i],X_Zero[y == i],c=c,label=target_names,s=5)

plt.legend()
plt.grid()

降维后的效果图如下:

(数据科学学习手札17)线性判别分析的原理简介&Python与R实现_第1张图片

下面正式对iris数据集进行LDA分类,这里用到一个新的方法,是sklearn.model_selection.train_test_split,它的作用是根据设置的训练集与测试集的比例进行随机分割,我们利用从样本集中分割的7成数据作为训练集,3成数据进行测试,过程及结果如下:

'''利用sklearn自带的样本集划分方法进行分类,这里选择训练集测试集73开'''
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3)
'''搭建LDA模型'''
lda = LinearDiscriminantAnalysis(n_components=1)
'''利用分割好的训练集进行模型训练并对测试集进行预测'''
ld = lda.fit(X_train,y_train).predict(X_test)
'''比较预测结果与真实分类结果'''
print(np.array([ld,y_test]))
'''打印正确率'''
print('正确率:',str(lda.score(X_test,y_test)))

结果如下:

(数据科学学习手札17)线性判别分析的原理简介&Python与R实现_第2张图片

可以看出,在iris上取得了非常高的准确率。

 

R

在R中做LDA需要用到MASS包中的lda(formula~feature1+feature2+...+featuren,data=df),其中formula表示数据集中表示分类标注的列,右边的各种feature表示将要使用到的分类特征值,也即是构建线性判别函数要用到的基础变量,data指保存全部数据的数据框,具体过程如下:

> #Fisher线性判别
> rm(list=ls())
> library(MASS)
> data(iris)
> data <- iris
> data$Species = as.character(data$Species)
> #创造类别变量
> data$type[data$Species == 'setosa'] = 1
> data$type[data$Species == 'versicolor'] = 2
> data$type[data$Species == 'virginica'] = 3
> #利用简单随机抽样将样本集划分为训练集与验证集
> sam <- sample(1:length(data[,1]),105)
> train <- data[sam,]
> test <- data[-sam,]
> #根据样本数据创建线性判别分析模型
> ld <- lda(type~Sepal.Length+Sepal.Width+Petal.Length+Petal.Width,data=train)
> #将样本集作为验证集求分类结果
> Z <- predict(ld)
> #保存预测类别
> newType <- Z$class
> #与真实分类结果进行比较
> cbind(train$type,Z$x,newType)
             LD1         LD2 newType
74  2 -2.0419740 -1.16551052       2
93  2 -1.2286978 -1.34868982       2
124 3 -4.2056288 -0.39378496       3
43  1  7.0499296 -0.38479505       1
141 3 -6.3297144  1.43240686       3
112 3 -5.2009805 -0.39665336       3
75  2 -1.2076317 -0.53028083       2
83  2 -0.8607055 -1.04255122       2
128 3 -3.7958024  0.38283281       3
134 3 -3.5521695 -0.79338125       2
84  2 -4.1930475 -0.86304028       3
80  2  0.2455854 -1.48598565       2
16  1  8.9754828  2.99336222       1
35  1  6.7862663 -0.74053169       1
64  2 -2.4399568 -0.53150309       2
79  2 -2.4024680 -0.24830622       2
142 3 -4.9647448  1.48158101       3
54  2 -2.2217845 -1.93554530       2
14  1  7.3092968 -0.99420385       1
119 3 -8.8253015 -0.67042900       3
71  2 -3.3701925  0.94839271       3
44  1  6.2098650  0.99672773       1
29  1  7.6841973  0.08156876       1
62  2 -1.7099434  0.15332464       2
59  2 -1.6648086 -0.67372500       2
77  2 -2.3932456 -0.84052244       2
27  1  6.6078477  0.36272030       1
101 3 -7.2690741  1.94879260       3
91  2 -2.1739793 -1.53761667       2
40  1  7.4330661  0.03443993       1
103 3 -5.9833783  0.46642759       3
121 3 -5.9399677  1.45493705       3
51  2 -1.3784535  0.24058778       2
88  2 -2.5153892 -2.12899905       2
6   1  7.5334551  1.60652494       1
148 3 -4.7085798  0.61397476       3
76  2 -1.3919721 -0.13234144       2
10  1  7.0733099 -0.92823629       1
7   1  7.0416936  0.27174257       1
104 3 -5.1966351 -0.20937105       3
24  1  6.0120553  0.24387476       1
137 3 -6.0383747  2.20984770       3
61  2 -1.2257443 -3.03469429       2
41  1  7.6466593  0.57623503       1
26  1  6.4775174 -1.04708182       1
70  2 -1.0443574 -1.74662921       2
31  1  6.5351351 -0.78766052       1
85  2 -2.5818237  0.01276122       2
36  1  7.5972776 -0.33972390       1
4   1  6.6085362 -0.73929708       1
130 3 -4.2970292 -0.42496657       3
95  2 -1.8418994 -0.99664464       2
111 3 -4.1644831  1.17871159       3
117 3 -4.7101562  0.09594447       3
48  1  6.9765285 -0.43315849       1
106 3 -7.0303766  0.13158736       3
69  2 -3.5167119 -2.05931691       2
131 3 -5.9675456 -0.52249341       3
92  2 -2.0719645 -0.22536449       2
22  1  7.3872921  1.18564382       1
39  1  6.6977209 -0.90199151       1
125 3 -5.3082628  1.33894916       3
47  1  7.9455957  1.02129249       1
108 3 -5.9474167 -0.54626897       3
123 3 -7.2281863 -0.62126561       3
25  1  6.4877846 -0.15448692       1
96  2 -0.9672993 -0.40896608       2
149 3 -5.4267988  2.11763536       3
58  2 -0.1967945 -1.90480909       2
100 2 -1.4146638 -0.69091758       2
107 3 -4.3326495 -0.90276305       3
89  2 -1.1216984 -0.17330958       2
67  2 -2.4633370  0.01193815       2
45  1  6.7958450  1.25408059       1
138 3 -4.5932952  0.35495424       3
30  1  6.6519961 -0.52865075       1
17  1  8.3010066  1.79668640       1
9   1  6.3297286 -1.20813011       1
135 3 -4.6952607 -1.73516107       3
18  1  7.5140148  0.52828312       1
145 3 -6.4564371  2.08976755       3
68  2 -0.6703941 -1.51304115       2
12  1  7.0634482 -0.01186583       1
23  1  8.4484974  0.79139590       1
20  1  7.8504400  1.25653745       1
15  1  9.4800594  1.72576966       1
147 3 -5.0367690 -0.77081719       3
46  1  6.4557628 -0.76347342       1
60  2 -1.7902528 -0.66467281       2
19  1  7.8221244  1.15898751       1
144 3 -6.3829867  1.36026786       3
1   1  7.8010583  0.34057853       1
90  2 -1.8695758 -1.41834884       2
53  2 -2.2846205  0.07502495       2
115 3 -6.4317784  0.89801781       3
146 3 -5.4512238  1.17626549       3
78  2 -3.3451866  0.14511863       2
99  2  0.3864166 -1.31670824       2
118 3 -6.0412297  2.34012590       3
38  1  8.1457196  0.41229523       1
139 3 -3.6631579  0.43078471       3
105 3 -6.4339941  0.70414177       3
132 3 -4.7729922  2.10651473       3
110 3 -6.3994587  2.67334311       3
72  2 -0.9858025 -0.64502336       2
> #打印混淆矩阵
> (tab <- table(newType,train$type))
       
newType  1  2  3
      1 35  0  0
      2  0 33  1
      3  0  2 34
> #显示正确率
> cat('Accuracy:',sum(diag(tab))/length(train[,1]))
Accuracy: 0.9714286
> #将验证集代入训练好的模型中计算近似泛化误差 > T <-predict(ld,test) > #与真实分类结果进行比较 > cbind(test$type,T$x,T$class) LD1 LD2 2 1 6.8020498 -0.95158956 1 3 1 7.2276597 -0.38602966 1 5 1 7.9179194 0.59958829 1 8 1 7.3738228 0.03485147 1 11 1 8.1391093 0.80900002 1 13 1 7.0298500 -1.13888262 1 21 1 7.2270204 -0.06187541 1 28 1 7.6684138 0.29262662 1 32 1 7.0367090 0.40861452 1 33 1 9.0120836 1.65651141 1 34 1 9.2707624 2.14912000 1 37 1 8.2299196 0.38647275 1 42 1 5.2371900 -2.52488607 1 49 1 8.0798659 0.80941155 1 50 1 7.3896062 -0.17620640 1 52 2 -1.6371815 0.52584233 2 55 2 -2.4742435 -0.55650250 2 56 2 -2.1822152 -0.88107904 2 57 2 -2.1911397 0.87747596 2 63 2 -1.2405414 -2.75931501 2 65 2 -0.3383635 -0.19420599 2 66 2 -1.1566243 0.12584525 2 73 2 -3.6967069 -1.47409522 2 81 2 -1.0878173 -1.95727554 2 82 2 -0.6088858 -2.09743977 2 86 2 -1.8089897 1.23238953 2 87 2 -2.0193315 0.17092876 2 94 2 -0.3136555 -2.16381886 2 97 2 -1.4304473 -0.47985972 2 98 2 -1.3261184 -0.52945776 2 102 3 -5.1726649 -0.29910341 3 109 3 -6.0478550 -1.34049085 3 113 3 -5.3935569 0.65782366 3 114 3 -5.6792727 -0.58064337 3 116 3 -5.4686329 1.64715619 3 120 3 -4.5946379 -2.29619567 3 122 3 -5.0183151 0.24310322 3 126 3 -4.9026833 0.37255836 3 127 3 -3.8968800 -0.08723483 3 129 3 -6.1746269 0.09473298 3 133 3 -6.4616705 0.28243757 3 136 3 -6.5857811 0.74428684 3 140 3 -4.9663213 0.96355072 3 143 3 -5.1726649 -0.29910341 3 150 3 -4.2980648 0.28857515 3 > #打印混淆矩阵 > (tab <- table(T$class,test$type)) 1 2 3 1 15 0 0 2 0 15 0 3 0 0 15 > #显示正确率 > cat('Accuracy:',sum(diag(tab))/length(test[,1])) Accuracy: 1 > #Fisher线性判别 > rm(list=ls()) > library(MASS) > data(iris) > data <- iris > data$Species = as.character(data$Species) > #创造类别变量 > data$type[data$Species == 'setosa'] = 1 > data$type[data$Species == 'versicolor'] = 2 > data$type[data$Species == 'virginica'] = 3 > #利用简单随机抽样将样本集划分为训练集与验证集 > sam <- sample(1:length(data[,1]),105) > train <- data[sam,] > test <- data[-sam,] > #根据样本数据创建线性判别分析模型 > ld <- lda(type~Sepal.Length+Sepal.Width+Petal.Length+Petal.Width,data=train) > #将样本集作为验证集求分类结果 > Z <- predict(ld) > #保存预测类别 > newType <- Z$class > #与真实分类结果进行比较 > cbind(train$type,Z$x,newType) LD1 LD2 newType 44 1 6.5356943 1.380095755 1 150 3 -5.0777607 0.353229454 3 48 1 7.2092979 -0.233679182 1 17 1 8.7810490 1.832128789 1 11 1 8.5106371 0.591364581 1 13 1 7.4261015 -0.949875862 1 82 2 -0.7168056 -1.805629506 2 19 1 8.2170070 0.870266056 1 123 3 -7.9020752 -1.233817765 3 23 1 8.7554448 0.980676590 1 59 2 -1.8788410 -0.915972039 2 70 2 -1.2367635 -1.516788339 2 146 3 -5.6852127 1.788579457 3 142 3 -5.0723482 2.082228707 3 115 3 -6.9597642 1.869494439 3 7 1 7.2188221 0.472577805 1 112 3 -5.6667730 -0.156736416 3 117 3 -5.4232144 -0.112017744 3 26 1 6.8888800 -0.860736454 1 111 3 -4.6405840 1.346460523 3 29 1 8.1491362 0.125965565 1 81 2 -1.2242456 -1.597693375 2 14 1 7.6286098 -0.598455540 1 110 3 -7.1444731 2.585994755 3 40 1 7.7798631 0.022078382 1 39 1 6.9782244 -0.504346286 1 68 2 -1.0104685 -1.671771426 2 20 1 8.0502422 1.117799195 1 31 1 6.7929797 -0.656064848 1 134 3 -4.1319678 -1.006493240 2 73 2 -4.0110318 -1.372974920 2 4 1 6.8227612 -0.537268125 1 108 3 -6.6972047 -1.217364668 3 54 2 -2.3695531 -1.376553027 2 148 3 -5.1575482 0.848139667 3 141 3 -6.8305530 1.864676411 3 109 3 -6.6173517 -1.422188092 3 50 1 7.7923809 -0.058826655 1 91 2 -2.7513769 -1.544578568 2 97 2 -1.8728790 -0.435815301 2 96 2 -1.4911207 -0.557876549 2 77 2 -2.5464899 -1.021564544 2 66 2 -1.2439344 0.003029162 2 145 3 -7.0771380 2.462076368 3 38 1 8.3218303 0.213545770 1 45 1 6.7744745 0.999164236 1 58 2 -0.3713519 -1.340382308 2 35 1 7.1622529 -0.552177665 1 36 1 8.1741719 -0.035844508 1 57 2 -2.7067243 0.719387605 2 27 1 6.9079284 0.551777520 1 132 3 -5.6031449 1.030132762 3 128 3 -4.3392145 0.561003821 3 87 2 -2.2635929 -0.006748767 2 88 2 -2.4886790 -1.851739918 2 118 3 -7.1004619 1.347087687 3 119 3 -9.4291409 -0.890616169 3 46 1 6.9234400 -0.316289540 1 33 1 9.0573501 1.063438766 1 10 1 7.4135836 -0.868970825 1 78 2 -3.6650096 0.105534552 2 32 1 7.6166932 0.640755164 1 139 3 -4.1962691 0.674830697 3 103 3 -6.5226603 0.373114540 3 43 1 7.2390794 -0.114882460 1 121 3 -6.4785762 1.623818440 3 15 1 10.1229028 1.482252025 1 125 3 -6.0718150 1.194903725 3 92 2 -2.5655637 -0.379597733 2 21 1 7.6071362 -0.210545218 1 116 3 -6.0199587 2.084095792 3 95 2 -2.2468978 -0.820309281 2 93 2 -1.3874483 -1.124059988 2 135 3 -5.6483660 -2.247095684 3 52 2 -1.9604386 0.420606745 2 37 1 8.8751646 0.414644969 1 127 3 -4.2307963 0.275427178 3 137 3 -6.8919261 2.468751536 3 98 2 -1.5631688 -0.569521563 2 72 2 -1.0384324 -0.432712540 2 80 2 0.2825956 -1.208391313 2 67 2 -3.1266046 0.070901690 2 120 3 -4.9979151 -2.051118149 3 114 3 -6.2027782 0.131952933 3 69 2 -3.4910411 -1.516772693 2 130 3 -4.8967334 -1.106964081 3 42 1 5.9270652 -1.555646363 1 64 2 -2.9521004 -0.683186676 2 86 2 -2.4035699 1.146743118 2 75 2 -1.3368410 -0.579461256 2 143 3 -5.8335378 0.090796722 3 140 3 -5.3380144 1.122071295 3 1 1 8.1663998 0.325667325 1 83 2 -1.0009115 -0.820471045 2 28 1 8.0234544 0.211840448 1 85 2 -3.3529324 0.080841383 2 89 2 -1.5995061 -0.127256512 2 18 1 7.9150690 0.642460485 1 147 3 -5.2724641 -0.214659306 3 105 3 -7.1968652 0.828583809 3 41 1 8.0580144 0.756287362 1 63 2 -1.1801689 -2.546513654 2 122 3 -5.6685504 0.829975549 3 107 3 -5.0735507 -0.234382628 3 47 1 8.0454637 0.692149004 1 > #打印混淆矩阵 > (tab <- table(newType,train$type)) newType 1 2 3 1 36 0 0 2 0 33 1 3 0 0 35 > #显示正确率 > cat('Accuracy:',sum(diag(tab))/length(train[,1])) Accuracy: 0.9904762> #将验证集代入训练好的模型中计算近似泛化误差 > T <-predict(ld,test) > #与真实分类结果进行比较 > cbind(test$type,T$x,T$class) LD1 LD2 2 1 7.2879346 -0.63805255 1 3 1 7.5785711 -0.12979200 1 5 1 8.1836634 0.52536908 1 6 1 7.7566121 1.39670067 1 8 1 7.6666992 0.02704823 1 9 1 6.5916877 -0.80793523 1 12 1 7.1842622 -0.07186911 1 16 1 9.2604596 2.57316476 1 22 1 7.6684840 1.23986044 1 24 1 6.3832248 0.56001189 1 25 1 6.4159345 -0.39844020 1 30 1 6.8102433 -0.45636309 1 34 1 9.5320477 1.66891133 1 49 1 8.3974733 0.59633443 1 51 2 -1.5423430 -0.14371955 2 53 2 -2.5494836 -0.23440252 2 55 2 -2.6250939 -0.47214778 2 56 2 -2.7716342 -0.95711830 2 60 2 -2.1825564 -0.15706564 2 61 2 -1.2921165 -2.34199387 2 62 2 -2.0187853 0.38256324 2 65 2 -0.4493874 0.22229672 2 71 2 -4.0485780 1.06926437 3 74 2 -2.5798663 -1.51150491 2 76 2 -1.4875258 -0.18673290 2 79 2 -2.8043766 -0.14370961 2 84 2 -4.8532177 -0.86952245 3 90 2 -2.1086982 -0.98708920 2 94 2 -0.3886155 -1.54008407 2 99 2 0.5024002 -0.51222584 2 100 2 -1.7471972 -0.52169018 2 101 3 -8.2981213 2.15538466 3 102 3 -5.8335378 0.09079672 3 104 3 -6.0360789 -0.40566699 3 106 3 -7.7496056 -0.41373390 3 113 3 -5.8377150 0.82345220 3 124 3 -4.5041691 -0.03313161 3 126 3 -5.6507584 -0.30162799 3 129 3 -6.8073348 0.34501073 3 131 3 -6.4535806 -0.88255921 3 133 3 -7.0586655 0.66180389 3 136 3 -6.8585571 0.75916772 3 138 3 -5.4059508 0.08768402 3 144 3 -7.1039586 1.41107423 3 149 3 -6.2415407 2.37464228 3 > #打印混淆矩阵 > (tab <- table(T$class,test$type)) 1 2 3 1 14 0 0 2 0 15 0 3 0 2 14 > #显示正确率 > cat('Accuracy:',sum(diag(tab))/length(test[,1])) Accuracy: 0.9555556

可以看出,和Python中的效果相差无几。

 

以上就是关于线性判别的基本内容,如有意见望提出。

 

你可能感兴趣的:((数据科学学习手札17)线性判别分析的原理简介&Python与R实现)