1 k-近邻算法概述
k-近邻算法采用测量不同特征值之间的距离方法进行分类。
| k-近邻算法 | |
|---|---|
| 优点 | 精度高、对异常值不敏感、无数据输入假定。 |
| 缺点 | 计算复杂度高、空间复杂度高。 |
| 适用数据范围 | 数值型、标称型。 |
KNN是通过测量不同特征值之间的距离进行分类。它的思路是：如果一个样本在特征空间中的k个最相似(即特征空间中最邻近)的样本中的大多数属于某一个类别，则该样本也属于这个类别，其中K通常是不大于20的整数。KNN算法中，所选择的邻居都是已经正确分类的对象。该方法在定类决策上只依据最邻近的一个或者几个样本的类别来决定待分样本所属的类别。
KNN算法的思想：就是在训练集中数据和标签已知的情况下，输入测试数据，将测试数据的特征与训练集中对应的特征进行相互比较，找到训练集中与之最为相似的前K个数据，则该测试数据对应的类别就是K个数据中出现次数最多的那个分类，其算法的描述为：
- 计算测试数据与各个训练数据之间的距离；
- 按照距离的递增关系进行排序；
- 选取距离最小的K个点；
- 确定前K个点所在类别的出现频率；
- 返回前K个点中出现频率最高的类别作为测试数据的预测分类。
1.1 准备：使用Python导入数据
import numpy as np
import operator
def createDataSet():
group = np.array([[1.0,1.1],[1.0,1.0],[0,0],[0,0.1]])
labels = ['A','A','B','B']
return group, labels
import kNN
group,labels = kNN.createDataSet()
group
Out[15]:
array([[1. , 1.1],
[1. , 1. ],
[0. , 0. ],
[0. , 0.1]])
labels
Out[16]: ['A', 'A', 'B', 'B']
1.2 实施kNN分类算法
'''
intX:用于分类的输入向量
dataSet:输入的训练样本集
labels:标签向量
k:用于选择最近邻居的数目
'''
# 使用k-近邻算法将每组数据划分到某个类中
def classify0(inX, dataSet, labels, k):
dataSetSize = dataSet.shape[0]
# 使用欧式距离公式计算距离
diffMat = np.tile(inX, (dataSetSize,1)) - dataSet
sqDiffMat = diffMat**2
sqDistances = sqDiffMat.sum(axis=1)
distances = sqDistances**0.5
# 按照距离递增次序排序（从大到小）
sortedDistIndicies = distances.argsort()
classCount={}
for i in range(k):
# 选择距离最小的k个点
voteIlabel = labels[sortedDistIndicies[i]]
classCount[voteIlabel] = classCount.get(voteIlabel,0) + 1
# 对前k个点类别频率进行排序
sortedClassCount = sorted(classCount.items(),key=operator.itemgetter(1), reverse=True)
# 返回前k个点频率最高的类别
return sortedClassCount[0][0]
- 测试
kNN.classify0([0,0], group, labels, 3)
Out[18]: 'B'
1.3 如何测试分类器
分类错误率：分类器给出错误结果的次数除以测试执行的总数。
2 实例：使用k-近邻算法改进约会网站的配对效果
2.1 准备数据：从文本文件中解析数据
numpy.zeros(shape, dtype=float, order='C')
- 返回来一个给定形状和类型的用0填充的数组；
# 处理输入格式问题
def file2matrix(filename):
fr = open(filename)
#get the number of lines in the file
numberOfLines = len(fr.readlines())
#prepare matrix to return
returnMat = np.zeros((numberOfLines,3))
#prepare labels return
classLabelVector = []
fr = open(filename)
index = 0
for line in fr.readlines():
line = line.strip()
listFromLine = line.split('\t')
returnMat[index,:] = listFromLine[0:3]
classLabelVector.append(int(listFromLine[-1]))
index += 1
return returnMat,classLabelVector
import importlib
importlib.reload(kNN)
Out[34]:
datingDataMat, datingLabels = kNN.file2matrix('datingTestSet2.txt')
datingDataMat
Out[36]:
array([[4.0920000e+04, 8.3269760e+00, 9.5395200e-01],
[1.4488000e+04, 7.1534690e+00, 1.6739040e+00],
[2.6052000e+04, 1.4418710e+00, 8.0512400e-01],
...,
[2.6575000e+04, 1.0650102e+01, 8.6662700e-01],
[4.8111000e+04, 9.1345280e+00, 7.2804500e-01],
[4.3757000e+04, 7.8826010e+00, 1.3324460e+00]])
datingLabels[0:20]
Out[37]: [3, 2, 1, 1, 1, 1, 3, 3, 1, 3, 1, 1, 2, 1, 1, 1, 1, 1, 2, 3]
2.2 分析数据：使用Matplotlib创建散点图
import matplotlib
import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(datingDataMat[:,1], datingDataMat[:,2])
Out[11]:
plt.show()
- 个性化标记散列图上的点
ax.scatter(datingDataMat[:,1], datingDataMat[:,2],15.0*np.array(datingLabels),15.0*np.array(datingLabels))
Out[9]:
plt.show()
ax.scatter(datingDataMat[:,0], datingDataMat[:,1],15.0*np.array(datingLabels),15.0*np.array(datingLabels))
Out[11]:
plt.show()
2.3 准备数据：归一化数值
- newValue = (oldValue-min)/(max-min)
# 归一化处理，将数字特征值转化为0到1的区间
def autoNorm(dataSet):
# 选取列中的最小值
minVals = dataSet.min(0)
# 选取列中的最大值
maxVals = dataSet.max(0)
ranges = maxVals - minVals
normDataSet = np.zeros(np.shape(dataSet))
m = dataSet.shape[0]
normDataSet = dataSet - np.tile(minVals, (m,1))
#element wise divide
return normDataSet, ranges, minVals
# 特征值相除
# numpy中矩阵除法需使用linalg.solve(matA,matB)
normDataSet = normDataSet/np.tile(ranges, (m,1))
- 测试
normMat, ranges, minvals = kNN.autoNorm(datingDataMat)
normMat
Out[17]:
array([[4.0920000e+04, 8.3269760e+00, 9.5279600e-01],
[1.4488000e+04, 7.1534690e+00, 1.6727480e+00],
[2.6052000e+04, 1.4418710e+00, 8.0396800e-01],
...,
[2.6575000e+04, 1.0650102e+01, 8.6547100e-01],
[4.8111000e+04, 9.1345280e+00, 7.2688900e-01],
[4.3757000e+04, 7.8826010e+00, 1.3312900e+00]])
ranges
Out[18]: array([9.1273000e+04, 2.0919349e+01, 1.6943610e+00])
minvals
Out[19]: array([0. , 0. , 0.001156])
2.4 测试算法：作为完整程序验证分类器
#分类器针对约会网站的测试代码
def datingClassTest():
#hold out 10%
hoRatio = 0.10
#load data setfrom file
datingDataMat,datingLabels = file2matrix('datingTestSet2.txt')
# 归一化特征值
normMat, ranges, minVals = autoNorm(datingDataMat)
# 计算测试向量的数量
m = normMat.shape[0]
numTestVecs = int(m*hoRatio)
errorCount = 0.0
for i in range(numTestVecs):
classifierResult = classify0(normMat[i,:],normMat[numTestVecs:m,:],datingLabels[numTestVecs:m],3)
print ("the classifier came back with: %d, the real answer is: %d" % (classifierResult, datingLabels[i]))
if (classifierResult != datingLabels[i]): errorCount += 1.0
print ("the total error rate is: %f" % (errorCount/float(numTestVecs)))
print (errorCount)
1.5 使用算法：构建完整可用系统
# 预测函数
def classifyPerson():
resultList = ['not at all','in small doses','in large doses']
percentTats = float(input("percentage of time spent playing video games?"))
miles = float(input("frequent fliter miles earned per year?"))
iceCream = float(input("liters of ice cream consumed per year?"))
datingDataMat,datingLabels = file2matrix('datingTestSet2.txt')
normMat,ranges,minVals = autoNorm(datingDataMat)
inArr = np.array([miles,percentTats,iceCream])
classifierResult = classify0((inArr-minVals)/ranges,normMat,datingLabels,3)
print("\nyou will probably like this person:",resultList[classifierResult - 1])
kNN.classifyPerson()
percentage of time spent playing video games?10
frequent fliter miles earned per year?10000
liters of ice cream consumed per year?0.5
you will probably like this person: in small doses
3 实例：手写识别系统
3.1 准备数据：将图像转换为测试向量
# 将图像转换为向量
def img2vector(filename):
# 创建1*1024的NumPy数组
returnVect = np.zeros((1,1024))
fr = open(filename)
for i in range(32):
lineStr = fr.readline()
for j in range(32):
returnVect[0,32*i+j] = int(lineStr[j])
return returnVect
testVector = kNN.img2vector('testDigits/0_13.txt')
testVector[0,0:31]
Out[28]:
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1.,
1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
3.2 测试算法：使用k-近邻算法识别手写数字
# 识别手写数字
def handwritingClassTest():
hwLabels = []
#load the training set
trainingFileList = listdir('trainingDigits')
m = len(trainingFileList)
trainingMat = np.zeros((m,1024))
for i in range(m):
fileNameStr = trainingFileList[i]
#take off .txt
fileStr = fileNameStr.split('.')[0]
classNumStr = int(fileStr.split('_')[0])
hwLabels.append(classNumStr)
trainingMat[i,:] = img2vector('trainingDigits/%s' % fileNameStr)
#iterate through the test set
testFileList = listdir('testDigits')
errorCount = 0.0
mTest = len(testFileList)
for i in range(mTest):
fileNameStr = testFileList[i]
#take off .txt
fileStr = fileNameStr.split('.')[0]
classNumStr = int(fileStr.split('_')[0])
vectorUnderTest = img2vector('testDigits/%s' % fileNameStr)
classifierResult = classify0(vectorUnderTest, trainingMat, hwLabels, 3)
print ("the classifier came back with: %d, the real answer is: %d" % (classifierResult, classNumStr))
if (classifierResult != classNumStr): errorCount += 1.0
print ("\nthe total number of errors is: %d" % errorCount)
print ("\nthe total error rate is: %f" % (errorCount/float(mTest)))
- 测试
kNN.handwritingClassTest()