3 K-Nearest Neighbor

3. K-Nearest Neighbor

Click here for notes and code

Nearest Neighbor(1NN)

• To classify a new example $x$:

  • Label $x$ with the label of the closest example to $x$in the training set.

• Euclidean Distance

  $$D\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=\sqrt{\sum _{k=1}^d{\left(x_{ik}-x_{jk}\right)}^2}$$

K-Nearest-Neighbors (KNN)

• Low latitude large-scale data

• $x$ is close to a blue point.

  But most of the next closest points are red.

• Find $k$ nearest neighbors of $x$.

• Label $x$ with the majority label within the $k$ nearest neighbors

Consider:How can we handle ties for even values of k?

KNN rule is certainly simple and intuitive,but does it work?

• KNN is a Goo Approximator on any smooth distribution:

  • Converges to perfect solution if clear separation

    $$\underset{n\to \infty }{\lim }{ϵ}_{\mathrm{KNN}}(n)\le 2{ϵ}^{\ast }\left(1-{ϵ}^{\ast }\right)$$

    (Cover Hart,1969)

• KNN is Good Model on complex distributions.

The Effect of K

• Increasing k simplifies the decision boundary

  Majority voting means less emphasis on individual points

• Choose best k on the validation set.(Often set k as an odd number).

KNN: Inductive Bias

• Similar points have similar labels.
• All dimensions are created equal!

Feature Normalization

• Z-score normalization:
  • For each feature dimension j,compute based on its samples:
  • Mean ${\mu }_j=\frac{1}{N}\sum _{i=1}^N{x}_{ij}$
  • Variance ${\sigma }_j=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left(x_{ij}-{\mu }_j\right)}^2}$
  • Normalize the feature into a new one: ${\hat{x}}_{ij}\leftarrow \frac{x_{ij}-{\mu }_j}{{\sigma }_j}$

Distance Selection

• Cosine Distance: $\cos \left({\mathbf{x}}_{\mathbf{i}},{\mathbf{x}}_{\mathbf{j}}\right)=\frac{⟨{\mathbf{x}}_{\mathbf{i}},{\mathbf{x}}_{\mathbf{j}}⟩}{\Vert {\mathbf{x}}_{\mathbf{i}}\Vert \Vert {\mathbf{x}}_{\mathbf{j}}\Vert }$
• Minkowski Distance: $D_p\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=\sqrt[p]{\sum _{k=1}^d{\left|x_{ik}-x_{jk}\right|}^p}$
• Mahalanobis Distance: $D_M\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=\sqrt{{\left({\mathbf{x}}_i-{\mathbf{x}}_j\right)}^{\mathrm{T}}\mathbf{M}\left({\mathbf{x}}_i-{\mathbf{x}}_j\right)}$

Weighted KNN

• Vanilla KNN uses majority vote to predict discrete outputs.

• Weighted KNN assigns greater weights to closer neighbors.

  • Weighting is a common approach used in machine learning.

• Distance-weighted Classification:

  $$\hat{y}={\mathrm{argmax}}_{c\in y}\sum _{{\mathbf{x}}^{\prime }\in {\mathrm{KNN}}_c(\mathbf{x})}\frac{1}{D{\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)}^2}$$

• Distance-weighted Regression:

  $$\hat{y}=\sum _{x^{\prime }\in \mathrm{KNN}(x)}\frac{1}{D{\left(x,x^{\prime }\right)}^2}{y}^{\prime }$$

• Other weighting function (e.g.exponential family)can also be used.

KNN Summary

• When to use:
  • Few attributes per instance (expensive computation)
  • Lots of training data (curse of dimensionality
• Advantages:
  • Agnostically learn complex target functions
  • Do not lose information (store original data)
  • Data number can be very large (big pro!)
  • Class number can be very large (biggest pro!)
    • All other ML algorithms may fail here!
• Disadvantages:
  • Slow at inference time (acceleration a must)
  • Ineffective in high dimensions(curse of dimensionality)
  • Fooled easily by irrelevant attributes (feature engineering crucial)