✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-075

K-Nearest Neighbors (KNN)

K-nearest neighbors is a supervised learning algorithm that predicts a new observation using the labels or values of its closest training examples.

Euclidean distance: d(x, xᵢ) = √[Σⱼ(xⱼ - xᵢⱼ)²]; Classification: ŷ = mode(y₁, y₂, ..., yₖ)

Explanation

K-nearest neighbors is a non-parametric, instance-based learning algorithm. Instead of learning a fixed mathematical model during training, it stores the training observations and uses them when a prediction is requested. For classification, it identifies the k nearest training examples and assigns the class receiving the majority vote. For regression, it commonly predicts the average or a distance-weighted average of the neighbors' target values. Distance measures such as Euclidean and Manhattan distance can be used. The choice of k affects model behavior: a small value can make predictions sensitive to noise, while a large value may smooth away meaningful local patterns. Feature scaling is important when numerical features have different ranges. KNN can become computationally expensive when the training dataset is very large.

Variables

SymbolMeaning

k

Number of nearest neighbors considered

x

New observation

xᵢ

Training observation i

d

Distance between observations

yᵢ

Target label of neighbor i

ŷ

Predicted class or target value

Example or application

A model classifies a fruit using weight and color measurements. If five nearby labeled examples contain three apples and two oranges, a KNN classifier with k = 5 predicts apple.

Text

KNN makes predictions by comparing a new data point with nearby observations in the training dataset.

Key Points

  • KNN is a non-parametric learning algorithm.

  • It can perform classification and regression.

  • The value of k controls the number of neighbors.

  • Distance metrics determine which observations are nearest.

  • Feature scaling can strongly affect distance calculations.

  • Prediction can be expensive for large datasets.

Applications

  • Pattern recognition

  • Recommendation systems

  • Image classification

  • Similarity-based retrieval

  • Medical classification

Learning Outcomes

  • Explain how KNN makes predictions.

  • Distinguish KNN classification from regression.

  • Understand the effect of the k parameter.

  • Explain why distance metrics and feature scaling matter.

Practice Questions

  • What does the k in KNN represent?

  • How does KNN classification differ from KNN regression?

  • What happens when k is too small?

  • Why is feature scaling important for KNN?

Related Concepts

  • Distance Metrics

  • Euclidean Distance

  • Instance-Based Learning

  • Classification

  • Regression

Common Mistakes

  • Choosing k without validation.

  • Ignoring feature scales when calculating distances.

  • Assuming the closest observations always provide reliable predictions.

  • Using KNN on very high-dimensional data without considering distance concentration.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Cover, T. M., and Hart, P. E. (1967). Nearest Neighbor Pattern Classification.
  • Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning